In biology, release of Ca$^{2+}$ ions in the cytosol is essential to trigger or control many cell functions. Calcium signaling acutely depends on lipid membrane permeability to Ca$^{2+}$. For proper understanding of membrane permeability to Ca$^{2+}$, both membrane hydration and the structure of the hydrophobic core must be taken into account. Here, we vary the hydrophobic core of bilayer membranes and observe different types of behavior in high-throughput wide-field second harmonic imaging. Ca$^{2+}$ translocation is observed through mono-unsaturated (DOPC:DOPA) membranes, reduced upon the addition of cholesterol, and completely inhibited for branched (DPhPC:DPhPA) and poly-unsaturated (SLPC:SLPA) lipid membranes. We propose, using molecular dynamics simulations, that ion transport occurs through ion-induced transient pores, which requires nonequilibrium membrane restructuring. This results in different rates at different locations and suggests that the hydrophobic structure of lipids plays a much more sophisticated regulating role than previously thought.