Lateral inhomogeneity plays a critical role for many properties of cholesterol-containing membranes, yet the thermodynamic forces involved in inhomogeneity remain poorly understood. Based on coarse-grained simulations of cholesterol in four increasingly unsaturated phospholipids, we computed lateral density fluctuations and free energies of domain formation, and we quantitatively relate those to variations in the chemical potential of cholesterol. Our simulations suggest that the lateral organization is dominated by weak repulsive cholesterol interactions, leading to a significantly more homogeneous distribution as compared to a two-dimensional ideal gas. Hence, phospholipids provide a “good” solvent for cholesterol. Unexpectedly, the degree of unsaturation of the phospholipid has only a minor effect on the lateral inhomogeneity of cholesterol in binary lipid mixtures. These results provide a link between functional properties and thermal fluctuations in lipid membranes.