Two Routes to Class Organization in a Solvable Model of Contrastive Learning

Contrastive learning can yield class-informative representations from unlabeled data, but how this class organization emerges as data scale grows remains unclear. We study this question in a high-dimensional InfoNCE model with data drawn from a latent Gaussian mixture and a single-layer encoder. In the proportional limit, a replica calculation under a reduced replica-symmetric (RS) and class-permutation-symmetric ansatz yields a free-energy functional and saddle-point equations for a small set of macroscopic order parameters. We quantify class organization by the fraction of representation variance explained by between-class mean differences. As sample complexity increases, we find two routes to this organization: a specialized branch either emerges continuously from the unspecialized branch or appears at finite amplitude and coexists with a locally stable unspecialized branch before becoming preferred through an RS free-energy crossing. The same nonlinear activation can exhibit either route as the number of classes changes, while the number of negatives shifts the transition location. A local Landau-type expansion of the profiled free energy identifies a cubic term associated with finite-amplitude competition. Its leading high-InfoNCE-temperature form reveals contributions from activation curvature, class structure, and contrastive weighting. Finite-dimensional simulations quantitatively reproduce the predicted representation statistics and qualitatively recover the branch structures.