acceptodds
Under review as a conference paper at ICLR 2027

Two Routes to Class Organization in a Solvable Model of Contrastive Learning

Abstract

Contrastive learning can yield class-informative representations from unlabeled data, but how this class organization emerges as data scale grows remains unclear. We study this question in a high-dimensional InfoNCE model with data drawn from a latent Gaussian mixture and a single-layer encoder. In the proportional limit, a replica calculation under a reduced replica-symmetric (RS) and class-permutation-symmetric ansatz yields a free-energy functional and saddle-point equations for a small set of macroscopic order parameters. We quantify class organization by the fraction of representation variance explained by between-class mean differences. As sample complexity increases, we find two routes to this organization: a specialized branch either emerges continuously from the unspecialized branch or appears at finite amplitude and coexists with a locally stable unspecialized branch before becoming preferred through an RS free-energy crossing. The same nonlinear activation can exhibit either route as the number of classes changes, while the number of negatives shifts the transition location. A local Landau-type expansion of the profiled free energy identifies a cubic term associated with finite-amplitude competition. Its leading high-InfoNCE-temperature form reveals contributions from activation curvature, class structure, and contrastive weighting. Finite-dimensional simulations quantitatively reproduce the predicted representation statistics and qualitatively recover the branch structures.

Primary area:
learning theory
Keywords:
contrastive learning, InfoNCE, replica method, statistical physics, phase transition, high-dimensional limit

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aC4iOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how class-organized representations emerge in contrastive learning as the sample complexity increases. The authors study a high-dimensional Gaussian mixture model with a single-layer encoder trained using InfoNCE. Using the replica method under replica-symmetric and class-permutation-symmetric assumptions, they derive a reduced free-energy formulation characterized by four scalar order parameters.

The main finding is that class organization can emerge through two distinct routes: (1) a continuous transition, where class specialization develops gradually as the sample complexity increases, and (2) a discontinuous transition, where a specialized solution coexists with an unspecialized solution and eventually becomes preferred through a free-energy crossing. The authors show that the transition behavior depends on the activation function and the number of classes.

To further characterize these transitions, the authors perform a Landau-type expansion of the free energy and derive a high-temperature expression for the cubic coefficient, relating it to activation curvature, class structure, and contrastive weighting. The theoretical predictions are compared with finite-dimensional simulations, which reproduce the predicted representation statistics and qualitatively support the two transition behaviors.

Critical strengths
  1. Interesting research question. The paper studies how class organization emerges in contrastive learning as the sample complexity increases. Rather than focusing only on the properties of learned representations, it investigates the emergence of these properties and identifies two qualitatively different transition behaviors. I find this an interesting perspective for understanding representation learning.

  2. Connection between theoretical analysis and numerical experiments. The paper develops a theoretical framework based on the replica method and uses it to characterize different stationary branches and their free-energy competition. The authors also conduct finite-dimensional experiments that qualitatively reproduce the predicted transition behaviors. This provides empirical support for the theoretical picture within the considered model.

Critical weaknesses
  1. The paper could be presented more clearly. It relies heavily on specialized terminology and dense mathematical derivations, making the main arguments difficult to follow. In particular, the theorems are not stated or interpreted in an accessible manner. The authors should provide more intuition, better motivate the assumptions, and explain the implications of the theoretical results more clearly.

  2. The theoretical analysis focuses primarily on between-class variance while largely overlooking within-class variance. Both are important for understanding class separation and the behavior of InfoNCE. An analysis that accounts for within-class variance would therefore provide a more complete and informative characterization of the objective.

Questions for authors

Could the authors provide an intuitive explanation of the main theoretical results? In particular, what does each central theorem reveal about InfoNCE, why are its assumptions necessary, and what insight does it provide beyond the formal derivation? A clearer explanation of these points would help me assess the significance and generality of the results. Does the analysis extend to within-class variance? If not, could the authors explain why focusing only on between-class variance is sufficient, or clarify how accounting for within-class variance might affect the conclusions?

3 comments
AuthorsOpenReview ↗

Rebuttal 1/2

We thank the reviewer for their constructive feedback and suggestions for improving the clarity and presentation of our work.

W1. Following your feedback, we have used the additional space to explain the intuition behind the results, the role of the assumptions, and their implications. We also want the final paper to be easier for the community to follow. Below, we focus on what the results reveal about InfoNCE and why they matter.

W2. Our original analysis already includes both between-class differences and within-class variation. Both enter our measure of class organization, as explained in Part 2.

Could the authors provide an intuitive explanation of the main theoretical results?

Our main finding is that, in the InfoNCE model we study, class organization need not develop gradually with more data. It can appear abruptly, while training may fail to find the strongly organized state that the objective already favors. Our main technical result turns the high-dimensional weight problem into a set of solvable equations, predicting the minimum regularized InfoNCE loss and the corresponding representation statistics.

Two distinctions help explain this finding: which state the objective favors versus which state training can reach, and gradual versus abrupt class organization.

Equilibrium analysis asks what representation corresponds to the global minimum of the loss. Training starts from an initialization and updates the weights step by step; it may remain near a higher-loss local minimum. A better representation can therefore exist without training finding it. Our theory predicts which states the objective favors, while experiments check which states training actually reaches.

Gradual emergence means that class structure develops little by little from the existing state. In abrupt emergence, weakly and strongly organized states coexist, and more data change which one the objective favors. Yet the weakly organized state can persist. The preferred representation changes, but the trained representation need not follow.

The experiments also show this gap. As input dimension increases from 2,000 to 20,000, the sample-to-dimension ratio needed for half of the random initializations to reach strong organization rises from about 0.17 to 0.22 (Appendix C.7). With same-class negatives allowed, random initialization does not reach the strongly organized branch over the tested range, whereas initialization aligned with the class directions can (Appendix C.6). In these settings, increasing scale does not automatically resolve the optimization difficulty.

What does each central theorem reveal about InfoNCE [...]?

Result 3.2 asks what representations InfoNCE supports at a given amount of data, and which ones the objective favors. To our knowledge, this is the first characterization of regularized InfoNCE empirical risk minimization in the model studied here through a closed set of equations in the high-dimensional proportional limit. It lets us ask not only whether class information can be recovered, but how class structure forms as data increase.

Result 3.3 asks whether a small change toward distinguishing classes would be favorable near a state without class organization. Together with the comparison between states, it explains how a strongly organized state can already be preferred while a small move toward class separation from the weakly organized state need not be favorable. This is a local test; whether training actually leaves that state is examined in our experiments.

In the supervised models discussed in our paper, ReLU units gradually develop distinct roles. In our InfoNCE model, keeping the same smooth-ReLU activation and changing the number of classes can instead change emergence from gradual to abrupt. Our local analysis offers a further clue: comparing positive and negative samples interacts with the activation's nonlinearity to produce local asymmetry. This is not, by itself, a rule for predicting the emergence route. A natural next step is to compare different contrastive losses, separating effects shared by positive-negative comparison from those specific to structures such as softmax, and looking for more general rules.

AuthorsOpenReview ↗

Rebuttal 2/2

What insight does it provide beyond the formal derivation?

Training may fail to learn clear class structure even when the data and objective already support it. In our model, the objective may favor a strongly organized state while training remains in a weakly organized one.

An abrupt equilibrium transition can also be hidden by a smooth average training curve. Different runs may reach strong organization at different dataset sizes. After averaging, this can look like gradual improvement. The theory helps distinguish two explanations: individual representations may be improving gradually, or more runs may be finding a different state that is already well organized.

Why are its assumptions necessary [...]? A clearer explanation [...] would help me assess the significance and generality.

These assumptions mainly make the problem manageable enough to solve; they do not mean that the phenomena can only occur under these conditions. Balanced classes, orthogonal directions, and matching the representation dimension to the number of classes give each class the same learning conditions. Weak-signal scaling keeps the signal and noise contributions comparable before the activation as dimension grows. A single-layer network and a simplified description of relationships between solutions then let us solve for and compare different representation states.

We retain the simple setting because it already reveals behavior worth understanding. Class organization can develop gradually or appear abruptly, and training may fail to find the state that the objective already favors. Building a coherent, testable theoretical picture of these phenomena is the main value of our work.

Class imbalance, correlated directions, and more general representation dimensions can, in principle, be included through more detailed order parameters, at the cost of more complicated equations and analysis. We have not established results for general deep networks or heterogeneous semantics, but the current model provides a testable starting point. By gradually adding more complex structures, we can then ask which phenomena survive and which changes reflect new mechanisms.

Does the analysis extend to within-class variance [...]?

Within-class variation is already part of the analysis. How clearly classes are separated depends both on how far apart their centers are and on how widely samples spread within each class. If class centers move apart while samples also spread out within each class, class structure need not become clearer. Our organization measure accounts for both effects. Its definition and the theoretical expression for within-class variance are given in Section 2 and Appendix B.2:

ηcls=Tr⁡(ΣB)Tr⁡(ΣB)+Tr⁡(ΣW).\eta_{\mathrm{cls}} = \frac{\operatorname{Tr}(\Sigma_B)}{\operatorname{Tr}(\Sigma_B)+\operatorname{Tr}(\Sigma_W)}.

The numerator measures between-class variation, while the denominator includes both between-class and within-class variation. Both depend on the learned representation and are predicted by the theory. Within-class variation is neither fixed nor ignored. Although the within-class input noise is specified by the data model, the within-class variance of the learned representations is not fixed. The gradual or abrupt emergence we describe therefore concerns how clearly class structure develops relative to the spread within classes, rather than just how far the class centers move apart.

We hope these explanations make our main findings and their significance clearer, and clarify how within-class variation is included in the analysis. If these responses address your main concerns, we would appreciate your reassessment of the paper in light of the revisions. We would be happy to discuss any remaining questions.

Thank you for your rebuttal. Unfortunately, the technical content of this work is beyond my current expertise, and I do not feel sufficiently qualified to assess the mathematical arguments rigorously. Therefore, I will maintain my original score, but I kindly ask the Area Chair (AC) to assign less weight to my evaluation when making the final decision.

Reviewer_HN3FOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This article profoundly proposes that contrastive learning does not necessarily smoothly learn category structures as the amount of data increases; Under certain conditions, the representation space may have two competing states that suddenly transition from an unorganized state to a class organized state through a physical phase transition.

Critical strengths

The paper addresses a clear theoretical question that is not captured by standard analyses of representation recovery: how class organization emerges as data scale increases.

The combination of replica analysis, branch continuation, free-energy comparison, and the local Landau expansion provides a coherent theoretical picture rather than a single isolated result.

The finite-(N) validation is unusually thorough for this type of theoretical work, including multiple initializations, scans over (C), (K), and temperature, alternative negative sampling, and downstream generalization analysis.

Critical weaknesses
  1. The central conclusions are derived under a fairly restrictive reduced replica-symmetric and class-permutation-symmetric ansatz. In particular, cross-replica overlaps are constrained to a scalar form, and the paper does not establish stability against more general replica-symmetry-breaking or non-permutation-symmetric solutions. This matters because branch coexistence and free-energy selection are central to the paper's conclusions.

  2. The solvable setting is highly specialized: orthogonal Gaussian-mixture class directions, weak-signal scaling, a single-layer encoder, and the symmetry-matched choice (M=C). These assumptions make the analysis tractable, but it remains unclear which aspects of the two-route picture should survive in deeper contrastive models with heterogeneous semantic structure.

  3. The real-data experiment does not directly validate the paper's main prediction. On CIFAR-10, ηcls\eta_{\mathrm{cls}} tracks projector linear-probe accuracy as dataset size increases, but the experiment does not demonstrate either continuous versus finite-amplitude emergence or coexistence of macroscopic states. It therefore supports the proposed statistic more directly than the central two-route theory.

  4. The cubic coefficient should not be interpreted too strongly as a signature of the emergence route. The paper itself finds (a_3<0) for the (C=3) smooth-ReLU setting despite continuous emergence of the specialized branch, showing that the cubic response diagnoses local asymmetry but is not sufficient to distinguish the two routes. The conceptual role of this result could therefore be stated more carefully.

  5. The practical implication of the two-route picture remains unclear. The analysis identifies temperature, negative sampling, activation curvature, and initialization as factors associated with the emergence landscape, but the paper does not show whether these insights can be used to steer optimization toward the better-organized branch. In fact, the finite-NN results show that the lower-free-energy branch may remain inaccessible from random initialization. A controlled experiment demonstrating that a theoretically motivated schedule or initialization can alter the emergence route and improve downstream performance would substantially strengthen the significance of the theory.

Questions for authors
  1. It would be useful to report an explicit stability analysis beyond the reduced RS subspace, or at least discuss which omitted perturbation modes are most likely to invalidate the branch-selection picture.

  2. Near the transition, the finite-(N) setting contains relatively few training groups and the empirical crossover is smoother and shifted relative to theory. A more quantitative finite-size scaling analysis would help distinguish finite-(N) effects from limitations of the asymptotic theory.

  3. In the unconditioned-negative experiment, lower RS free energy does not guarantee that random initialization reaches the strongly organized state. This optimization-versus-equilibrium gap deserves more discussion because practical contrastive learning follows optimization dynamics rather than thermodynamic branch selection.

5 comments
AuthorsOpenReview ↗

Rebuttal 1/2

We thank the reviewer for the insightful comments, which have helped us strengthen the technical work and explain its significance more clearly.

W1. The central conclusions are derived under a fairly restrictive [...] ansatz [...] the paper does not establish stability against more general [...] solutions.

We agree that this matters for coexistence and branch selection. Appendix D.6 now adds two checks beyond the four-variable description.

Nonuniform class alignment. We perturb RR along two directions outside the reduced ansatz while keeping the weight-overlap and response matrices fixed. For smooth ReLU with C=10C=10 at α=0.174\alpha=0.174, near the reduced free-energy crossing, the curvatures are:

BranchDiagonal directionOff-diagonal direction
Forward3.028 [2.753, 3.303]2.940 [2.492, 3.388]
Backward2.975 [2.657, 3.293]4.490 [4.080, 4.900]

Values are in units of 10−310^{-3}; brackets give 95% sampling confidence intervals. Both branches are stable against these perturbations, and independent finite-difference checks confirm the positive signs.

Off-diagonal overlaps. We allow off-diagonal weight overlaps and responses while retaining RS and class-permutation symmetry, then solve the extended saddle-point equations. The emergence routes remain unchanged in every configuration examined, while quantitative agreement with trained networks improves. Configurations with continuous emergence change only slightly; the finite-amplitude branch shifts toward stronger organization and smaller sample complexity. Appendix D.6 reports the following strong-branch comparison for smooth ReLU with C=10C=10 and K=5K=5:

α\alpha0.200.210.220.24
Four-variable ηcls\eta_{\mathrm{cls}}0.2650.2920.3150.353
Six-variable ηcls\eta_{\mathrm{cls}}0.3590.3740.3890.414

The numerical deviations are reported in Q2 below. These checks support the four-variable theory as a compact description of the two mechanisms, with quantitative refinements documented in the appendix. We have also made the RS scope explicit in the Limitations section.

W2. The solvable setting is highly specialized [...] deeper contrastive models with heterogeneous semantic structure.

The current setting lets us compute and compare representation states. It already reveals continuous emergence, abrupt changes through state competition, and a gap between what the objective favors and what training reaches. Appendix C.5 further tests M≠CM\neq C and class imbalance. Class organization still forms, but its location and growth depend on class structure. The full equilibrium coexistence structure in these settings remains to be established.

More complex semantics means that some classes are more similar than others, and individual classes may contain substructures. These structures may form at different points, so organization could develop in stages. Deeper networks may also change how easily training finds different states. We agree that it is worth asking whether state competition and the gap between objective preference and actual training persist, and how the transitions themselves change.

Correlated class directions, unequal class probabilities, and more general representation dimensions can, in principle, be included through more detailed order parameters. We have not established results for general deep networks or heterogeneous semantics, but the current model offers a starting point that can be extended and tested step by step.

W3. The real-data experiment [...] supports the proposed statistic more directly than the central two-route theory.

We agree. The CIFAR experiment illustrated the meaning of ηcls\eta_{\mathrm{cls}} and its distinction from spectral measures of collapse; it did not directly test the two routes. We have removed it to focus the paper on controlled tests of the emergence mechanisms, their equilibrium description, and the gap with actual training.

W4. [...] a3<0a_3<0 for C=3C=3 despite continuous emergence [...] the conceptual role [...] could be stated more carefully.

The abstract and Section 5 now explicitly present the cubic term as a diagnostic of local asymmetry. We retain the C=3C=3 result to show that a negative cubic coefficient does not rule out continuous emergence of the specialized branch. The route is still determined through branch continuation and RS free-energy comparison.

W5. [...] steer optimization toward the better-organized branch [...] A controlled experiment [...] would strengthen the significance of the theory.

We agree, and thank you for pointing to a concrete way to strengthen the theory's significance. Section 4 now discusses initialization and parameter schedules guided by the branch structure, and how to evaluate strong-branch recovery and downstream performance at matched final objectives, data, and computational budgets. We discuss this further in Q3 below.

AuthorsOpenReview ↗

Rebuttal 2/2

Q1. [...] stability analysis beyond the reduced RS subspace [...] which omitted perturbation modes are most likely to invalidate the branch-selection picture.

Appendix D.6 adds curvature checks for nonuniform RR perturbations and an extended analysis allowing off-diagonal overlaps; see W1 above. The main text also discusses possible RSB and its effects on branch selection, and the Limitations section makes clear that our conclusions remain conditional on RS. If more general partially specialized or glassy states become preferred earlier, the transition picture could change. A full thermodynamic picture needs a dedicated study of possible RSB and its effects on state competition, especially near onset and free-energy crossings, and under weaker regularization or lower InfoNCE temperatures.

Q2. A more quantitative finite-size scaling analysis would help distinguish finite-NN effects from limitations of the asymptotic theory.

Appendix C.7 reports five system sizes, using six data seeds and four Gaussian initializations per dataset at each sampled α\alpha, retaining converged outcomes. Both encoders use C=10C=10 and K=5K=5. The grids cover α=0.06\alpha=0.06–0.170.17 for linear encoders and 0.120.12–0.240.24 for smooth ReLU. The table combines relevant columns from Tables 4 and 5:

NNLinear MAEsReLU α50\alpha_{50}sReLU α90−α10\alpha_{90}-\alpha_{10}sReLU strong-state bias
2,0000.070350.172500.06000+0.09388
5,0000.039140.183330.05900+0.12470
10,0000.027530.203750.04200+0.12703
20,0000.024780.217670.04096+0.11322
30,0000.02067UnresolvedUnresolved+0.11731

sReLU denotes smooth ReLU. αp\alpha_p is the sample complexity at which the empirical frequency of strongly organized outcomes (ηcls>0.25\eta_{\mathrm{cls}}>0.25) reaches p%p\%. These quantities describe recovery under the training protocol, rather than equilibrium critical points. Bias is the mean signed difference between strongly organized outcomes and the preferred four-variable RS reference at matching α\alpha.

Linear MAE falls by 70.6%. For smooth ReLU, the resolved recovery crossovers narrow but shift to larger sample complexities. At N=30,000N=30{,}000, recovery reaches only 30.56% at the largest sampled value, α=0.24\alpha=0.24, so the 50% recovery point and crossover width remain unresolved.

Even among runs that reach strong organization, the smooth-ReLU bias stays around +0.11+0.11 to +0.13+0.13 for N≥5000N\geq5000. We therefore do not attribute all deviations to finite size or failed recovery. Finite-size scatter plots also show these trends.

Appendix D.6 compares the strongly organized outcomes with the off-diagonal extension. Using the stated averaging and rescaling protocol, we obtain:

NNMean signed deviation from the extended reference
10,0000.0491
20,0000.0353
30,0000.0394

The off-diagonal extension improves agreement with simulations while preserving the qualitative routes. We therefore give reproducible ηcls\eta_{\mathrm{cls}} values and simulation comparisons for a representative configuration, rather than new branch-switching plots for every configuration. As discussed in Limitations, these results support the four-variable theory as a minimal description of the two emergence mechanisms.

Q3. [...] This optimization-versus-equilibrium gap deserves more discussion [...] practical contrastive learning follows optimization dynamics.

We agree that better training methods would strengthen the theory's significance. Section 4 now discusses how training can reliably reach the strongly organized state already favored by the objective.

The new finite-size experiments reinforce this question. Over the range studied, strong organization already exists, yet larger systems need higher sample complexity for random starts to reach it at comparable rates. Increasing scale alone therefore does not resolve accessibility; guiding training toward existing good solutions deserves dedicated study.

The revision discusses initialization and temperature schedules guided by the branch structure, and how negative-sample selection affects state preference and training accessibility. This provides a natural starting point for future optimization work: exploring strategies that do not use class information and make strongly organized states easier to reach, with better downstream performance, at matched final objectives, data, and computational budgets.

We hope these additions address your questions. We welcome further discussion and thank you again for your thoughtful feedback.

Reviewer_HN3FOpenReview ↗

Tend to maintain scores

I appreciate the author's response and have carefully reviewed it. I still have doubts about some issues.

  1. Does the author have sufficient evidence that the two emergence mechanisms remain unchanged even after further relaxing symmetry constraints or introducing RSB, given that the six-variable model shifts the phase transition point from 0.175 to 0.134? At present, there are only two fixed perturbation direction tests for other parameters, which are insufficient to establish this point.

  2. Can the author provide at least one theoretically guided initialization or scheduling experiment that does not rely on the true category direction, and demonstrate its improvement in the recovery rate of strong organizational branches under controlled training budget conditions? Even if only completed on the current synthetic model, it is more convincing than adding future research discussions.

Overall, the contribution of the article still lacks clear support for reliable theoretical boundaries. If the author strictly limits the contribution to a solvable model of contrastive learning, the theoretical value of the paper is established; But if we want to use it to explain a wider range of deep contrastive learning behaviors, the existing evidence is still insufficient.

Reviewer_gEphOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how class-informative representations emerge as data scale grows in a high-dimensional InfoNCE model with a single-layer encoder trained on data drawn from a balanced latent Gaussian mixture. Working in the proportional limit (N,P→∞N, P \to \infty at fixed α=P/N\alpha = P/N, with C,K,κ,τ,λw=ΘN(1)C, K, \kappa, \tau, \lambda_w = \Theta_N(1)), the authors perform a replica calculation under a reduced replica-symmetric + class-permutation-symmetric ansatz. This yields a zero-temperature free-energy functional in four scalar order parameters (rd,ro,q,χ)(r_d, r_o, q, \chi) and saddle-point equations (Eq. 11–12). They quantify "class organization" by ηcls\eta_{\text{cls}}, the fraction of representation variance explained by between-class mean differences, and link it to the specialization amplitude Δr=rd−ro\Delta r = r_d - r_o (ηcls=O((Δr)2)\eta_{\text{cls}} = O((\Delta r)^2)).

The main in-theory finding is "two routes" to organization as α\alpha increases: (i) a specialized branch that emerges continuously from the unspecialized branch (linear encoder), or (ii) a finite-amplitude specialized branch that coexists with a locally stable unspecialized branch and only becomes preferred via an RS free-energy crossing (smooth ReLU, larger CC). Notably, the same smooth-ReLU activation exhibits route (i) at C=2C = 2 and route (ii) at C∈{8,10}C \in \{8,10\}, so the route depends on C,K,τC, K, \tau. A local Landau expansion of the profiled free energy identifies a cubic coefficient a3a_3, whose high-τ\tau leading form (Eq. 17) decomposes into activation curvature, class structure ((C−2)/C)\big((C-2)/C\big), and contrastive weighting (K/(K+1))\big(K/(K+1)\big). Validation: finite-NN (N=10000N=10000) training from 50 initializations reproduces (rd,ro,q,ηcls)(r_d, r_o, q, \eta_{\text{cls}}) quantitatively and branch structure qualitatively; a 12-configuration (C,K)(C,K) scan, an (α,τ)(\alpha,\tau) phase map, and a CIFAR-10 linear-probe trend round it out.

Critical strengths
  1. It targets a real, under-explored question and gives a clean, self-consistent theory. Characterizing the emergence route (continuous vs. finite-amplitude) as a function of α\alpha, and showing it switches with CC under the same activation, is a concrete and useful result; the replica derivation is internally consistent and the localization to a four-scalar single-group problem (Eq. 8) is elegant. This is consequential because it turns "how does organization emerge?" into a tractable, testable object.
  2. Honest scoping and breadth of validation for a theory paper. The authors clearly separate state existence, local stability, and thermodynamic preference, and back the theory with a finite-NN scan, a (C,K)(C,K) sweep, an (α,τ)(\alpha,\tau) phase map, and a CIFAR-10 probe trend. This breadth makes the qualitative claims credible even where they are not quantitative.
Critical weaknesses
  1. The "two routes" is a parameter-dependent phenomenon, not a sharp dichotomy, and the headline local signature does not distinguish them. The paper itself shows the same activation gives continuous onset at C=2C=2 and finite-amplitude at C≥8C \ge 8, and that a3a_3 "diagnoses local asymmetry rather than the global emergence route." Consequence: the central framing is looser than the abstract implies, and the a3a_3 result, while neat, is not a decision rule—so the theoretical "signature" contributes less than advertised.
  2. Unproven RS ansatz + highly idealized model limits the strength of the conclusions. Because the ansatz's key quantity (cross-replica QQ) is unobservable and the data model is far from realistic (balanced classes, orthogonal directions, M=CM=C, weak signal), the claims are conditional on assumptions that are plausible but not established. This is the main reason I cannot rate soundness higher.
Questions for authors
  1. (Clarification) The setup requires C≤NC \le N for N−1ΞΞ⊤=ICN^{-1}\Xi\Xi^\top = I_C with mutually orthogonal rows, but this is never stated; it is only implicitly satisfied (C=ΘN(1)C=\Theta_N(1), N→∞N\to\infty; N=10000N=10000 in sims). Could you state the constraint C≤NC \le N explicitly so the model is well-posed as written?
  2. (Clarification) Given a3a_3 does not by itself distinguish the two routes, what is the intended operational use of the cubic coefficient—prediction of the route from a local expansion, or only a diagnostic of local asymmetry? If the former, can you give a configuration where it succeeds without reconstructing the specialized branch?
  3. (Clarification) How sensitive is the finite-amplitude/coexistence result to relaxing the symmetry-matched M=CM=C restriction and to unbalanced classes? Any evidence (even partial) that the route depends on CC for M≠CM \ne C?
  4. (Suggestion, future work) Could the unobservable cross-replica QQ be bounded or inferred from ensembles of independently trained networks, to strengthen validation of the ansatz?
2 comments
AuthorsOpenReview ↗

Rebuttal 1/2

We thank the reviewer for recognizing the value of our question and the breadth of our validation, and for helping us clarify the scope of our conclusions.

W1. The “two routes” is a parameter-dependent phenomenon, not a sharp dichotomy [...] the headline local signature does not distinguish them.

We agree that the two routes depend on the parameter setting. Our main finding is that class organization under InfoNCE can emerge smoothly with more data, or appear abruptly through competition between representation states. High-dimensional equilibrium analysis looks at the overall representations formed by many weights and how the objective's preference among states changes with the parameters. The same activation exhibiting different routes under different settings is therefore part of what we aim to reveal.

We agree that a3a_3 is not a universal rule for identifying the route. The abstract now explicitly describes the routes as “parameter-dependent” and presents the cubic analysis as a complement to the branch analysis. We discuss this further in Q2 below.

W2. Unproven RS ansatz + highly idealized model limits the strength of the conclusions [...] the claims are conditional on assumptions that are plausible but not established.

We understand this concern. RS is indeed a theoretical assumption underlying our derivation. Balanced classes, orthogonal directions, and M=CM=C make the model tractable, while weak-signal scaling keeps the signal and noise contributions comparable before the activation. In principle, class imbalance, correlated directions, and more general representation dimensions can also be described within this framework using more detailed order parameters, at the cost of more complicated equations and analysis.

This simple model already shows rich behavior, including gradual and abrupt class organization in contrastive learning, as well as a gap between equilibrium preference and actual training. We have also added comparisons of networks trained independently on the same dataset to test the cross-replica overlap assumption numerically. Please see our response to Q4.

Q1. Could you state the constraint C≤NC\leq N explicitly [...]?

Thank you for pointing this out. Section 2 now explicitly states 2≤C≤N2\leq C\leq N in the data-model definition, ensuring that the required mutually orthogonal class directions can exist in RN\mathbb{R}^N.

Q2. [...] prediction of the route from a local expansion, or only a diagnostic of local asymmetry?

The latter is the intended role. These coefficients can be evaluated near the unspecialized branch without reconstructing the specialized branch, but a3a_3 alone cannot determine the route. Coexistence and preference still require branch continuation and RS free-energy comparison.

There is also an interesting difference from the continuous transition in the wide supervised ReLU model studied by Afanah and Rosenow [1]. In our model, changing the number of classes can give different routes even with the smooth-ReLU activation held fixed. The local analysis offers a key clue: at leading order in the high-InfoNCE-temperature limit, the activation's nonlinearity produces a cubic response through the comparison of positive and negative similarities. This suggests that the key factor to examine may be the contrastive mechanism of favoring positive pairs over negative pairs, rather than just the particular form of softmax. This motivates a systematic comparison of different contrastive losses. By checking whether the local response and emergence patterns persist when the coupling between positive and negative samples changes, we may find more general rules governing the routes.

AuthorsOpenReview ↗

Rebuttal 2/2

Q3. How sensitive is the finite-amplitude/coexistence result to relaxing [...] and to unbalanced classes? Any evidence [...] for M≠CM\neq C?

We added two sets of experiments in Appendix C.5 to directly examine how class organization changes when these assumptions are relaxed.

Class number still affects emergence at fixed network width (Fig. 11). We fix M=10M=10, vary C∈{2,3,8}C\in\{2,3,8\}, and train from random initializations using three data seeds at N=20,000N=20{,}000. Clear class organization forms in every configuration. However, more classes delay its growth and lower the organization reached over the sampled data range. Even when network width stays fixed, class number itself still affects how organization develops with more data. The class dependence observed in the main text is therefore not limited to M=CM=C.

Class imbalance gradually smooths the initially abrupt rise in organization (Fig. 10). We set the probability ratio between the most and least frequent classes to 2, 5, and 10, using six data seeds at N=10,000N=10{,}000. The transition toward strong organization remains visible, but as imbalance increases, the rise spreads over a wider range of data sizes and the average curve becomes smoother. Class-by-class observations are consistent with frequent classes organizing earlier, so different classes develop their structure at different data sizes. We always evaluate organization with equal class weights. The change therefore reflects a change in the learned representations, rather than greater weight being assigned to frequent classes in the metric.

Q4. Could the unobservable cross-replica overlap be bounded or inferred from ensembles of independently trained networks [...]?

Following your suggestion, we added comparisons between independently trained networks in Appendix C.2 (Table 3). Each configuration uses three data seeds and eight independent training runs per dataset. We first align the networks and then compare outcomes within the same representation state. Linear encoders use orthogonal Procrustes alignment; smooth-ReLU encoders are aligned by permuting encoder units to match the teacher classes.

The aligned cross-network overlap is

Q^ab=1NW~aW~b⊤.\widehat{Q}_{ab}=\frac{1}{N}\widetilde{W}_a\widetilde{W}_b^\top.

The relative error ϵQcross\epsilon_Q^{\mathrm{cross}} measures how close it is to a scalar multiple of the identity: smaller values mean closer agreement with the reduced ansatz. We also report ρ\rho, the similarity between the aligned weight matrices, where values closer to 1 mean greater similarity.

EncoderCCα\alphaϵQcross\epsilon_Q^{\mathrm{cross}}, N=5000N=5000ϵQcross\epsilon_Q^{\mathrm{cross}}, N=20000N=20000ρ\rho, N=20000N=20000
Linear100.050.403±0.0500.403\pm0.0500.185±0.0160.185\pm0.0160.9970
Linear100.500.170±0.0070.170\pm0.0070.205±0.0240.205\pm0.0240.9721
smooth-ReLU100.050.610±0.0060.610\pm0.0060.331±0.0100.331\pm0.0100.9649
smooth-ReLU100.500.0740±0.00340.0740\pm0.00340.0379±0.00360.0379\pm0.00360.9994
smooth-ReLU20.050.200±0.0160.200\pm0.0160.190±0.0010.190\pm0.0010.9987
smooth-ReLU20.100.2616±0.00690.2616\pm0.00690.2622±0.00280.2622\pm0.00280.9995

Uncertainties are standard deviations across data seeds; ρ\rho reports the mean.

The clearest support comes from the strongly organized smooth-ReLU setting with C=10C=10 and α=0.5\alpha=0.5. As NN increases from 5,000 to 20,000, the overlap error falls from 0.0740 to 0.0379, bringing the measured structure closer to the reduced ansatz. Two networks being very similar does not, by itself, mean that their overlap matrix has the assumed scalar structure. This is why we report both measures. These results provide an independent numerical check of the overlap assumption. However, independently trained networks are not direct samples from the Gibbs measure, so we treat this as empirical support rather than a rigorous proof of the cross-replica overlap assumption.

We hope these responses address your concerns and strengthen your confidence in our results. We would be happy to discuss any remaining questions and would appreciate your reassessment of the paper in light of these revisions.

Reference

[1] Assem Afanah and Bernd Rosenow (2026). Continuous Specialization Transition in the Soft Committee Machine with ReLU Activation. arXiv:2603.20010. https://arxiv.org/abs/2603.20010

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revised Manuscript

We have uploaded an updated manuscript incorporating the changes described in our rebuttal, with second-round revisions marked in red and first-round revisions marked in blue.

Red additions

  • Appendix C.8 — Added experiments on label-free, theory-guided recovery of the strongly organized branch.
  • Appendix D.6 — Added local stability tests under coupled matrix perturbations.

Blue revisions

  • Abstract and Sections 1–3 — Clarified the four-variable theory and the two routes to class organization.
  • Section 4 and Appendices C.5–C.7 — Expanded the analysis of class imbalance, representation width, finite-size effects, and branch accessibility with same-class negatives.
  • Section 5 — Clarified the role of the cubic term and how the two emergence routes are identified.
  • Appendix C.2 — Added overlap measurements across independently trained encoders.
  • Section 7 and Appendix D.6 — Extended the theory to six variables and clarified the scope of branch selection under replica symmetry.
AuthorsOpenReview ↗

Rebuttal Summary

We thank all reviewers for their detailed and constructive feedback. We have responded to each of their comments.

Reviewer gEph focused on the role of the cubic coefficient, whether the emergence routes change when the model assumptions are relaxed, and whether the cross-replica overlap assumption is reasonable. We clarified that the cubic coefficient characterizes local asymmetry in the free energy, while identifying the emergence route still requires comparing branches. We also showed that when the representation dimension MM differs from the number of classes CC, the emergence of class organization still depends on CC, and that increasing class imbalance smooths the abrupt rise in average class organization. In addition, we tested the overlap assumption by comparing independently trained networks.

Reviewer HN3F helped us strengthen both the theoretical analysis and the experimental validation. In addition to the interpretation of the cubic coefficient, the reviewer asked how the two emergence routes depend on the model assumptions and how finite system size affects the experimental results. In response, we relaxed some of the symmetry constraints, tested the local stability of both branches at the sampled points, and added quantitative comparisons between theoretical predictions and training outcomes across system sizes. The reviewer’s follow-up question about whether the theory can guide optimization also prompted us to add a label-free experiment testing whether, at a matched computational budget, theory-guided initialization and updates help training reach the branch with stronger class organization. Following the reviewer’s feedback, we also removed the CIFAR-10 experiment because it did not directly test the two emergence routes, bringing the paper’s claims into closer alignment with the evidence.

Reviewer aC4i focused on the clarity of the main results. We added intuitive explanations of why a few variables can describe the collective behavior of many weights, why training may struggle to reach a theoretically preferred state, and why averaging across training runs can obscure an abrupt transition.

All three reviewers recognized the value of the research question. We also recognize, however, that assessing our theoretical contribution requires clarity about which conclusions the evidence supports and under what conditions they hold. We therefore explicitly define the scope of our claims in the revised manuscript. In our solvable model of contrastive learning, the analysis under the replica-symmetry (RS) assumption identifies two possible routes to class organization. Both routes persist in the settings examined after we relax some of the model constraints. A full characterization of equilibrium still requires investigating whether other branches with lower free energy exist and whether relaxing the RS assumption changes the conclusions. It also remains unknown whether the two routes identified in our model can explain the behavior of deep contrastive learning on real data. Developing such a theory will require progressively relaxing the model assumptions and testing its predictions in more realistic settings, and our results provide a testable starting point for that effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.