- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates how class-organized representations emerge in contrastive learning as the sample complexity increases. The authors study a high-dimensional Gaussian mixture model with a single-layer encoder trained using InfoNCE. Using the replica method under replica-symmetric and class-permutation-symmetric assumptions, they derive a reduced free-energy formulation characterized by four scalar order parameters.
The main finding is that class organization can emerge through two distinct routes: (1) a continuous transition, where class specialization develops gradually as the sample complexity increases, and (2) a discontinuous transition, where a specialized solution coexists with an unspecialized solution and eventually becomes preferred through a free-energy crossing. The authors show that the transition behavior depends on the activation function and the number of classes.
To further characterize these transitions, the authors perform a Landau-type expansion of the free energy and derive a high-temperature expression for the cubic coefficient, relating it to activation curvature, class structure, and contrastive weighting. The theoretical predictions are compared with finite-dimensional simulations, which reproduce the predicted representation statistics and qualitatively support the two transition behaviors.
Critical strengths
-
Interesting research question. The paper studies how class organization emerges in contrastive learning as the sample complexity increases. Rather than focusing only on the properties of learned representations, it investigates the emergence of these properties and identifies two qualitatively different transition behaviors. I find this an interesting perspective for understanding representation learning.
-
Connection between theoretical analysis and numerical experiments. The paper develops a theoretical framework based on the replica method and uses it to characterize different stationary branches and their free-energy competition. The authors also conduct finite-dimensional experiments that qualitatively reproduce the predicted transition behaviors. This provides empirical support for the theoretical picture within the considered model.
Critical weaknesses
-
The paper could be presented more clearly. It relies heavily on specialized terminology and dense mathematical derivations, making the main arguments difficult to follow. In particular, the theorems are not stated or interpreted in an accessible manner. The authors should provide more intuition, better motivate the assumptions, and explain the implications of the theoretical results more clearly.
-
The theoretical analysis focuses primarily on between-class variance while largely overlooking within-class variance. Both are important for understanding class separation and the behavior of InfoNCE. An analysis that accounts for within-class variance would therefore provide a more complete and informative characterization of the objective.
Questions for authors
Could the authors provide an intuitive explanation of the main theoretical results? In particular, what does each central theorem reveal about InfoNCE, why are its assumptions necessary, and what insight does it provide beyond the formal derivation? A clearer explanation of these points would help me assess the significance and generality of the results. Does the analysis extend to within-class variance? If not, could the authors explain why focusing only on between-class variance is sufficient, or clarify how accounting for within-class variance might affect the conclusions?
3 comments
Rebuttal 2/2
What insight does it provide beyond the formal derivation?
Training may fail to learn clear class structure even when the data and objective already support it. In our model, the objective may favor a strongly organized state while training remains in a weakly organized one.
An abrupt equilibrium transition can also be hidden by a smooth average training curve. Different runs may reach strong organization at different dataset sizes. After averaging, this can look like gradual improvement. The theory helps distinguish two explanations: individual representations may be improving gradually, or more runs may be finding a different state that is already well organized.
Why are its assumptions necessary [...]? A clearer explanation [...] would help me assess the significance and generality.
These assumptions mainly make the problem manageable enough to solve; they do not mean that the phenomena can only occur under these conditions. Balanced classes, orthogonal directions, and matching the representation dimension to the number of classes give each class the same learning conditions. Weak-signal scaling keeps the signal and noise contributions comparable before the activation as dimension grows. A single-layer network and a simplified description of relationships between solutions then let us solve for and compare different representation states.
We retain the simple setting because it already reveals behavior worth understanding. Class organization can develop gradually or appear abruptly, and training may fail to find the state that the objective already favors. Building a coherent, testable theoretical picture of these phenomena is the main value of our work.
Class imbalance, correlated directions, and more general representation dimensions can, in principle, be included through more detailed order parameters, at the cost of more complicated equations and analysis. We have not established results for general deep networks or heterogeneous semantics, but the current model provides a testable starting point. By gradually adding more complex structures, we can then ask which phenomena survive and which changes reflect new mechanisms.
Does the analysis extend to within-class variance [...]?
Within-class variation is already part of the analysis. How clearly classes are separated depends both on how far apart their centers are and on how widely samples spread within each class. If class centers move apart while samples also spread out within each class, class structure need not become clearer. Our organization measure accounts for both effects. Its definition and the theoretical expression for within-class variance are given in Section 2 and Appendix B.2:
The numerator measures between-class variation, while the denominator includes both between-class and within-class variation. Both depend on the learned representation and are predicted by the theory. Within-class variation is neither fixed nor ignored. Although the within-class input noise is specified by the data model, the within-class variance of the learned representations is not fixed. The gradual or abrupt emergence we describe therefore concerns how clearly class structure develops relative to the spread within classes, rather than just how far the class centers move apart.
We hope these explanations make our main findings and their significance clearer, and clarify how within-class variation is included in the analysis. If these responses address your main concerns, we would appreciate your reassessment of the paper in light of the revisions. We would be happy to discuss any remaining questions.
Thank you for your rebuttal. Unfortunately, the technical content of this work is beyond my current expertise, and I do not feel sufficiently qualified to assess the mathematical arguments rigorously. Therefore, I will maintain my original score, but I kindly ask the Area Chair (AC) to assign less weight to my evaluation when making the final decision.
Rebuttal 1/2
We thank the reviewer for their constructive feedback and suggestions for improving the clarity and presentation of our work.
W1. Following your feedback, we have used the additional space to explain the intuition behind the results, the role of the assumptions, and their implications. We also want the final paper to be easier for the community to follow. Below, we focus on what the results reveal about InfoNCE and why they matter.
W2. Our original analysis already includes both between-class differences and within-class variation. Both enter our measure of class organization, as explained in Part 2.
Our main finding is that, in the InfoNCE model we study, class organization need not develop gradually with more data. It can appear abruptly, while training may fail to find the strongly organized state that the objective already favors. Our main technical result turns the high-dimensional weight problem into a set of solvable equations, predicting the minimum regularized InfoNCE loss and the corresponding representation statistics.
Two distinctions help explain this finding: which state the objective favors versus which state training can reach, and gradual versus abrupt class organization.
Equilibrium analysis asks what representation corresponds to the global minimum of the loss. Training starts from an initialization and updates the weights step by step; it may remain near a higher-loss local minimum. A better representation can therefore exist without training finding it. Our theory predicts which states the objective favors, while experiments check which states training actually reaches.
Gradual emergence means that class structure develops little by little from the existing state. In abrupt emergence, weakly and strongly organized states coexist, and more data change which one the objective favors. Yet the weakly organized state can persist. The preferred representation changes, but the trained representation need not follow.
The experiments also show this gap. As input dimension increases from 2,000 to 20,000, the sample-to-dimension ratio needed for half of the random initializations to reach strong organization rises from about 0.17 to 0.22 (Appendix C.7). With same-class negatives allowed, random initialization does not reach the strongly organized branch over the tested range, whereas initialization aligned with the class directions can (Appendix C.6). In these settings, increasing scale does not automatically resolve the optimization difficulty.
Result 3.2 asks what representations InfoNCE supports at a given amount of data, and which ones the objective favors. To our knowledge, this is the first characterization of regularized InfoNCE empirical risk minimization in the model studied here through a closed set of equations in the high-dimensional proportional limit. It lets us ask not only whether class information can be recovered, but how class structure forms as data increase.
Result 3.3 asks whether a small change toward distinguishing classes would be favorable near a state without class organization. Together with the comparison between states, it explains how a strongly organized state can already be preferred while a small move toward class separation from the weakly organized state need not be favorable. This is a local test; whether training actually leaves that state is examined in our experiments.
In the supervised models discussed in our paper, ReLU units gradually develop distinct roles. In our InfoNCE model, keeping the same smooth-ReLU activation and changing the number of classes can instead change emergence from gradual to abrupt. Our local analysis offers a further clue: comparing positive and negative samples interacts with the activation's nonlinearity to produce local asymmetry. This is not, by itself, a rule for predicting the emergence route. A natural next step is to compare different contrastive losses, separating effects shared by positive-negative comparison from those specific to structures such as softmax, and looking for more general rules.