- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper is a mostly theoretical paper tackling the question of why neural networks trained in different regimes like different datasets, architectures, or training algorithms still converge to similar representations. The work mostly focuses on theoretical proof in mostly linear student-teacher setups focusing on the CCE metric. The main insight is that in the student-teacher setup CCE is governed by the signal-to-noise ratio.
Critical strengths
- The paper is written very clearly and hits the right balance between presenting mathematically rigorous results but not overloading notation. This earns the presentation points.
- The question asked is very timely and important in times of, e.g., "The Platonic Representation Hypothesis (https://arxiv.org/abs/2405.07987). Their main results on linear networks are interesting.
Critical weaknesses
I think it partly misses the point of the wider question the community has. The authors open with the question of how alignment can be similar for "networks trained with different architectures, training protocols, or training datasets". However, their main model is just about dataset size as well as noise. To me, it is not clear how this extends to the more realistic setup of changing architectures or actually different datasets (that are not just different i.i.d. draws, e.g.). The empirical experiment on CIFAR-10 also does not close this gap. One very surprising result (at least to me) is that a nonlinearity can act as a ridge. However, here no theoretical insight or deeper explanation is offered.
Questions for authors
- Why does adding ReLU act as a ridge at the interpolation threshold?
- In the light of works like this https://arxiv.org/abs/2405.07987, can you give a broader discussion of how your setup connects to the general findings that large models converge to similar representation?
- Can your model be used to analyze what happens in deep linear networks, at least some empirical results?
- What role does overparameterization and superposition play (https://www.anthropic.com/research/superposition-memorization-and-double-descent)?
2 comments
Summary of revision based on Reviewer nQMR review
We have uploaded a revised version of the paper. It contains the following revisions based on your review:
- We added the discussion of the leaky ReLU and weight decay experiment results in Sec. 4.2 (Figs. 10–11).
- We added the two insights related to the Platonic Representation Hypothesis to the conclusion (Sec. 5).
Thanks
We thank the reviewer for the valuable comments and for finding our work timely and interesting. Our responses are below.
1. Why does adding ReLU act as a ridge at the interpolation threshold?
We proposed the implicit-ridge effect of the ReLU as a hypothesis to explain the non-zero alignment at the interpolation threshold, building on the high-dimensional result that the nonlinear component of the activation acts on the linear component as an implicit ridge [1]. Following Reviewer uh36 (comment 2), we tested it directly: we replaced the ReLU with a leaky ReLU and varied its negative slope a from 0 (ReLU) to 1 (linear) at α=1. At a=1, we recover the linear theory: CCE≈0 and the test MSE explodes. As the nonlinearity increases (a→0), the collapse disappears continuously: the CCE rises and the test MSE decreases. The nonlinearity thus acts as a regularizer of tunable strength, suppressing the small-eigenvalue instability behind the collapse. Likewise, explicit weight decay on the ReLU networks raises the CCE at α=1 and lowers the error peak, as in the linear theory. These experiments will be added to the Appendix.
2. In the light of works like [5], can you give a broader discussion of how your setup connects to the general findings that large models converge to similar representations?
The Platonic Representation Hypothesis (PRH) [5] posits that neural networks, regardless of architecture, training data, or modality, converge toward a shared statistical model of reality. This convergence can be driven by the training protocol, the architecture, or the training data; we isolate the role of the data statistics. Although simple, our setting yields two insights:
i. Shared ground truth. The alignment between two models is predicted by the product of each model's cosine similarity with the ground truth, because their dataset-specific components are uncorrelated. Independently trained models thus align because each approximates the same underlying reality. This is consistent with [2], which proves that models trained separately on any number of modalities learn the same representation, up to a single rotation, whenever their paired inputs observe the same scene.
ii. Data quality and quantity. Alignment grows monotonically with data quality and, often non-monotonically, with sample size, providing a theoretical counterpart to the empirically supported "convergence via task generality" argument of [5].
3. Can your model be used to analyze what happens in deep linear networks, at least some empirical results?
At the global minimum we analyze, every hidden layer of a deep linear network is a rescaled or orthogonally rotated version of the others, so the CCE is identical across layers. We therefore did not study depth empirically. We expect depth to matter in the learning dynamics, i.e., in how representations develop during training, which remains an open question, as stated in the paper.
4. What role does overparameterization and superposition play [4]?
Overparameterization: In our theory, overparameterization limits alignment precisely. Below the threshold (α<1), the gradient-descent solution lies in the span of the network's training inputs, so two networks trained on independent data share only the overlap of their data spans, ETr(PaPb)=n2/d, with Pa,Pb the corresponding projectors. The correlation is ρ∗=αSNR/(SNR+1−α1), which vanishes as α→0. Here the data underdetermine the solution, so the implicit bias of training selects it; this explains why a shared initialization increases alignment at small n in the CIFAR-10 experiments.
Superposition: Our linear theory cannot capture superposition, since linear networks represent at most one feature per hidden dimension [3]. Still, [4] suggests a bridge: in their toy ReLU models, small datasets lead to storing data points in superposition (memorization) and large datasets to storing features (generalization), with the double-descent peak at the transition. Our linear analysis has the same structure without superposition: below the threshold, the learned direction lies in the span of the network's own training inputs and is dataset-specific; above it, the solution converges to the teacher's features, which are shared across datasets. Representations storing data points should thus not align across networks trained on disjoint data, whereas those storing features should, so the alignment minimum would signal the transition from memorization to generalization. This hypothesis can be tested in the direction of pushing the theory beyond the linear case.
[1] https://arxiv.org/abs/1908.05355
[2] https://arxiv.org/abs/2610.39302
[3] https://transformer-circuits.pub/2022/toy_model/index.html
[4] https://www.anthropic.com/research/superposition-memorization-and-double-descent
[5] https://arxiv.org/abs/2405.07987