- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper investigates the phenomenon of linear mode connectivity. LMC is important for understanding the geometry of loss landscapes, training dynamics and for several practical use cases. It studies LMC through the lens of neural scaling limits (µP, completeP), using Tensor Programs for the theory.
The merits of the paper are twofold. First, it shows that scale consistency, previously established for local curvature, extends to a global quantity, namely the loss along the interpolation path. This is a major accomplishment, in particular because the agreement across scales persists even where the local quadratic approximation fails. Second, the paper shows that under completeP the loss barrier stays flat as depth grows, where one would expect it to blow up through variance collapse. This is the paper’s most surprising result.
The biggest downside is the gap between what is analyzed (fixed-depth MLPs with SGD) and what is run experimentally (Transformers with AdamW, scaled in depth). The empirical consistency is cleaner than the theory, which suggests a stronger result is available.
Critical strengths
- Scale consistency shown for a global quantity
- Barriers stay flat with depth under completeP, against the expectations of possible "variance collapse"
- Broad, well-executed experiments
- The C = G − J split is clean and has explanatory power
Critical weaknesses
- The theorems show existence without rate, limiting values etc. This is no "deal breaker", but certainly leaves room for improvements and future work
None of these I consider "critical".
- Theory (fixed-depth MLP, SGD) does not cover the experiments (Transformers, AdamW, depth scaling). Perhaps surprisingly, the epmpirical results are cleaner than the theory.
- The model merging section is a proof of concept rather than evidence: small models, small gains, and TIES transfer across depth fails.
- Thm. 4.2 is less substantial than Thm 4.1. and could also be downgraded to a remark. Specifically G->0 is obvious due to asymptotic linearity.
- More of a proof sketch in the main text would be good.
Questions for authors
Nothing super important. One suggestion: Can you show the µP limit in the setting it is proved for, i.e. small smooth MLP with SGD, and check that C_N converges at the claimed N^{−1/2} rate?
Response to reviewer nGhE
Thank you for the valuable comments and for recognizing the significance and novelty of our work. We are glad you found the result on variance collapse interesting! We will address the weaknesses and questions below.
Indeed, we see the theory more as a basic justification for what we do (i.e., that there actually is a limit), and try to be open about the fact that it has limited explanatory value for the other empirical observations we make (see, e.g., l. 273–276 and l. 1549–1553). Also, the results likely extend to other optimizers and architectures within the NE$\otimes$OR$\top$ framework [1], which we also mention in App. D, but we deliberately choose not to do this to keep it as simple as possible, since we do not think establishing the statement Thms. 4.1 and 4.2 would give significantly more insight or bridge the gap to the empirical observations. In fact, it would be exciting to analyze more formally how this is connected to gradient noise and to where in the Hessian spectrum the interpolation directions lie.
We fully agree, and we would like to note that the merging section is only one of several contributions of the paper, so we are aware that the evidence is still limited, and that there is a lot more to explore in this direction. We also explore just a slice of the merging loss landscape. It would be very interesting to see which HPs depend more on the parameterization and which on the interaction between the experts, what happens to GN and remainder curvature, and to see this scaled up to larger models. Here we are bound by our academic compute, since for a principled comparison one would have to pretrain the models.
We agree that G→0 under NTP is expected from asymptotic linearity, but we still choose to present it as a theorem since this makes it easier to compare the two parameterizations next to each other.
We give a brief sketch for μP in l. 250–253 and the intuition for the NTP result in l. 264–268, but it is hard to include more due to space constraints. The full proofs are in App. D.
Thanks for the suggestion! In App. E, we added a toy experiment in a setting where the theory applies, namely, learning a single-index model with a two-hidden-layer tanh MLP in μP with plain SGD. Similar to the other experiments, the barrier after early branching grows with the LR, is dominated by G, and is consistent across widths (see Fig. 12). Furthermore, in Fig. 13 we plot the initialization fluctuation and bias for the loss and C, J and G against width. The fluctuation roughly tracks the predicted N−1/2, though somewhat more slowly in some cases, and the bias lies below the fluctuation and is roughly N−1, though it is relatively noisy.
[1] Greg Yang, Etai Littwin. Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limit. arXiv:2308.01814, 2023.