acceptodds
Under review as a conference paper at ICLR 2027

How Does Loss Landscape Geometry Transfer Across Scales?

Abstract

Neural scaling limits, such as the maximal update parameterization (P), have become a powerful tool for zero-shot hyperparameter transfer as model size grows. But which aspects of the loss landscape itself exhibit analogous transfer across scales? We study this question through the lens of *linear mode connectivity* (LMC), which asks whether models are connected in weight space by a linear path of low loss. We find evidence that interpolation barriers approach scale-consistent behavior, can already be stable at practically relevant widths and depths, and that such transfer can persist over long training horizons, beyond the regime of local quadratic approximation. Experiments with convolutional networks and transformers for vision and language show consistent behavior across width and depth under a range of training settings, and demonstrate that this consistency extends to practical weight-space operations such as model merging. As a theoretical foundation for our observations, we establish a scaling limit for loss barriers in a simplified setting. Our analysis offers a unifying perspective on neural scaling theory and weight space geometry, laying a foundation for understanding when interpolation, weight averaging, and model merging behave consistently across scales.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
loss landscape, mode connectivity, scaling limits, model merging

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_nGhEOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates the phenomenon of linear mode connectivity. LMC is important for understanding the geometry of loss landscapes, training dynamics and for several practical use cases. It studies LMC through the lens of neural scaling limits (µP, completeP), using Tensor Programs for the theory.

The merits of the paper are twofold. First, it shows that scale consistency, previously established for local curvature, extends to a global quantity, namely the loss along the interpolation path. This is a major accomplishment, in particular because the agreement across scales persists even where the local quadratic approximation fails. Second, the paper shows that under completeP the loss barrier stays flat as depth grows, where one would expect it to blow up through variance collapse. This is the paper’s most surprising result.

The biggest downside is the gap between what is analyzed (fixed-depth MLPs with SGD) and what is run experimentally (Transformers with AdamW, scaled in depth). The empirical consistency is cleaner than the theory, which suggests a stronger result is available.

Critical strengths
  • Scale consistency shown for a global quantity
  • Barriers stay flat with depth under completeP, against the expectations of possible "variance collapse"
  • Broad, well-executed experiments
  • The C = G − J split is clean and has explanatory power
Critical weaknesses
  • The theorems show existence without rate, limiting values etc. This is no "deal breaker", but certainly leaves room for improvements and future work

None of these I consider "critical".

  • Theory (fixed-depth MLP, SGD) does not cover the experiments (Transformers, AdamW, depth scaling). Perhaps surprisingly, the epmpirical results are cleaner than the theory.
  • The model merging section is a proof of concept rather than evidence: small models, small gains, and TIES transfer across depth fails.
  • Thm. 4.2 is less substantial than Thm 4.1. and could also be downgraded to a remark. Specifically G->0 is obvious due to asymptotic linearity.
  • More of a proof sketch in the main text would be good.
Questions for authors

Nothing super important. One suggestion: Can you show the µP limit in the setting it is proved for, i.e. small smooth MLP with SGD, and check that C_N converges at the claimed N^{−1/2} rate?

1 comment
AuthorsOpenReview ↗

Response to reviewer nGhE

Thank you for the valuable comments and for recognizing the significance and novelty of our work. We are glad you found the result on variance collapse interesting! We will address the weaknesses and questions below.

W1/W2: The theorems show existence without rate, limiting values etc. [...] Theory (fixed-depth MLP, SGD) does not cover the experiments (Transformers, AdamW, depth scaling). Perhaps surprisingly, the empirical results are cleaner than the theory.

Indeed, we see the theory more as a basic justification for what we do (i.e., that there actually is a limit), and try to be open about the fact that it has limited explanatory value for the other empirical observations we make (see, e.g., l. 273–276 and l. 1549–1553). Also, the results likely extend to other optimizers and architectures within the NE$\otimes$OR$\top$ framework [1], which we also mention in App. D, but we deliberately choose not to do this to keep it as simple as possible, since we do not think establishing the statement Thms. 4.1 and 4.2 would give significantly more insight or bridge the gap to the empirical observations. In fact, it would be exciting to analyze more formally how this is connected to gradient noise and to where in the Hessian spectrum the interpolation directions lie.

W3: The model merging section is a proof of concept rather than evidence: small models, small gains, and TIES transfer across depth fails.

We fully agree, and we would like to note that the merging section is only one of several contributions of the paper, so we are aware that the evidence is still limited, and that there is a lot more to explore in this direction. We also explore just a slice of the merging loss landscape. It would be very interesting to see which HPs depend more on the parameterization and which on the interaction between the experts, what happens to GN and remainder curvature, and to see this scaled up to larger models. Here we are bound by our academic compute, since for a principled comparison one would have to pretrain the models.

W4: Thm. 4.2 is less substantial than Thm. 4.1 and could also be downgraded to a remark. Specifically G→0G \to 0 is obvious due to asymptotic linearity.

We agree that G→0G \to 0 under NTP is expected from asymptotic linearity, but we still choose to present it as a theorem since this makes it easier to compare the two parameterizations next to each other.

W5: More of a proof sketch in the main text would be good.

We give a brief sketch for μ\muP in l. 250–253 and the intuition for the NTP result in l. 264–268, but it is hard to include more due to space constraints. The full proofs are in App. D.

Q1: Can you show the μ\muP limit in the setting it is proved for, i.e.\ small smooth MLP with SGD, and check that CNC_N converges at the claimed N−1/2N^{-1/2} rate?

Thanks for the suggestion! In App. E, we added a toy experiment in a setting where the theory applies, namely, learning a single-index model with a two-hidden-layer tanh MLP in μ\muP with plain SGD. Similar to the other experiments, the barrier after early branching grows with the LR, is dominated by GG, and is consistent across widths (see Fig. 12). Furthermore, in Fig. 13 we plot the initialization fluctuation and bias for the loss and CC, JJ and GG against width. The fluctuation roughly tracks the predicted N−1/2N^{-1/2}, though somewhat more slowly in some cases, and the bias lies below the fluctuation and is roughly N−1N^{-1}, though it is relatively noisy.

[1] Greg Yang, Etai Littwin. Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limit. arXiv:2308.01814, 2023.

Reviewer_QnLeOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper makes an interesting connection between scaling limits and linear mode connectivity. It asks how the interpolation behavior between branched models relate across models of different sizes, when they are trained with different parameterizations. The paper observes that under mup/completeP, the interpolation losses (and their decompositions) have consistent behaviour across widths/depth. The authors explain this theoretically (for the width case, in fixed training time) by proving that these quantities converge to non-trivial limit. The authors also did substansive empirical exploration to understand how this consistency depends on the decompositions to the G and J component, how it depends on branching and training time, and how quadratic approximation helps to explain this consistency. Finally, the author also consider model merging, and shows practical implication: HP transfer can happen for model merging.

Critical strengths
  • The authors raise an interesting, under-explored research question regarding the connection of loss landscape across models of different sizes, and approach it from the perspective of LMC. It is well-motivated, and accompanied with in-depth exploration.
  • The empirical studies have a clear set of research questions to understand, and carefully design the controlled experiments to explore them. The experiment results clearly shows the stated pattern, which shows that it unveil something conceptual. The claims are carefully stated and supported with strong empirical evidence.
  • The paper not only tries to thereotically understand LMC, but also explores practical implications on model merging. The premises of hyperparameter transfer for model merging is interesting.
Critical weaknesses

My main concern is the conceptual significance of the empirical findings to the LMC community

  • The main observation seems to be that C, and its decomposition J,G are consistent across scales under the correct parameterization. This is not surprising to people familiar with the scaling-limit literature: the point of muP/completeP is that under these carefully designed parameterization, models of different sizes have very similar behavior because they are finite approximations of the same, non-trivial limit. They are "similar" in the sense that nice scalar summaries are similar, and in this case C,J,G are those summaries. Even though the authors claim that theory does not say anything about whether C_N is actually close to the limit, and whether such behavior persist for long training time, these are usually expected to hold for various scalar quantities because of existing empirical evidence, so it shouldn't be surprising to hold in the LMC case.
  • The authors go beyond this and have multiple finer empirical findings regarding whether finite-error accumulates during training, whether the C is dominated by G or J, whether quadratic approximation is faithful etc. But I'm not sure whether I understand the conceptual significance of these observations. Why is it important to understand these in the first place, and does this give us any interesting insights for model merging? Maybe this is because of my lack of expertise in LMC, and I hope the authors can clarify. I'm happy to adjust my overall assessment if the authors can make a point on the sigificance of the findings.

limited theoretical contribution

  • the theory is limited to the use of tensor program to establish the limit, which only explains one of the the observations, for the width case. The depth case is not treated, eventhough there are also tools to understand the infinite-depth behavior.
  • there is also a lack of theoretical explanation for other empirical observations in the paper.

minor points about writing and presentation

  • Some parts of the introduction are confusing in the first read. "Variance collapse", "REPAIR" and "superconsistency" are mentioned in the introduction without properly giving the necessary background to understand what they are.
  • Fig 8 is unclear: it is quite hard to distinguish the different line types (solid - dashed - dotted) especially since the plot size is small. I'd fix that since distinguishing the line types is particularly important for conveying the plots' main message.
  • The paper mentioned in different parts that hints some connection and comparison with Noci et al., but it is scattered, and what Noci et al does exactly is not explained. So to me, the connection is not conveyed very effectively.
Questions for authors

Below are all clarification questions for the authors

  • I am not very familiar with the LMC literature, but I thought the case where C<0C<0 or CC is small is the more interesting case, as it matters most for model merging etc. However, the paper seems to pay equal attention to the case C<0, C small and C big, why is that?
  • Related to the above, Theorem 4.2 and Fig 1 shows that, under NTP (and also i suppose similar behavior arise for other lazy-training regime), G→0G\to 0. Assuming loss is convex, one should have limiting J∞≥0J_{\infty}\geq 0 , so that the C→−J∞≤0C\to -J_{\infty} \leq 0. So for sufficiently large width, there is no barrier: the midpoint loss is at most the average of the endpoint losses. Should I interpret this as lazy regime is particularly good for model merging? (even though I can see that the endpoint losses are typically larger for lazy-training as compared to feature-learning) Does this near-linearity of the weight-to-function map also explain why late branching leads to small interpolation loss?
  • As mentioned in the weakness part, I'm confused about the connection and comparison with Noci et al. Is the following understanding correct: Noci et al focuses on the Hessian, and does similar decomposition as eq(5). The quadratic approximation given in eq(6) is exactly written in terms of the Hessian and its decomposition. Hence, the super-consistency of Hessian directly implies super-consistency of CC, if quadratic approximation holds. So it is important to test the accuracy of quadratic approximation, thus question (3)?
  • I found it hard to understand finding 3(b)'s implication. Noci et al says sharpness is increasingly GN-dominated over training. What's the meaning or implication of that? Is GN-domination a separate matter from the distance vs curvature contribution that you discussed? Also do I understand correctly that 3(b) is saying that GN- domination depends on branching time b?
  • If I understand correctly, the theorem 4.1 and 4.2 might be extended from the LMC setting the model merging setting (specifically for task arithmetic case where the merging is a linear combination of the models), since you can train the branched models with different datasets and tasks, and still write the whole thing with a tensor program? That, if holds, should directly justify the hyperparameter transfer for TA model merging.
5 comments
AuthorsOpenReview ↗

Response to reviewer QnLe (1/3)

Thank you very much for the constructive and thoughtful review, and for recognizing our experiments as substantive and carefully designed. We are really glad you find the research question well-motivated, interesting and under-explored! We address the raised weaknesses and questions below.

Re W1 (conceptual significance):

The main observation seems to be that CC, and its decomposition JJ, GG are consistent across scales under the correct parameterization. This is not surprising to people familiar with the scaling-limit literature [...]

We agree that the existence of a scaling limit alone might not be surprising. However, the existing evidence does not suggest that all such scalar summaries converge at the same rate or agree at practical sizes: For example, Noci et al. [1] find that this holds only for the largest Hessian eigenvalues and not further into the bulk, while the loss still improves with size. Moreover, to the best of our knowledge, the evidence also mostly concerns scalar summaries of a single training run, e.g., Vyas et al. [2] compare predictions under the same data order at every width, and still find that finite-width effects can grow over training. In contrast, we train two copies of the same model with different data orders, so these results do not directly apply. CC is caused entirely by this difference (with the same data order, both copies would coincide and C=0C = 0), and it is evaluated at a point in weight space that neither of the two runs visits during training. Early in training, this can be quite brittle: which linearly connected region a run ends up in is only determined after an initial phase [3, 4], during which e.g. perturbing even a single weight can cause large barriers [5]. Since barriers can result from such small differences being amplified, we argue that it is not clear a priori that this behaves similarly at every size. We therefore did not trivially expect barriers to agree this well across a 16$\times$ range in width/depth, for vastly different LRs and branching times and in transformers of up to 500M parameters trained from scratch.

We would further like to note that for depth, the LMC/merging literature even suggests a different expectation: variance collapse has been identified as a key obstacle to interpolating deep networks, and has been argued and observed to compound with depth [6]. Our results challenge the view that this is intrinsic to depth, as under completeP, variance collapse can still be present but is roughly stable across depths, and barriers do not grow (whereas they do grow under SP, see App. G.2).

[...] I'm not sure whether I understand the conceptual significance of these observations. Why is it important to understand these in the first place, and does this give us any interesting insights for model merging?

Thank you for bringing this up. With these findings, we try to understand which mechanism is responsible for a barrier. Whether two models can be merged is decided by the two opposing effects JJ (output diversity) and GG (nonlinearity). The 3 findings you mention address the following questions: (i) Which of the two decides whether a barrier appears? The dependence of CC on branching time comes mostly from GG, while JJ depends little on it, and we find that this dependence transfers across scales (Finding 1 and 2a). (ii) How reliably do small models predict the behavior of larger ones over training? JJ fluctuates little across initializations, while for GG after early branching, the fluctuation grows together with GG (Finding 2b). The prediction of GG from a small model can thus be improved by averaging over seeds rather than by increasing the model size. (iii) Does the agreement already follow from known results? We argue no, as it holds even where the quadratic approximation fails (Finding 3a), so it does not follow from the super consistency of the Hessian [1] alone.

For model merging, this implies that whether models can be merged depends on the nonlinearity GG along the merging direction, which stays roughly the same as the model grows under feature-learning scalings, so larger models are thus not automatically easier to merge. Further, by (i), sharing more of the training before branching reduces the cost of merging (GG) much more than its gain (JJ). Since both increase with the optimizer noise, but at different rates, this also suggests an explanation for why merging success depends non-monotonically on the noise scale [7] (we discuss this also in App. A). Finally, we demonstrate that merging effects can be reasonably transferred across scales (Fig. 11).

AuthorsOpenReview ↗

Response to reviewer QnLe (2/3)

Re W2 (limited theoretical contribution):

the theory is limited to the use of tensor program to establish the limit, which only explains one of the observations, for the width case. The depth case is not treated [...] there is also a lack of theoretical explanation for other empirical observations in the paper.

We use the theory to establish that the objects we look at are well-defined in the limit, and try to be upfront about the fact that it has limited explanatory value for the other empirical observations we make (see, e.g., l. 273–276 and l. 1549–1553). We chose not to treat the most general case, e.g., other architectures and optimizers (which we mention in App. D), since even where this is possible, we are not sure it would give much more insight: as discussed for W1, as how well/persistently finite models agree is not explained by it in either case. It would be exciting to analyze in future work more formally how this is connected to gradient noise and to where in the Hessian spectrum the interpolation directions lie.

Re W3 (writing and presentation):

"Variance collapse", "REPAIR" and "superconsistency" are mentioned in the introduction without properly giving the necessary background to understand what they are.

Thank you for the suggestion. We agree and have updated the introduction to properly explain these terms when we introduce them (l. 66–74).

Fig 8 is unclear: it is quite hard to distinguish the different line types (solid - dashed - dotted) especially since the plot size is small.

In what is now Fig. 11, we use different colors with solid lines only and shows fewer curves. We hope this is now easier to distinguish.

The paper mentioned in different parts that hints some connection and comparison with Noci et al., but it is scattered, and what Noci et al does exactly is not explained.

We now explain this in the introduction (l. 72–79): the super consistency of the largest Hessian eigenvalues [1] could explain consistent barriers only where the loss along the path is locally quadratic and the interpolation direction lies sufficiently high in the spectrum, whereas we find that barriers remain consistent beyond this regime.

Re questions:

Q1: I thought the case where C<0C<0 or CC is small is the more interesting case, as it matters most for model merging etc. However, the paper seems to pay equal attention to the case C<0C<0, CC small and CC big, why is that?

Indeed, C≤0C \leq 0 is the case one wants for model merging. However, to know when this is the case, one needs to understand both sides, as the sign change of CC is exactly the boundary of the window in which averaging is safe. Apart from this, while arguably less important in practice, positive barriers have themselves received significant attention in the LMC literature, as their height quantifies how connected the landscape is. For example, [8] describe models as separated by ridges of varying heights, from a small bump after late branching to a wide ridge after early branching, and ask whether the extent of the final barrier can be predicted early. Note that in Fig. 3, positive barriers span about two orders of magnitude across LRs and branching times, i.e., the averaged model ranges from slightly worse than its endpoints to entirely destroyed.

Q2: [...] Should I interpret this as lazy regime is particularly good for model merging? [...] Does this near-linearity of the weight-to-function map also explain why late branching leads to small interpolation loss?

Yes to both. Since G→0G \to 0 under NTP, the weight average behaves like an ensemble of outputs and C≤0C \leq 0. The lazy regime is thus good for model merging, although of course not desirable for pretraining. Also note that linearity is only needed along the merging direction, this is also what happens after late branching. Indeed, there is some evidence that fine-tuning can be close to this regime, e.g., it has been found that between models fine-tuned from the same checkpoint, the map from weights to features is approximately linear [9] or that linearizing fine-tuning can improve task arithmetic further [10].

AuthorsOpenReview ↗

Response to reviewer QnLe (3/3)

Q3: Is the following understanding correct: [...] the super-consistency of Hessian directly implies super-consistency of CC, if quadratic approximation holds. So it is important to test the accuracy of quadratic approximation, thus question (3)?

Yes, indeed for the argument of [1] to imply consistency of CC, one would need, besides the quadratic approximation, that the interpolation direction lies sufficiently high in the spectrum (l. 74–77), and that the distance between the branches is itself consistent across sizes. This is exactly why we test the quadratic approximation (Finding 3a): it fails for larger LRs and earlier branching and accumulates more finite-size effects than CC itself. Also note that in what is now Fig. 6, the positive interpolation curves are also very visibly non-quadratic (e.g., they have inflection points), yet agree across sizes.

Q4: I found it hard to understand finding 3(b)'s implication. [...] Is GN-domination a separate matter from the distance vs curvature contribution that you discussed? Also do I understand correctly that 3(b) is saying that GN-domination depends on branching time bb?

Yes to both. Within the quadratic approximation, JJ and GG share the same distance factor and differ only in the curvature along the interpolation direction (GN for JJ, remainder for GG). Finding 3b shows that the remainder curvature becomes much weaker for later bb, while the GN curvature is comparatively insensitive to it (Fig. 8), so GN-domination depends on the branching time.

Q5: If I understand correctly, the theorem 4.1 and 4.2 might be extended from the LMC setting to the model merging setting (specifically for task arithmetic case where the merging is a linear combination of the models) [...]?

Thanks for the suggestion, this is indeed true. We have added a remark after the formal statement of the μ\muP limit (l. 1547–1549).

[1] Lorenzo Noci, Alexandru Meterez, Thomas Hofmann, Antonio Orvieto. Super Consistency of Neural Network Landscapes and Learning Rate Transfer. NeurIPS 2024.

[2] Nikhil Vyas, Alexander Atanasov, Blake Bordelon, Depen Morwani, Sabarish Sainathan, Cengiz Pehlevan. Feature-Learning Networks Are Consistent Across Widths At Realistic Scales. NeurIPS 2023.

[3] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, Michael Carbin. Linear Mode Connectivity and the Lottery Ticket Hypothesis. ICML 2020.

[4] Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M. Roy, Surya Ganguli. Deep Learning versus Kernel Learning: An Empirical Study of Loss Landscape Geometry and the Time Evolution of the Neural Tangent Kernel. NeurIPS 2020.

[5] Gül Sena Altıntaş, Devin Kwok, Colin Raffel, David Rolnick. The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions. ICML 2025.

[6] Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, Behnam Neyshabur. REPAIR: REnormalizing Permuted Activations for Interpolation Repair. ICLR 2023.

[7] Chenxiang Zhang, Alexander Theus, Damien Teney, Antonio Orvieto, Jun Pang, Sjouke Mauw. How Does the Optimizer Implicitly Bias the Model Merging Loss Landscape? ICLR 2026.

[8] Sidak Pal Singh, Linara Adilova, Michael Kamp, Asja Fischer, Bernhard Schölkopf, Thomas Hofmann. Landscaping Linear Mode Connectivity. ICML Workshop on High-Dimensional Learning Dynamics 2024.

[9] Zhanpeng Zhou, Zijun Chen, Yilan Chen, Bo Zhang, Junchi Yan. On the Emergence of Cross-Task Linearity in Pretraining-Finetuning Paradigm. ICML 2024.

[10] Guillermo Ortiz-Jimenez, Alessandro Favero, Pascal Frossard. Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models. NeurIPS 2023.

Reviewer_Cg8FOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies linear mode connectivity (LMC) between models that branch from a shared checkpoint and train on different minibatches. It asks whether loss barriers remain consistent across width and depth under µP, depth-µP, and completeP.

The midpoint chord deviation is decomposed exactly as C=G−JC = G - J. Here, J≥0J \geq 0 is the loss reduction from averaging the endpoint logits, while GG is the additional loss incurred by averaging weights instead of logits. Locally, these terms correspond to the Gauss–Newton and non-Gauss–Newton contributions to curvature along θB−θA\theta_B-\theta_A.

By representing both training branches and the interpolated networks in one tensor program, the authors establish an infinite-width limit for the barrier curve under µP (Theorem 4.1). Under NTP, they show that G→0G \to 0, giving C→−J≤0C \to -J \leq 0 (Theorem 4.2).

The main experiments use GPT-2-style models with 8M–500M parameters, trained on 300M tokens under completeP, with additional vision experiments. At fixed learning rate and branching time, CC, JJ, and GG are approximately consistent across model sizes. The dependence on branching time comes mainly from GG, and the agreement extends beyond the regime where a quadratic approximation is accurate. The paper also studies variance collapse with depth and transfers merging hyperparameters for task arithmetic, Squisher, and TIES from smaller to larger models.

Critical strengths

1. The paper identifies parameterization as an important control in comparisons of loss landscapes across model sizes.

Changing width can also change update scales, signal propagation, and the extent of feature learning. Figure 1 demonstrates the consequences: barriers increase with width under SP, approach zero or negative values under NTP, and approach a positive value under µP. This suggests that some reported improvements in connectivity with width may reflect increasingly lazy training. Theorem 4.2 supports this interpretation by showing that GG vanishes under NTP.

2. The decomposition helps interpret the empirical results.

The identity C=G−JC = G - J separates a quantity determined by endpoint predictions from the additional effect of interpolating in weight space. Its connection to the Hessian gives the terms a local curvature interpretation.

The depth experiments also provide a useful result. With appropriate residual scaling, variance collapse remains bounded rather than compounding unchecked across blocks, consistent with (1−ε/L)L→e−ε(1-\varepsilon/L)^L \to e^{-\varepsilon}. In the BatchNorm ResNet experiment, the midpoint barrier stays small under K−1K^{-1} scaling but grows to nearly 60 percentage points without it. This provides evidence that parameterization can prevent the problem addressed by post-hoc methods such as REPAIR.

Critical weaknesses

1. The theory establishes a limit, but does not explain the finite-width agreement or characterize the limiting geometry.

The distinction between JJ and GG matters for the paper’s central claim. JJ depends only on endpoint predictions and does not evaluate the interpolated network. Its agreement across sizes is therefore a function-space result, related to the prediction consistency reported by Vyas et al. (2023). GG captures the additional effect of weight interpolation. Yet Finding 2b shows that finite-size discrepancies in GG accumulate during training, while discrepancies in JJ often decrease. The evidence for transferring weight-space geometry is therefore less clear than the aggregate barrier plots suggest.

The tensor-program analysis does not resolve this issue. It establishes convergence of the interpolated forward passes, including uniform convergence along the interpolation path. However, it does not characterize the sign or magnitude of C∞C_\infty, or explain when it vanishes as branching time bb increases. Nor does the convergence rate explain why barriers agree closely at the widths studied. The loss itself has the same N−1/2N^{-1/2} convergence rate, despite differing by approximately 0.75 nats between N=128N=128 and N=2048N=2048 in Figure 2.

A possible explanation is that the shared initialization and training prefix produce similar finite-width corrections in the endpoint and midpoint losses. These corrections could largely cancel when computing CC or GG. Sections 4.2 and D.5 discuss additive versus relative barriers mainly as a normalization choice, but this also raises a question about the interpretation of the observed agreement. Such cancellation would still be a useful result; it would support a narrower claim than general transfer of loss-landscape geometry.

2. The practical value of small models as proxies remains unclear, particularly over longer training horizons.

All model sizes are trained for the same token budget, and the theory holds training time TT fixed as width increases. This leaves open the setting in which larger models are also trained longer. Finding 2b makes this limitation consequential: discrepancies in GG grow with training time. The current evidence therefore does not establish that transfer persists over the longer horizons relevant to large models.

The merging experiments do not fully address the practical question. The merging hyperparameters α\alpha and ρ\rho can be tuned at the target scale using merges and evaluations, without retraining. This can still be costly, but the paper does not quantify the savings from using a small proxy. Transfer of the fine-tuning learning rate is also already covered by µTransfer.

A stronger demonstration would use a small model to select an expensive training decision, then validate that decision at a larger scale. For example, can a small model predict when branches become safe to average, or select a synchronization interval for local SGD, DiLoCo, or branch-train-merge?

Overall assessment (Rating: 3 — Weak acceptance).

The parameterization comparison and the C=G−JC=G-J decomposition are useful contributions, supported by a substantial set of experiments. My main reservations concern the interpretation of finite-width agreement and the practical claim that small models predict large-model behavior. The theory establishes convergence but does not explain the observed agreement, and the experiments leave longer training horizons unresolved.

I lean toward acceptance, but would reconsider if the response to Question 1 indicates that the agreement is largely due to shared finite-width offsets, without further evidence supporting the broader geometry-transfer claim.

Questions for authors

1. How much of the cross-width agreement comes from shared finite-width corrections?

Could you compare the width dependence of the midpoint loss with that of the average endpoint loss? For example, report

LN(1/2)−L2N(1/2)L_N(1/2)-L_{2N}(1/2)

alongside

LN(0)+LN(1)2−L2N(0)+L2N(1)2.\frac{L_N(0)+L_N(1)}{2}-\frac{L_{2N}(0)+L_{2N}(1)}{2}.

If these differences are large and nearly equal, shared offsets explain much of the agreement in CC. An SP control at the same transformer sizes would help establish whether this cancellation depends on the parameterization. I would also like to understand what evidence distinguishes agreement in this particular loss difference from broader transfer of geometry.

2. What determines the limiting barrier C∞C_\infty?

The branches differ only in their minibatch sequences. Near a minimum, the stationary-fluctuation picture cited from Liu et al. (2021) suggests an η/B\eta/B dependence for branch disagreement and, under suitable local assumptions, for JJ. Does the measured JJ follow this dependence? How does changing batch size, fixed at 32 sequences in the main experiments, affect CC?

Even an analysis of a simpler model that identifies when C∞>0C_\infty>0 would help explain the empirical observations beyond establishing that a limit exists.

3. Which sources of nonlinearity contribute to GG?

Dey et al. motivate completeP by arguing that individual blocks become effectively linear in their own weights under depth-µP as L→∞L \to \infty, whereas completeP preserves block nonlinearity. How does this distinction relate to GG? Figures 17 and 20 appear similar despite the different parameterizations.

Does composition across blocks sustain GG even when individual blocks become locally linear in their weights? What roles do attention softmax and the readout play? Interpolating one block at a time, or measuring layer-wise feature connectivity as in Zhou et al. (2023b), could help locate the effect and assess the interpretation of GG as disrupted features.

4. Does the TIES implementation depend on the stored parameter scales?

Task arithmetic and Fisher-style averaging commute with consistent per-layer rescaling. TIES may not if trimming ranks update magnitudes globally across parameters. Lemma B.1 makes this relevant because stored layer scales can change under equivalent parameterizations.

Under completeP, both update magnitudes and the fraction of parameters in embeddings versus transformer blocks change with width and depth. At N=512N=512, embeddings account for roughly half the parameters at L=8L=8, but only about a tenth at L=64L=64. Is trimming performed globally or per tensor? If globally, could changes in which layers survive trimming explain the 4.4–6.9 percentage-point degradation when transferring TIES hyperparameters from deep to shallow models?

5. What does the analysis predict for SP with a learning rate tuned separately at each size?

Under NTP, the limiting barrier is nonpositive; under µP, the experiments retain a positive barrier. Many practical training setups use SP with a learning rate selected separately for each model size. What behavior should we expect in that setting? How would it relate to reports that connectivity or mergeability improves with width, such as Ito et al. (2026)?

Minor clarification. Section 4 states that the Figure 1 ConvNet is trained for T=100T=100 epochs, while Section E.1.1 gives T=50T=50.

Possible follow-up, not required for acceptance. Fort et al. (2020) connect the onset of LMC to stabilization of the NTK. Testing whether GG tracks kernel changes after branching, across model sizes, could provide a mechanism for the observed behavior. A depth-limit analysis for completeP would also be useful.

3 comments
AuthorsOpenReview ↗

Response to reviewer Cg8F (1/3)

Thank you very much for your thorough review and for finding our barrier decomposition helpful for interpreting the empirical results. We address the raised weaknesses and questions below.

W1/Q1: The theory establishes a limit, but does not explain the finite-width agreement or characterize the limiting geometry. [...] How much of the cross-width agreement comes from shared finite-width corrections?

Thank you for bringing this point up. The comparison you propose is in fact equivalent to the collapse of chord deviations across sizes, since

CN(12)−C2N(12)=[LN(12)−L2N(12)]−[LN(0)+LN(1)2−L2N(0)+L2N(1)2],C_N(\tfrac12) - C_{2N}(\tfrac12) = \Big[L_N(\tfrac12) - L_{2N}(\tfrac12)\Big] - \Big[\tfrac{L_N(0) + L_N(1)}{2} - \tfrac{L_{2N}(0) + L_{2N}(1)}{2}\Big],

so the two differences are nearly equal exactly when CC agrees across sizes, which is precisely what we report in Fig. 3 (which was Fig. 2 of the submission). Furthermore, it is visible from the same figure that both differences are large, as validation losses still change notably with size (l.h.s. of Fig. 3). So yes, this indicates the shared finite-size offsets of midpoint and endpoint losses cancel in CC. Hence, we think that this is not something separate to check, but precisely the observation that we make (see also l. 275–281 and App. D.5). We are of course aware that LMC, i.e., looking at one-dimensional slices of the loss landscape, cannot show all ways in which the geometry of the loss landscape could transfer, but it is one that has relevant applications and has received significant interest.

We agree that JJ depends only on endpoint predictions and is thus a function space quantity and related to the prediction consistency of [1]. However, the agreement of JJ does not follow from their result: while they compare networks trained with the same data order and their prediction variance they study is over initialization seeds, JJ measures how much the predictions of two branches with shared initialization and training prefix diverge due to SGD noise, which does not vanish with width, and, to the best of our knowledge, that this diversity agrees across sizes is a new observation.

Regarding the SP control, thank you for the suggestion! We replicated the sweep from sec. 5.1 in App. G.1.4 (Figs. 25–26), see also our response to Q5. Under SP, the agreement across width for the same LRs is much worse compared to completeP, which is partly due to the stable LR window not transferring (for width scaling), but even at small stable LRs one can see an upwards drift in JJ and GG, while the curves collapse for completeP. For depth scaling, we find that there can be some transfer under SP as well. However, for (post-BatchNorm) ResNets under SP, chord deviations grow strongly with depth (see App. G.1.9).

W2: The practical value of small models as proxies remains unclear, particularly over longer training horizons. [...] A stronger demonstration would use a small model to select an expensive training decision, then validate that decision at a larger scale.

Thank you for this suggestion! We now discuss a practical application of our LMC results in l. 436–449 and App. G.1.7. Indeed, in what is now Fig. 9 we show that small models can predict when it is safe to average weights: both the speedup from averaging and the window after branching during which the weight average is at least as good as its endpoints agree across sizes (the smallest model predicts the safe window of the largest within a median factor of 1.19 (width) and 1.27 (depth), compared to 1.03 and 1.04 across seeds of the largest model, at 53$\times$ and 6.9$\times$ fewer FLOPs). Under SP, this prediction degrades substantially, see also Table 10.

Furthermore, HP transfer for model merging is already another practical application, and we now quantify its savings (l. 523–526 and Table 11): given pretrained models, tuning the fine-tuning LR η\eta and merging HPs on the smallest model instead of the largest saves 10.6–12.0$\times$ (width) and 4.6–4.8$\times$ (depth) FLOPs. While α\alpha and ρ\rho can indeed be tuned without retraining, we note that they interact with η\eta, i.e., the best LR for each fine-tuned model need not be the best one for the merge, and the range of α\alpha that merges well depends on η\eta (Fig. 43), so this goes beyond LR transfer as in μ\muTransfer. We also only explore a small slice of the merging HP landscape (e.g., one could tune η\eta per task or weigh the task vectors individually), so savings could be even greater for larger sweeps.

Regarding longer horizons, we agree that since all sizes are trained for the same token budget, the models here are not necessarily compute-optimal. However, this is the standard setting for scaling limits and HP transfer, where the smaller models serve as proxies and are not necessarily of interest themselves. It would certainly be exciting to see how loss barriers behave at longer training horizons, and to select synchronization intervals for local SGD.

AuthorsOpenReview ↗

Response to reviewer Cg8F (2/3)

Q2: What determines the limiting barrier C∞C_\infty? [...] Does the measured JJ follow this dependence? How does changing batch size, fixed at 32 sequences in the main experiments, affect CC?

Thanks for suggesting this. We reran the main sweep of Sec. 5.1 with a larger batch size of 256 (App. G.1.3) and tested how well η/B\eta/B describes JJ (App. G.1.6, Fig. 31). For late branching (b=5b = 5k) where JJ dominates, we find that indeed JJ is approximately a function of η/B\eta/B. Furthermore, we observe that CC largely keeps its sign across batch sizes (Figs. 15 and 20). This is largely in line with [2], since their η/B\eta/B dependence holds to leading order in the LR near a minimum under the common approximation that the gradient noise covariance is proportional to the curvature. Furthermore, this might also offer an additional perspective on [3], who find that merging gains for different LRs and batch sizes align as a function of the effective noise scale.

We agree that an analysis of a simpler model that identifies when C∞>0C_\infty > 0 would be very interesting. This is not straightforward, as the standard simplifications (e.g., a quadratic loss around a minimum, or the NTK regime) exclude positive barriers. A simple model thus has to capture nonlinear feature learning after early branching. We have tried this with deep linear networks, but have not found a clean enough characterization yet that makes transferable predictions.

Q3: Which sources of nonlinearity contribute to GG? [...] Does composition across blocks sustain GG even when individual blocks become locally linear in their weights? What roles do attention softmax and the readout play?

Thank you for bringing this up. We would argue that depth-μ\muP and completeP do not behave the same in App. G.1.5. Under depth-μ\muP, the optimal LR shifts upward with depth and GG shows a visible downward drift with depth, which would be consistent with the network becoming lazier. In contrast, neither happens under completeP. Independently, we observe a similar effect for ResNets on CIFAR-10 (Fig. 36).

Regarding individual blocks, attention softmax and readout: interpolating one block at a time would indeed be very interesting to explore. We did not get to trying this yet, but thank you for the suggestion.

Q4: Does the TIES implementation depend on the stored parameter scales? [...] Is trimming performed globally or per tensor?

This is a good point, thank you for bringing it up! We looked into it, and indeed we used an implementation of TIES that trims globally across tensors, which might explain the worse transfer we observe. For the time being, we state this in App. F.3. We are still looking into whether trimming each tensor separately changes the results, and will share our findings as soon as possible, depending on our available compute (if not by the end of the discussion period, we will report them in the final version). Please note that we reran the merging experiments with longer pretraining, so the numbers in what is now Fig. 11 differ slightly from the original submission.

Q5: What does the analysis predict for SP with a learning rate tuned separately at each size?

Under SP, there is no general feature learning width limit (however, [4] show controlled divergence under cross-entropy loss which we discuss in App. B), so our analysis does not predict consistent limits of CC, JJ and GG. Empirically, we added the sweep of Sec. 5.1 under SP (App. G.1.4, Figs. 25–26) and observe that under width scaling, the optimal LR shifts downward and the curves do not collapse, while under depth scaling there is some transfer in NanoGPT. For ResNets (Fig. 36), the optimal LR indeed seems to transfer somewhat across depth under SP, while CC, JJ and GG still transfer much less than under completeP. However, we argue that this is, in some sense, also the idea of using scaling limits in the first place, as their premise is precisely that they behave predictably under width and depth scaling, while SP does not, and its stability is very architecture-dependent. We also checked what happens when the LR is tuned separately at each size. For width scaling at batch size 32, this brings CC, JJ and GG under SP into roughly comparable agreement as under completeP. However, at batch size 256, CC and GG still differ by up to 5$\times$ across widths, so tuning per size can partly restore the agreement under SP, but not reliably (also, it requires an LR sweep at every size which is precisely what HP transfer is meant to avoid).

AuthorsOpenReview ↗

Response to reviewer Cg8F (3/3)

Regarding connectivity improving with width, this mostly concerns models trained from independent initializations. With permutation alignment [5, 6] this improvement mainly reflects the matching problem itself, which we discuss in App. G.2.1. Without alignment, indeed [7] find that the merged model behaves like an implicit ensemble (i.e., the weights of two independently trained models are approximately orthogonal so that the merged model can embed both functions without interference). This mechanism does not apply to branches from a shared initialization, so we do not expect these results to carry over to the setting we study in this paper.

Section 4 states that the Figure 1 ConvNet is trained for T=100T=100 epochs, while Section E.1.1 gives T=50T=50.

Thanks for pointing this out. We have now made them the same (T=50T = 50 epochs), which is now in both cases the setup from App. F.1.1.

Testing whether GG tracks kernel changes after branching, across model sizes, could provide a mechanism for the observed behavior. A depth-limit analysis for completeP would also be useful.

This is an interesting point. We will investigate whether GG tracks kernel changes after branching, and potentially include it in the final version of the paper. We also agree that a depth-limit analysis would be an exciting extension of this work.

[1] Nikhil Vyas, Alexander Atanasov, Blake Bordelon, Depen Morwani, Sabarish Sainathan, Cengiz Pehlevan. Feature-Learning Networks Are Consistent Across Widths At Realistic Scales. NeurIPS 2023.

[2] Kangqiao Liu, Liu Ziyin, Masahito Ueda. Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent. ICML 2021.

[3] Chenxiang Zhang, Alexander Theus, Damien Teney, Antonio Orvieto, Jun Pang, Sjouke Mauw. How Does the Optimizer Implicitly Bias the Model Merging Loss Landscape? ICLR 2026.

[4] Moritz Haas, Sebastian Bordt, Ulrike von Luxburg, Leena Chennuru Vankadara. On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling. NeurIPS 2025.

[5] Rahim Entezari, Hanie Sedghi, Olga Saukh, Behnam Neyshabur. The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks. ICLR 2022.

[6] Samuel K. Ainsworth, Jonathan Hayase, Siddhartha Srinivasa. Git Re-Basin: Merging Models modulo Permutation Symmetries. ICLR 2023.

[7] Akira Ito, Masanori Yamada, Daiki Chijiwa, Atsutoshi Kumagai. Do We Really Need Permutations? Impact of Model Width on Linear Mode Connectivity. ICLR 2026.

Reviewer_s2rwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Broadly, the paper is aimed at understanding how geometric features of the loss landscape transfer from small to properly-scaled large models. It focuses on one particular probe of that geometry: the loss along a straight path connecting two models in weight space. More precisely, the authors consider pairs of models that coincide at a training checkpoint and then continue training with independent optimization randomness. For the resulting parameters θA,θB\theta_{A}, \theta_{B}, they study the signed chord deviation

C(s)=L((1−s)θA+sθB)−(1−s)L(θA)−sL(θB),s∈[0,1].C(s) = \mathcal{L}((1-s)\theta_{A} + s \theta_{B}) - (1-s) \mathcal{L}(\theta_{A}) - s \mathcal{L}(\theta_{B}), \qquad s \in [0, 1].

The paper investigates whether this quantity behaves consistently across widths and depths under suitable scaling parametrizations. To study C(s)C(s), they introduce a decomposition

C(s)=G(s)−J(s).C(s) = G(s) - J(s).

Here, GG is the nonlinear interpolation gap which measures the loss difference between weight interpolation and output interpolation; JJ is the ensembling gain, measuring the loss reduction from averaging outputs relative to averaging the endpoint losses.

Theoretically, the paper establishes infinite-width limits for the interpolation curves of smooth MLPs trained with SGD at fixed depth over a finite horizon under both μP\mu\mathsf{P} and NTP. Just as these parameterizations yield qualitatively different limiting training dynamics, they also yield different limiting interpolation geometry. Under μP\mu\mathsf{P}, nonlinear weight-interpolation effects can persist, allowing C∞=G∞−J∞C_\infty=G_\infty-J_\infty to have either sign. Under NTP, asymptotic linearization makes G∞=0G_\infty=0, leaving C∞=−J∞≤0C_\infty=-J_\infty\leq0 for convex prediction losses.

Empirically, the authors investigate agreement across practical model sizes and whether that agreement persists over training (superconsistency). In the motivating CIFAR-10 experiment, the midpoint chord-deviation trajectories align tightly across the tested widths under μP\mu\mathsf{P} throughout training, whereas under NTP they shift downward as width increases. This provides evidence of approximate superconsistency under μP\mu\mathsf{P} in that setting, beyond what existence of either limit establishes. Broader experiments examine consistency across width and depth and across vision/language modalities.

Critical strengths

(1) The paper investigates a meaningful aspect of geometry beyond local curvature. Agreement of Hessian spectra across model sizes does not establish agreement of the loss along a segment connecting two trained models. The experiments directly examine these interpolation paths, and Figures 4–5 provide evidence that cross-size agreement can persist even when a local quadratic approximation is quantitatively inaccurate. This makes the contribution a substantive extension of earlier work on landscape consistency. Experiments across widths, depths, branching times, learning rates, and vision and language architectures support the relevance of the observations beyond a single configuration.

(2) The decomposition and theory provide a coherent foundation for the empirical study. Separating C=G−JC=G-J distinguishes output-ensembling effects from nonlinear weight-interpolation effects, and its local Gauss–Newton correspondence connects these measurements to more-familiar quantities in the study of local loss curvature. For smooth MLPs at fixed depth and finite training horizon, Appendix D establishes full-curve width limits under both μP\mu\mathsf{P} and NTP, distinguishing persistent nonlinear interpolation effects from their disappearance in the lazy limit. This is a useful application of tensor programs that appears mathematically sound. As for readability, it does appear (upon skimming the arguments) that this application of tensor programs is a clear exposition, and potentially of use to theory community.

Critical weaknesses

(1) The scope of superconsistency of CC needs clearer quantification. Positive examples coexist with discrepancy growth after early branching (Figures 11 and 14) and acknowledged width dependence in Figure 27. Since RC(t)R_C^{(t)} measures growth relative to a baseline, it does not establish whether discrepancies remain small. Please report unnormalized DC(t)D_C^{(t)} trajectories with uncertainty from existing runs and identify which learning-rate, branching-time, and size regimes maintain close agreement throughout training. Distinguish the broad midpoint diagnostics from the selected final-time full-chord comparisons.

(2) The theory–experiment connection needs clearer exposition. Figure 1 highlights sustained cross-width alignment under μP\mu\mathsf{P} versus width-dependent drift under NTP. Lines 256–258 instead contrast a property of the μP\mu\mathsf{P} limit with movement toward the NTP limit. Both parameterizations admit limits; the theorems do not explain the stronger finite-width agreement under μP\mu\mathsf{P}. The caveat in lines 260–262 and definitions in lines 295–298 help, but introducing the distinction between limiting geometry, practical consistency, and superconsistency when interpreting Figure 1 would clarify the contribution.

Questions for authors

(1) Can the authors report unnormalized DC(t)D_C^{(t)} trajectories with uncertainty from the existing runs, and identify which learning-rate, branching-time, and size regimes maintain small cross-size discrepancies throughout training? Please distinguish conclusions about the midpoint from those supported along the full interpolation path.

(2) When interpreting Figure 1, can the authors explicitly distinguish the limiting geometry characterized by Theorems 4.1–4.2 from practical finite-width consistency and its persistence over training? In particular, please clarify how lines 256–258 relate to the definitions of consistency and superconsistency in lines 295–298.

2 comments
AuthorsOpenReview ↗

Response to reviewer s2rw

Thank you for the insightful review and for recognizing our work as a substantive contribution to the consistency of loss landscapes. We address the raised weaknesses and questions below.

W1/Q1: The scope of superconsistency of CC needs clearer quantification. [...] Can the authors report unnormalized DC(t)D_C^{(t)} trajectories with uncertainty from the existing runs, and identify which learning-rate, branching-time, and size regimes maintain small cross-size discrepancies throughout training? Please distinguish conclusions about the midpoint from those supported along the full interpolation path.

Thank you very much for raising this point. In the updated paper, we have sharpened the discussion of the time dependence of the discrepancies in JJ and GG (see updated Finding 2b, l. 356–357 and l. 371–397 and further details in App. D.6–D.7). Namely, each run Xn,r(t)X_{n,r}^{(t)} (X∈{J,G}X \in \{J, G\}) has two sources of randomness: the initialization and the minibatch history, which we decompose (Eq. 7) into a deterministic expected limit and a finite-size bias, plus the randomness of the minibatch history (which persists in the limit) and the initialization fluctuation. We estimate how bias and fluctuation (which are both finite-size effects) decay as the network grows, and fit power laws in time.

What is now Fig. 4 shows how fast bias and fluctuation grow over training for the different LRs and branching times, how large they are relative to JJ and GG at the end of training, and which of the two dominates. We find that GG and JJ behave quite differently in time: after early branching, the fluctuation of GG grows together with GG itself, while the fluctuation of JJ contracts until its bias eventually dominates. As you requested, we further report the unnormalized discrepancies between adjacent sizes Xn,r(t)−X2n,r(t)X_{n,r}^{(t)} - X_{2n,r}^{(t)} over training (now split up in bias and fluctuation), in the new Fig. 5 and for further LRs/depth in Fig. 17. We have also performed this analysis with a larger batch size (App. G.1.3), ViTs (App. G.1.8) and ResNets (App. G.1.9).

Regarding the midpoint: Indeed, most of our analysis concerns the midpoint s=1/2s = 1/2 and we would argue that this characterizes the overall behavior of the curves well. Since the two branches are exchangeable, the distribution of interpolation curves is precisely symmetric around s=1/2s = 1/2, and in all of our experiments the curves attain their minimum or maximum close to the midpoint. Furthermore, this is also the common point at which barriers are mostly reported in the literature. The full curve in what is now Fig. 6 mainly serves the purpose to show that the interpolation curves are clearly non-quadratic.

W2/Q2: The theory–experiment connection needs clearer exposition. [...] When interpreting Figure 1, can the authors explicitly distinguish the limiting geometry characterized by Theorems 4.1–4.2 from practical finite-width consistency and its persistence over training? In particular, please clarify how lines 256–258 relate to the definitions of consistency and superconsistency in lines 295–298.

Thank you for the suggestion. We tried to make this clearer in what is now l. 273–276, i.e., that Thms. 4.1 and 4.2 establish the existence of limits at any fixed horizon tt, but they make no statement about whether CN(t)C_N^{(t)} is already close to its limit or what happens as tt grows. We have also reworded l. 206–207 (where the theory is first related to Fig. 2). We make this distinction after the theorems rather than at Fig. 2, since at that point the figure only serves to motivate that a limit may exist. Regarding l. 256–258 of the submission (now l. 268–270): this sentence only relates the limits of Thms. 4.1 and 4.2 to the trends in Fig. 2, and is not a statement about consistency in the sense of l. 287–291. Under μ\muP, the limits of CC, JJ and GG are deterministic given the minibatch histories, and neither G∞G_\infty nor C∞C_\infty needs to vanish. Under NTP, GG vanishes in the limit, so GNG_N at finite width itself could be seen as a finite-size correction.

Thank you for your response and attentive revision. The changes address my concerns. The discussion around the (now) Figure 2 clears up for me what the theorems establish vs empirical agreement across practical widths. I think this clarifies how the theory and experiments relate.

The additional analysis of GG and JJ also clarifies things. A plot of the paired discrepancy in CC from existing runs could be useful, since differences in GG and JJ might cancel partially. I don't think this is a super necessary addition, so I'm willing to say I'm satisfied with the authors' response.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all reviewers for their thoughtful and constructive feedback! We have uploaded a revision of the manuscript that addresses it, see the highlighted parts for updates. Most importantly, we have made the following changes:

  • We made the introduction more self-contained (Sec. 1).
  • We sharpened Finding 2b on how finite-size effects evolve over training (Sec. 5.1, Figs. 4–5, App. D.6–D.7).
  • We show that small models predict the speedup from weight averaging and the window in which averaging is safe (Sec. 5.1, Fig. 9), and quantify the resulting FLOP savings (App. G.1.7), and similarly for model merging HP transfer (Sec. 6, App. G.3.2).
  • We added a toy experiment in the setting in which Thm. 4.1 applies (App. E).
  • We reran the experiments, hence the exact numerical values changed slightly while all conclusions stay the same. For model merging, we changed the pretraining to 10B tokens on FineWeb-Edu (Sec. 6).
  • We added experiments with a larger batch size (App. G.1.3), the standard parameterization (App. G.1.4), and a more comprehensive sweep for variance collapse (Sec. 5.2, App. G.2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.