acceptodds
Under review as a conference paper at ICLR 2027

A Defense of the Quadratic Model

23 views

Abstract

Due to the complexity of neural network loss landscapes, optimization theory is forced to rely on idealized models, and there is generally a tradeoff between how theoretically tractable the model is, and how accurately it describes the true optimization dynamics. In this work, we stress test the simplest possible model of optimization – the quadratic model – and show that it can be surprisingly predictive in an LLM setting with 150M parameters and 3B training tokens. Specifically, we show that Taylor expanding the model and the loss function at intermediate checkpoints through training can accurately predict the optimization dynamics over windows that can last up to 10% of training. Having established this agreement, we then turn to analyzing the structure of these local quadratic optimization problems through two lenses: the Hessian spectrum and local stability. Using Lanczos quadrature with extremely deep probes, we are able to estimate the Hessian spectrum deep into the tail, and we find a surprising amount of structure in both the eigenvalues and eigenvectors, which depends on the batch size, preconditioner, and training time. We also empirically test local linear stability at intermediate checkpoints and compare it to theoretical predictions to demonstrate that optimization in LLMs typically occurs at a stochastic edge of stability, whose nature is also determined by batch size. Our results indicate the quadratic model may be a theoretically tractable proxy for pretraining optimization dynamics.

Primary area:
optimization
Keywords:
quadratic, llm, pretraining, models, training, optimization, lanczos, slq

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QAoJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether local quadratic models can accurately describe the optimization dynamics of LLM pretraining. Through experiments on a 150M-parameter Transformer, the authors show that Gauss–Newton quadratic approximations can predict the training loss trajectory over relatively long windows, particularly in the later stages of training. The paper further analyzes the Hessian spectrum, gradient–Hessian alignment, and edge-of-stability behavior, revealing interesting structures and connections to existing optimization theory.

Critical strengths
  1. Quadratic models are widely used in optimization theory, but their relevance to practical neural network training remains insufficiently understood. This paper directly investigates this fundamental question, helping bridge the gap between theoretical optimization models and practical LLM pretraining.
  2. The observation that local quadratic approximations can accurately predict the loss trajectory over a nontrivial fraction of training is surprising and encouraging. The findings provide empirical support for using quadratic models to understand neural network optimization.
  3. Beyond validating the quadratic approximation, the paper investigates the Hessian spectrum, eigenvector structure, gradient–Hessian alignment, and optimization stability. These complementary analyses provide useful insights into LLM pretraining dynamics.
  4. The authors invest substantial computational effort in estimating the Hessian spectrum using deep Lanczos probes. The numerical validation and additional experiments in the appendix strengthen the credibility of the spectral analyses. Overall, the paper is well motivated and presents its findings in a clear and organized manner.
Critical weaknesses
  1. All experiments are conducted on a single model architecture at approximately 150M parameters. While the experiments are computationally demanding, it remains unclear whether the findings generalize to larger LLMs, different architectures, or other optimizers. This is particularly important given the paper's broader motivation of understanding LLM pretraining.
  2. As shown in Figure 1 and Table 1, the approximation is considerably less accurate during early training, and its performance also depends on the learning rate schedule. In particular, the constant-learning-rate experiments show substantially less reliable agreement. These observations suggest that the applicability of the quadratic model may be more restricted than the paper's broad conclusions imply.
  3. Although the empirical agreement between the quadratic approximation and the original training dynamics is interesting, the paper provides limited insight into why this agreement emerges in later training stages. For example, it is unclear whether this behavior is primarily explained by reduced learning rates, changes in the Hessian, or other properties of the optimization trajectory.
  4. In the stability experiments, the authors freeze the Adam preconditioner and analyze preconditioned SGD on a convex quadratic approximation. The experiments also filter extreme outliers to avoid numerical instability.
Questions for authors
  1. In Section 2, the agreement between the quadratic approximation and the original model is evaluated primarily through validation loss. Have the authors also examined the similarity of the resulting parameter updates or optimization trajectories? It would be interesting to understand whether similar loss curves correspond to similar trajectories in parameter space.
  2. In Table 1, the authors report prediction windows starting from the 100% training checkpoint. Could the authors clarify how the learning rate schedule is handled when continuing beyond the original training budget?
  3. In Section 3.2, the authors observe that the spectral tails are largely invariant to batch size and preconditioning, up to rescaling. Is there a possible theoretical explanation for this observation, or any additional evidence suggesting when such universality should be expected?
2 comments

We thank the reviewer for their kind words about our work. We begin by addressing the weaknesses they raised:

  1. We have run experiments for the spectrum measurement at scales from 9M - 1B (which we have now added to the manuscript in Appendix G, Figure 18), and we are currently investigating longer training runs (up to 32x Chinchilla). However, given that the spectrum does not change much after warmup (Figure 2), we expect the results will remain the same.

  2. We believe that the agreement between the losses is unlikely to come from a smaller step size. Per Table 1, at 70% of training the losses agree on about 1% of samples, with the learning rate at 0.32 of its peak. At 80% of training, the learning rate is 0.2 of its peak, but agreement jumps to 5.5% of samples. The learning rate drops by a factor of about 1.6 between these two points. If agreement scaled with step size, we would expect only about 1.6% agreement at 80%. We conjecture that the main disagreement comes from the network being at EoS during training. From prior work, [1] when a network stays at EoS, there is a cubic regularization term which pushes down the curvature, causing the dynamics to deviate from the quadratic approximation. This nonlinearity is partially captured by prox, which may be why it shows better agreement. The step size likely has some effect, we believe it is not the main reason behind the disagreement.

  3. We refer to the answer from weakness 2, and speculate that later in training the agreement improves because the model leaves EoS for cosine decay as per Figure 5, which is why the agreement improves for quad, whereas for constant the model stays at EoS during training (see Figure 15), and quad does not match throughout.

  4. We would like to point out that this is a necessary methodological choice and not a weakness. One important takeaway from our work is that we can linearize the model itself, but not the Adam optimizer as it has to handle these rare divergent events through nonlinear damping. However, if we freeze the optimizer, filtering out these outlier events suffices for maintaining the underlying SGD dynamics, which are stable (depending on η\eta and BB). This also allows us to conclude that, at least with regards to stability, one of the main roles of Adam is handling these outliers. Furthermore, this is the standard definition of EoS for Adam, as established by [2].

Regarding the questions:

  1. We thank the reviewer for the insightful question. We did not measure parameter agreement, as it is not obvious what the correct metric to compare the weights would be in: for example, L2L_2 norm would be dominated by the difference in params with large magnitude. We also want to emphasize that in some sense the metric we have chosen is the strongest criterion possible, because we take the parameters trained using the linearized dynamics, and plug them back in the nonlinear model for evaluation. A simpler criterion would be to just show the validation loss of the linearized model itself, but this would be a less honest evaluation.

  2. We apologize for not explaining this aspect. For all the intermediate checkpoints we follow the respective scheduler (cosine or constant), and for the checkpoints starting at 100%, we keep the learning rate fixed at the end learning rate for that scheduler.

  3. This is indeed a good question. One hypothesis is that the bulk of the spectrum is in a random or incoherent basis, and Adam is only good at whitening Hessians that are approximately diagonal. It would be interesting to explore whether this is still true for optimizers that work in different bases, such as Muon, SOAP or Shampoo, but we leave to future work.

[1] Damian, Alex, Eshaan Nichani, and Jason D. Lee. "Self-stabilization: The implicit bias of gradient descent at the edge of stability." arXiv preprint arXiv:2209.15594 (2022).

[2] Cohen, Jeremy M., et al. "Adaptive gradient methods at the edge of stability." arXiv preprint arXiv:2207.14484 (2022).

Thanks for the clarification. I think this work is valuable. I have raised my rating to 3.

Reviewer_SeFpOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper builds on prior work, which employs a quadratic loss approximation to understand otherwise intractable optimisation dynamics. In particular, this paper focuses on the pretraining dynamics of LLMs, using a 150M-parameter LLM trained with Adam on 3B tokens, and analysing both the raw and Adam-preconditioned curvature. The authors show empirically that this quadratic model can be predictive of the pretraining dynamics of LLMs.

Firstly, the authors show that for both a linearised LLM and a Taylor-expanded loss around intermediate checkpoints, the training dynamics tend to match those of the full model and loss, particularly in mid-to-late training regimes. They test this approximation on windows of 10% of the total training budget.

Secondly, the authors employ Lanczos quadrature to approximate the spectra of the Hessian and Gauss-Newton matrices (and their preconditioned versions) governing the quadratic optimisation dynamics. For synthetic spectra, they validate the accuracy of their approach. With the estimated spectra, they then analyse how the mass of each eigenvector is distributed across parameter groups (MLPs, attention) have across eigenvalues, identifying a split in the spectrum into a head (containing roughly vocabulary-size many eigenvalues, dominated by the unembedding layer) and a tail which appears to follow a power law. This tail seems to be shared across batch sizes and is unaffected by the Adam preconditioner. From the spectral curves, the authors fit power laws, identifying both the decay of the eigenvalues (α\alpha) and how the remaining loss is spread across eigenspace dimensions (β\beta). The authors find that β<1\beta < 1, whereas in the literature it is typically assumed that β>1\beta > 1.

Lastly, the authors test the edge of stability of the LLM (a learning rate beyond which training becomes unstable), by perturbing the learning rate and batch size around intermediate checkpoints. Their empirical findings agree with linear stability theory, showing that training generally operates within a factor of 2 of the edge of stability. Moreover, they find that for smaller batch sizes, LLMs operate close to the stochastic edge of stability, whereas for larger batch sizes, they operate close to the deterministic edge of stability.

Critical strengths
  1. The paper is exceptionally thorough (Lanczos spectral estimates validated against ground-truth spectra and an additional probe with a second random starting vector, Figure 17; ablations of precision, sample size and probe count, Appendix D, Figure 16; a 5×55 \times 5 (η,B)(\eta, B) stability grid with 10 seeds per cell, Section 4; and all spectral results reported for the four matrices GG, HH, GPG_P and HPH_P, Section 3). The authors also make a clear effort to frame their results relative to existing literature (e.g. neural collapse and prior work on the edge of stability). This is consequential because the paper's contributions are largely empirical, so careful validation and ablation strengthen the credibility of their findings.

  2. The paper demonstrates that the quadratic model not only confirms known findings in the literature, but can also provide new insights into LLM pretraining. For instance, it confirms prior findings that large-batch training operates at the deterministic edge of stability (Section 4, L458-462). The authors also challenge existing assumptions, showing that the source exponent β\beta need not be greater than 1 (L327-332), and that the edge of stability does not disappear at small batch sizes, but instead becomes stochastic (L463-469). Beyond these, the paper provides interesting new insights: the quadratic model accurately predicts training behaviour for up to 10% of the training budget from mid-to-late checkpoints (Figure 1, Table 1), and the power-law tail of the spectrum (particularly for the Gauss-Newton matrix) remains consistent across batch sizes and is largely unaffected by the Adam preconditioner (L257-262). This is consequential because it demonstrates that simple, theoretically tractable models can be effective tools for analysing LLM pretraining.

Critical weaknesses
  1. Scale and generality: all experiments use a single 150M-parameter model trained at 1x Chinchilla, with one architecture and Adam without weight decay (Appendix E). This is acknowledged by the authors as a necessary trade-off for the probe depth (L479-481). Moreover, the local expansions fail early in training and perform poorly with a constant learning rate with EMA (Table 1, Figure 11, L482-485). This is consequential because it is unclear how well the spectral findings and the validity of the quadratic model transfer to larger, more typical LLMs and training setups.

  2. Some spectral claims are under-specified or insufficiently supported. The disappearance of the head at B=1024B = 1024 is attributed to Adam preconditioning it away (L281-284), but the head is also absent from the non-preconditioned spectra (Figure 7), and at B=1024B = 1024 it is present early in training before disappearing (Figure 9). The provided explanation doesn't address these. The fitting ranges for α\alpha and β\beta and the definition of the "tail" are not stated explicitly, and some summaries are imprecise (e.g. the introduction states α≈1\alpha \approx 1 (L61), which holds for the Gauss-Newton matrix but not the Hessian, α≈0.65\alpha \approx 0.65 at L252). This is consequential because the proposed mechanism for the head-tail split at large batch sizes is not supported by the paper's own figures, and the robustness of tail universality depends on how the tail is defined and fitted.

Beyond these, the presentation could be improved (Sections 3.2 and 3.3 alternate between discussing Figures 2 and 3, which is a bit confusing, and some terms, such as ϑj\vartheta_j, are used before being defined), but this does not affect the core message, nor does it influence my score.

Questions for authors
  1. Could the authors clarify which index range and method was used to fit α\alpha (L251-253)? More generally, I think it would be useful to define the "tail" once and state all fitting ranges explicitly in the text. Currently, the head/tail split is at i=Vi = V (L241-243), universality is assessed over [105,107][10^5, 10^7] (L261), spectra are aligned at 10610^6 (Figure 2), and β\beta appears to be fitted over [104,106][10^4, 10^6] (Figures 4 and 10).

  2. Why does the head disappear during training at B=1024B = 1024 (Figure 9), including in the non-preconditioned spectra (Figure 7)? Could this be related to the up projection outliers discussed in L287-293? Moreover, could the authors verify that gradient noise is negligible for the unembedding parameters at B=1024B = 1024, for instance by comparing ∥E[g]∥2\|\mathbb{E}[g]\|^2 with E∥g−E[g]∥2\mathbb{E}\|g - \mathbb{E}[g]\|^2?

  3. I would also be interested in understanding how sensitive the results are to some of the methodological choices. Firstly, has the Lanczos depth mm been ablated (L196), and how much do the fitted values of α\alpha and β\beta change for different depths (e.g. m=600m = 600 vs 12001200)? Secondly, could the single-probe check in Figure 17 also be shown for B=1024B = 1024, where the spectrum differs most? Lastly, the stability experiments filter the top 1% of sequences by ∥P−1Gi∥F2\|P^{-1}G_i\|_F^2 (L361-362), without which the authors state that nearly all hyperparameters diverged. How sensitive are the conclusions of Section 4 to this threshold?

  4. The expansions appear to track the true loss relatively poorly when using a constant learning rate with EMA (Figure 11, L484-485). Do the authors have an explanation for why this happens, and does this limit how generally the quadratic model can be considered predictive of pretraining dynamics?

  5. Some minor clarifications: a) The introduction states α≈1\alpha \approx 1 (L61), whereas L252 reports α≈0.65\alpha \approx 0.65 for the Hessian. Could the authors specify that the former refers to the Gauss-Newton matrix? b) The reported range β∈[0.1,0.2]\beta \in [0.1, 0.2] (L327-329) seems to only hold at later checkpoints, as Figure 10 gives values of 0.290.29 and 0.310.31 at T=10%T = 10\%. Could this be stated? c) Is the pv\sqrt{p_v} fit (L247-248) the "tok\sqrt{\text{tok}} probe" line in Figure 2? If so, could this be referenced in the text, and could the same comparison be shown for B=64B = 64 and B=1024B = 1024? d) Does the tail universality claim (L257-262) also hold at T=10%T = 10\% (Figure 7)? e) Why are EMA parameters evaluated (L145-146), and is the EMA re-initialised at θk\theta_k in the prox/quad continuations, or carried over from the original run? f) Could the authors define ϑj\vartheta_j at first use (L206-208)?

I understand that these are quite a few questions and requests. I would be happy to raise my score if the authors clarify the fitting ranges and definition of the tail (Question 1), and either provide further evidence for the explanation behind the disappearance of the head at B=1024B = 1024 (Question 2), or present this claim more tentatively. I think the remaining questions could be addressed in a revised version of the paper.


Post-rebuttal: the authors addressed the great majority of my questions (and follow-ups), and I have raised the soundness score and rating accordingly. My main remaining concern is scale (Weakness 1). The additional spectra from 9M to 1B parameters (Appendix G) are welcome, but the validity of the quadratic model and the stability results are still only shown for a single 150M-parameter model. I do not think this undermines the contribution, and the authors have stated that longer training runs are in progress.

4 comments

Regarding the weaknesses:

Scale and generality: all experiments use a single 150M-parameter model trained at 1x Chinchilla…

We have added 9M - 1B spectrum measurement experiments in Appendix G, Figure 18, and we are further exploring long training runs of up to 32x Chinchilla currently.

Regarding the questions:

  1. We define the “tail” as the eigenvalues past the vocab dimension (8192), as in our experiments we observe a separation in the powerlaws up to and after this point (Figure 2 and 3). We will add explicitly this definition in the text and fix the consistency issues, thank you for pointing this out. α\alpha was fitted on 10510^5 to 10710^7 with a linear fit in the log-log plot.

  2. The head part of the spectrum disappears at large batch size due to preconditioning. From Figure 3, we see that the head of the spectrum corresponds to the unembedding matrix, whose Hessian is close to diagonal. Due to this structure, Adam is able to almost fully whiten this part of the spectrum in full batch, whereas at small batch it only achieves a factor of H−1/2H^{-1/2} (shown in detail in [1, 2], specifically Table 1 in [2]). To ensure gradient noise is not an issue, we have compared the spectrum experiments for 10M and 100M tokens in Figure 16 and found minimal differences.

  3. We have ablated between 1200 and 600 (running for mm steps also gives the results for any m’≤mm’ \leq m for free) and the plots remained identical until very deep in the tail where the uncertainty was slightly reduced (past the current point where we fit the power laws). We will add these plots, and we are also happy to run and include the single-probe check for B=1024B=1024 in the next version of the paper, as we are unable to run them during the rebuttal period due to computational constraints. Regarding the outlier filtering, we have not tried other thresholds, but we have preliminary experiments where we freeze the embedding and unembedding matrices without any filtering (as this is where the rare token explosions were happening), and the results are consistent with the current plots. In the histograms used for the top 1%, there is a small number of extreme outliers, so it is possible that only filtering these would suffice.

  4. While for quad it is true that the losses do not agree, for prox the agreement is much better. We speculate that this is because for constant lrs, the model stays at the edge of stability during training (Figure 15), whereas for cosine the model leaves the edge when the cosine decays towards the end (but still stays within a factor of 2 of the edge). Prior work [3] has shown that when a network trains at EoS, there is a cubic regularization term which pushes down the curvature, causing the dynamics to deviate from the quadratic approximation. This nonlinearity is partially captured by prox, which may be why it shows better agreement. However, this is all conjectural and we leave a rigorous analysis to future work.

  5. Thank you for raising these points and we will fix these inconsistencies in the main body. For questions a), b), c), you are correct. For question d), the overlap is worse for the 10% checkpoint compared to the rest, so the collapse only appears (at least in our current experiments) after 50% of training, which we will clarify. For question e), we evaluate the EMA because the last iterate trajectory has variance across steps, which can hide the actual loss behaviour - moreover, EMA is standard in pretraining and necessary for the constant learning rate runs. The EMA parameters are carried over from the original run at the checkpoint, but is then updated independently with the iterates coming from training the linearized model (note that ​​although the EMA retains memory of the original run, the slope of the averaged parameter trajectory adjusts to the new updates). For f), we will introduce this in the main body.

[1] Cohen, Jeremy M., et al. "Understanding optimization in deep learning with central flows." arXiv preprint arXiv:2410.24206 (2024).

[2] Liu, Bingbin, et al. "Adam or gauss-newton? a comparative study in terms of basis alignment and sgd noise." arXiv preprint arXiv:2510.13680 (2025).

[3] Damian, Alex, Eshaan Nichani, and Jason D. Lee. "Self-stabilization: The implicit bias of gradient descent at the edge of stability." arXiv preprint arXiv:2209.15594 (2022).

Thank you for the detailed response and the additional experiments in Appendix G. The definition of the tail and the fitting range for α\alpha address Question 1.

However, I do not think my concern in Question 2 is addressed yet. The proposed mechanism (Adam's preconditioner whitening the near-diagonal unembedding curvature at large batch size) only applies to the preconditioned spectra. It does not explain why the head is also absent from the non-preconditioned spectra at B=1024B = 1024 (Figure 7), or why it is present at T=10%T = 10\% and then disappears (Figure 9).

Moreover, the cited whitening result concerns full-batch training. Figure 16 varies the number of tokens used to estimate the spectrum after training, so it shows that this estimate is accurate. It does not show that gradient noise for the unembedding is small at B=1024B = 1024, since that noise influences Adam's second-moment estimate during training, before the spectrum is estimated. A direct check could be to compare ∥E[g]∥2\|\mathbb{E}[g]\|^2 with E∥g−E[g]∥2\mathbb{E}\|g - \mathbb{E}[g]\|^2 for the unembedding at B=1024B = 1024.

Could the authors either explain Figures 7 and 9 under the proposed mechanism, or present this mechanism as a hypothesis in the revised text? It would help if the authors could share the revised wording of L281-284. I would be happy to raise my score if this is done.

Thanks for the follow up! Indeed, our answer only applies to the preconditioned spectra. We’ve added a plot in Figure 19 (Appendix G) which computes the signal ∣E[g]∣2\\|E[g]\\|^2 and noise E∣g−E[g]∣2E\\|g - E[g]\\|^2 for each unembedding row at B=1024 in order of decreasing token frequency. The plot shows that the most frequent tokens are signal dominated while the less frequent tokens are noise dominated, and the total SNR is 5.1.

We can also predict these curves by modelling the gradients as Bernoulli random variables, where qvq_v is the marginal of token vv under the model, and pvp_v is the marginal under the true distribution. Then the variance is roughly Dpv(1−pv)/(BL)D p_v(1-p_v)/(BL) and the mean squared is D(qv−pv)2D(q_v - p_v)^2, where DD comes from ∣h∣2=D\\|h\\|^2=D for hh being the input to the unembedding matrix after the RMS norm.

Regarding the non-preconditioned head, we do not claim it is vanishing at B=1024 in Figure 7, as there is still a dip at index VV, in comparison to the preconditioned spectrum. We have amended the current draft at lines 281-284: “One hypothesis for why the preconditioned head spectrum vanishes for B=1024 is that due to the block diagonal structure (rank-1 blocks) of the Hessian of the top layer under neural collapse (Papyan et al.), Adam in full batch is able to completely whiten the spectrum of this matrix (Liu et al.). Note that the signal dominates the noise as shown in Figure 19, with a high total SNR. This hypothesis is consistent with the small batch result, where we only see a square root preconditioning from Adam, following (Liu et al.).”

Thank you, the revised wording and Figure 19 address my concern, and I will raise my score.

Reviewer_5qBiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work proposes a quadratic model as a tool to explain the trainings of LLMs. Despite the actual training of LLMs being done on a highly nonconvex loss landscape, the authors argue that quadratic models are sufficient in capturing certain aspects of the training, and sometimes (especially around the end of the training run) be very well-predicting the training dynamics, where the quadratic proxy and the actual training agreeing around 5 to 10 percent of the total training duration.

Critical strengths

The work provides a quadratic proxy of the full model, which enables a tractable analysis on its own. Empirical results show that such a model can be useful in certain cases, matching the training dynamics much more than one might expect. Thus, conditional on the assumption that the proxy is a sufficiently explanative alternative to the full model, this work opens a new path toward a better understanding of the training dynamics of LLMs, which have been poorly understood so far.

Critical weaknesses

For the claimed "defense" to function, the ultimate premise should be that the quadratic model does in fact give a good approximation of the actual training dynamics. This, however, seems to be the weakest link. For constant learning rates, the quadratic proxies do not sufficiently match the actual training, and when cosine decay is used, the quadratic proxy becomes meaningful only near the end of training. The Hessian analyses in Section 3 also show that the Gauss-Newton matrix, used in the quadratic model, is quantitatively distinct from the actual Hessian. Individual studies on the Gauss-Newton and the Hessian matrices can be meaningful, but the results weigh against the "defense" of the quadratic model. The results in Section 4 are also somewhat interesting on their own, but the section's objective is also unclear in the sense that it does not explain why the proposed quadratic proxy should be studied.

Questions for authors
  1. The analyses in Section 4 seem to rely on well-known results, as the quadratic model reduces the updates to linear recurrences. Also, it seems unclear how these EoS analyses of the quadratic model help us understand the actual training dynamics. What is the main takeaway of this section?
  2. The agreement between the quadratic proxy and the actual model seems to be most visible around the end of training, when cosine decay is used. But this also raises the question of whether the agreement is due to a better approximation at later checkpoints, or rather it is just a trivial consequence of the stepsize being so small that Taylor approximation is just working well in that regime.
  3. Can the authors elaborate further on why the Gauss-Newton matrix would be sufficient for approximating the actual training dynamics? Negative curvature can play an important role in training (such as promoting avoidance of local saddle points), so completely ruling them out, especially in the early to middle stages of training, seems to be an oversimplification.
  4. Several key quantities in Section 3.2, such as v0v_0, μ(v0)\mu(v_0), and ϑj\vartheta_j, either lack a clear definition or are defined only in the appendix, which makes the section hard to follow. It would be helpful if those missing definitions were properly included in the main text. Also, the definition of i⋆(λ)i^\star(\lambda) as an "index" seems unclear for an arbitrary threshold λ\lambda. In addition, please also clarify how the probe-weighted spectral mass is related to the actual eigenvalue index, how the “best fit curve” is constructed, and what is meant by “small multiplicative error.”
  5. A minor note: VV is used ambivalently in Section 3.3; in line 272 it is used to denote the basis from algorithm 1, and in line 279 it is used to denote the vocabulary size. I suggest amending the notation.
3 comments

We thank the reviewer for their comments. We will address each weakness mentioned by the reviewer pointwise.

For constant learning rates, the quadratic proxies do not sufficiently match the actual training, and when cosine decay is used,  the quadratic proxy becomes meaningful only near the end of training

While for quad it is true that the losses do not agree, for prox the agreement is much better. We speculate that this is because for constant lrs, the model stays at the edge of stability during training (Figure 15), whereas for cosine the model leaves the edge when the cosine decays towards the end (but still stays within a factor of 2 of the edge). Prior work [1] has shown that when a network trains at EoS, there is a cubic regularization term which pushes down the curvature, causing the dynamics to deviate from the quadratic approximation. This nonlinearity is partially captured by prox, which may be why it shows better agreement. However, this is all conjectural and we leave a rigorous analysis to future work.

that it does not explain why the proposed quadratic proxy should be studied.

The main motivation is that when the quadratic approximation holds, we have a lot of tools in linear algebra to study the resulting dynamics and reason about loss curves, scaling laws, hyperparameters, etc. Moreover, from a more general scientific perspective, we believe that the best way to understand a system is to first fit the simplest possible model to it, and then observing where this model breaks can tell you where to look next.

On the questions section:

  1. A fundamental question that we can ask to understand about LLM optimization is what regime are we operating in. This has immediate implications for interpreting the hyperparameters (i.e. are we in a learning rate dominated, corresponding to a deterministic EoS, or temperature dominated regime corresponding to a stochastic EoS [2]), instabilities and loss spikes during training, and training efficiency (by characterizing the critical batch size [3]).

  2. The agreement between the losses is likely not due to a smaller step size. At 70% of training, the agreement is for around 1% of samples (per Table 1) and step size fraction of the peak lr is 0.320.32. At 80% however, the agreement is much larger, 5.5% of samples with the lr fraction 0.20.2. So the ratio between the learning rates at 7070% and 8080% is roughly 1.6, which would predict an agreement 1.6%, if the agreement were proportional to the step size. While the step size likely has a small impact, we believe the disagreement may be caused by the edge of stability effect explained above.

  3. The logic in our paper is that because the Hessian of quad is the Gauss-Newton matrix, we can understand the dynamics of quad through the spectrum of the Gauss-Newton matrix, and we include the Hessian spectrum in Section 3 for completeness. The tension you raised comes from the assumption that the full quadratic Taylor expansion of the loss (using the Hessian instead of the Gauss-Newton) would be a better approximation for the dynamics. However, we have no evidence that this is the case. For example, the Hessian has some very negative eigenvalues, and a full quadratic Taylor expansion of the dynamics would predict that in very few steps the loss would decrease exponentially towards −∞-\infty. We therefore believe it is plausible that the Gauss-Newton spectrum could be more informative for the overall dynamics than the Hessian, and this is supported by our linearization experiments which only rely on the Gauss-Newton.

  4. Thank you for bringing this up. We define for any λ\lambda, i∗(λ)i^*(\lambda) as the count of eigenvalues at least as large as λ\lambda. The probe weighted spectral mass directly defines the eigenvalue index: the i-th eigenvalue is defined by having i−1i-1 eigenvalues greater than it, which corresponds to the CDF of the empirical spectral density. The best fit curve is chosen to be equidistant from the two Gauss-Radau error-band boundaries. We refer to “small multiplicative error” for an estimated eigenvalue λ^i\hat{\lambda}_i for the true eigenvalue λi\lambda_i as ∣log⁡(λ^i/λi)∣≤ϵ|\log(\hat{\lambda}_i / \lambda_i)| \leq \epsilon for a small ϵ\epsilon. We will make this clear in the paper.

  5. We will fix the notation, thank you for pointing this out.

[1] Damian, Alex, Eshaan Nichani, and Jason D. Lee. "Self-stabilization: The implicit bias of gradient descent at the edge of stability." arXiv preprint arXiv:2209.15594 (2022).

[2] Mulayoff, Rotem, and Tomer Michaeli. "Exact mean square linear stability analysis for SGD." The Thirty Seventh Annual Conference on Learning Theory. PMLR, 2024.

[3] Zhang, Hanlin, et al. "How does critical batch size scale in pre-training?." International Conference on Learning Representations. Vol. 2025. 2025.

Thank you for the rebuttal. Unfortunately, the rebuttal did not address most of my concerns.

I think the main point we disagree on is why we should trust the proposed quadratic model and how exactly it helps us understand the actual training dynamics of LLMs. I agree with the explanations of the main motivation per se. But the phrase "when the quadratic approximation holds" is doing a lot of work here. The approximation is generally poor across the training run. Quadratic models turn out to be useful around the end of training, and the authors suggest that training being in the EoS regime is the reason the quadratic model breaks. Putting aside that this is conjectural, this works against their argument for why the quadratic model should be trusted in the first place.

Moreover, why should we believe that a less accurate local approximation using the Gauss-Newton matrix is better? A second order Taylor approximation being bad does not automatically make the Gauss-Newton quadratic term valid. If the quadratic term in the approximation is not exactly the Hessian, then the gradients already start to mismatch as we move away from the checkpoint. On top of that, the agreement of quad is still bad. What I can only guess from here is that quadratic approximations typically hold only quite locally, so higher-order terms may be needed to explain the dynamics beyond that local regime. If we accept the latter, then EoS would make more sense.

I also strongly agree with the point that "the best way to understand a system is to first fit the simplest possible model to it", but this is also exactly why I am so disappointed with this work. The quadratic model is chosen for its tractability, but does not tell us much about the "loss curves, scaling laws, hyperparameters", and is itself not a satisfactory approximation. When the model falls short, the authors conveniently blame EoS for the failure. Then, why not study EoS directly in the first place? Where is the “defense” of the quadratic model?

In addition, in the rebuttal to Q2, I don't understand why the agreement window should be inversely proportional to the learning rate. There is no reason to assume that proportion holds, so dismissing a smaller learning rate as the explanation is premature. Moreover, if the authors insist that EoS is actually at work here, shouldn't they also account for a smaller learning rate moving the training dynamics away from the "edge" of stability and thus improving the agreement?

The phrase “when the quadratic approximation" is doing a lot of work here. [...] the agreement of `quad` is still bad.

We agree it’s doing a lot of work, and the main purpose of Section 2 is testing it. However, we disagree that the agreement of quad is bad. For the cosine runs in Figure 1 and Table 1, both prox and quad successfully predict the training curves over 5-10% windows near the end of training, which was already striking to us.

We also completely agree that the quadratic approximation holds only in some local region in weight space, and the central question is how long it takes Adam to leave this region. Because the movement of Adam per step is proportional to the learning rate, the simplest heuristic is that halving the learning rate should double the number of steps to travel the same distance. However, as you pointed out, this would also cause training to leave the edge of stability, and we believe both effects contribute to the superlinear increase in agreement time. Our experiments do not yet disentangle these two effects, and we’re currently investigating these directions as a follow-up.

On GN vs. the Hessian: the Hessian approximation must fail, since its negative eigenvalues quickly drive the loss to −∞-\infty. We agree there is no a priori reason the GN approximation should succeed, which is why we treat it as a hypothesis and test it empirically in Section 2. One plausible explanation, but purely speculative, is that the non-GN term of the Hessian is weighted by the residual pθ(x)−yp_\theta(x)-y, which oscillates in sign during training e.g. see reference [1, Fig. 2(d)]. Its contribution could then average out over many steps, so the non-GN term at any fixed iterate can be misleading.

[1] Alain, Guillaume, Nicolas Le Roux, and Pierre-Antoine Manzagol. "Negative eigenvalues of the Hessian in deep neural networks." arXiv preprint arXiv:1902.02366 (2019).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.