- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates whether local quadratic models can accurately describe the optimization dynamics of LLM pretraining. Through experiments on a 150M-parameter Transformer, the authors show that Gauss–Newton quadratic approximations can predict the training loss trajectory over relatively long windows, particularly in the later stages of training. The paper further analyzes the Hessian spectrum, gradient–Hessian alignment, and edge-of-stability behavior, revealing interesting structures and connections to existing optimization theory.
Critical strengths
- Quadratic models are widely used in optimization theory, but their relevance to practical neural network training remains insufficiently understood. This paper directly investigates this fundamental question, helping bridge the gap between theoretical optimization models and practical LLM pretraining.
- The observation that local quadratic approximations can accurately predict the loss trajectory over a nontrivial fraction of training is surprising and encouraging. The findings provide empirical support for using quadratic models to understand neural network optimization.
- Beyond validating the quadratic approximation, the paper investigates the Hessian spectrum, eigenvector structure, gradient–Hessian alignment, and optimization stability. These complementary analyses provide useful insights into LLM pretraining dynamics.
- The authors invest substantial computational effort in estimating the Hessian spectrum using deep Lanczos probes. The numerical validation and additional experiments in the appendix strengthen the credibility of the spectral analyses. Overall, the paper is well motivated and presents its findings in a clear and organized manner.
Critical weaknesses
- All experiments are conducted on a single model architecture at approximately 150M parameters. While the experiments are computationally demanding, it remains unclear whether the findings generalize to larger LLMs, different architectures, or other optimizers. This is particularly important given the paper's broader motivation of understanding LLM pretraining.
- As shown in Figure 1 and Table 1, the approximation is considerably less accurate during early training, and its performance also depends on the learning rate schedule. In particular, the constant-learning-rate experiments show substantially less reliable agreement. These observations suggest that the applicability of the quadratic model may be more restricted than the paper's broad conclusions imply.
- Although the empirical agreement between the quadratic approximation and the original training dynamics is interesting, the paper provides limited insight into why this agreement emerges in later training stages. For example, it is unclear whether this behavior is primarily explained by reduced learning rates, changes in the Hessian, or other properties of the optimization trajectory.
- In the stability experiments, the authors freeze the Adam preconditioner and analyze preconditioned SGD on a convex quadratic approximation. The experiments also filter extreme outliers to avoid numerical instability.
Questions for authors
- In Section 2, the agreement between the quadratic approximation and the original model is evaluated primarily through validation loss. Have the authors also examined the similarity of the resulting parameter updates or optimization trajectories? It would be interesting to understand whether similar loss curves correspond to similar trajectories in parameter space.
- In Table 1, the authors report prediction windows starting from the 100% training checkpoint. Could the authors clarify how the learning rate schedule is handled when continuing beyond the original training budget?
- In Section 3.2, the authors observe that the spectral tails are largely invariant to batch size and preconditioning, up to rescaling. Is there a possible theoretical explanation for this observation, or any additional evidence suggesting when such universality should be expected?
2 comments
Thanks for the clarification. I think this work is valuable. I have raised my rating to 3.
Reply
We thank the reviewer for their kind words about our work. We begin by addressing the weaknesses they raised:
We have run experiments for the spectrum measurement at scales from 9M - 1B (which we have now added to the manuscript in Appendix G, Figure 18), and we are currently investigating longer training runs (up to 32x Chinchilla). However, given that the spectrum does not change much after warmup (Figure 2), we expect the results will remain the same.
We believe that the agreement between the losses is unlikely to come from a smaller step size. Per Table 1, at 70% of training the losses agree on about 1% of samples, with the learning rate at 0.32 of its peak. At 80% of training, the learning rate is 0.2 of its peak, but agreement jumps to 5.5% of samples. The learning rate drops by a factor of about 1.6 between these two points. If agreement scaled with step size, we would expect only about 1.6% agreement at 80%. We conjecture that the main disagreement comes from the network being at EoS during training. From prior work, [1] when a network stays at EoS, there is a cubic regularization term which pushes down the curvature, causing the dynamics to deviate from the quadratic approximation. This nonlinearity is partially captured by
prox, which may be why it shows better agreement. The step size likely has some effect, we believe it is not the main reason behind the disagreement.We refer to the answer from weakness 2, and speculate that later in training the agreement improves because the model leaves EoS for cosine decay as per Figure 5, which is why the agreement improves for
quad, whereas for constant the model stays at EoS during training (see Figure 15), andquaddoes not match throughout.We would like to point out that this is a necessary methodological choice and not a weakness. One important takeaway from our work is that we can linearize the model itself, but not the Adam optimizer as it has to handle these rare divergent events through nonlinear damping. However, if we freeze the optimizer, filtering out these outlier events suffices for maintaining the underlying SGD dynamics, which are stable (depending on η and B). This also allows us to conclude that, at least with regards to stability, one of the main roles of Adam is handling these outliers. Furthermore, this is the standard definition of EoS for Adam, as established by [2].
Regarding the questions:
We thank the reviewer for the insightful question. We did not measure parameter agreement, as it is not obvious what the correct metric to compare the weights would be in: for example, L2 norm would be dominated by the difference in params with large magnitude. We also want to emphasize that in some sense the metric we have chosen is the strongest criterion possible, because we take the parameters trained using the linearized dynamics, and plug them back in the nonlinear model for evaluation. A simpler criterion would be to just show the validation loss of the linearized model itself, but this would be a less honest evaluation.
We apologize for not explaining this aspect. For all the intermediate checkpoints we follow the respective scheduler (cosine or constant), and for the checkpoints starting at 100%, we keep the learning rate fixed at the end learning rate for that scheduler.
This is indeed a good question. One hypothesis is that the bulk of the spectrum is in a random or incoherent basis, and Adam is only good at whitening Hessians that are approximately diagonal. It would be interesting to explore whether this is still true for optimizers that work in different bases, such as Muon, SOAP or Shampoo, but we leave to future work.
[1] Damian, Alex, Eshaan Nichani, and Jason D. Lee. "Self-stabilization: The implicit bias of gradient descent at the edge of stability." arXiv preprint arXiv:2209.15594 (2022).
[2] Cohen, Jeremy M., et al. "Adaptive gradient methods at the edge of stability." arXiv preprint arXiv:2207.14484 (2022).