acceptodds
Withdrawn from ICLR 2027

Simulating Training Spikes of Deep Neural Networks by Layer-wise Random Feature Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Intermittent “spikes” in loss, during the pre-training of Large Language Models (LLMs), remains a mysterious phenomenon both theoretically and empirically. In this paper, we show that simple quadratic models with heavy-tailed, power-law distributed inputs can replicate these spikes. We introduce a theoretical framework and simulation algorithm to model per-layer training dynamics of Deep Neural Networks (DNNs), using such quadratic models. In a case study on an Adam hyperparameter configuration (where is set close to ) known to induce spikes in Transformer LMs, our simulation successfully predicts bumps indicating spiky behavior from a given checkpoint. Furthermore, we derive an appropriate schedule from observed metrics, which empirically improves performance.

Primary area:
optimization
Keywords:
training spikes, stochastic optimization, random feature models, power-law

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PJ6MOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies training-loss spikes using power-law random-feature models and proposes a checkpoint-calibrated simulator for layer-wise neural-network dynamics. It connects spikes to stochastic gradient noise and spectral concentration, with experiments on synthetic models and a 1.3B-parameter Transformer trained on C4. It also proposes a measurement-informed Adam β2\beta_2 schedule and evaluates its effect on training stability and validation loss.

Critical strengths
  1. Interpretable modeling framework. Section 2 and Figure 2.1 demonstrate spike-like behavior in simple quadratic random-feature models under SGD and Adam. These controlled experiments provide a useful setting for studying how spectral structure and stochastic optimization relate to training instability.

  2. Concrete and lightweight simulator. Section 3.3 and Algorithm 1 translate checkpoint measurements into a procedure for simulating layer-wise dynamics. After calibration, the simulation is computationally inexpensive: Table 1 reports five seconds on CPU for 1,500 steps of one layer.

  3. Substantial Transformer case study. Section 4 evaluates a 1.3B-parameter Transformer trained on C4, with controlled changes to Adam’s second-moment parameter. Figures 1.1 and 4.1 connect the synthetic investigation to observed training instability and measured layer-wise behavior.

  4. Empirical checks of modeling assumptions. Appendix D.2 examines activation spectra, while Appendix D.3 tests the scalar gradient-norm relationship in Equation (6). These measurements help assess the plausibility of specific assumptions underlying the simulator (Figures D.2 and D.4).

Critical weaknesses
  1. The gradient approximation lacks sufficient theoretical justification (major). Section 3.2, Eq. (5) neglects dependence terms that can remain significant when output gradients align with input eigenvectors. Additional conditions or error bounds are needed. Appendix D.3 validates only the scalar relation in Eq. (6), leaving the full second-moment approximation insufficiently supported.

  2. Real-model experiments provide limited evidence of predictive accuracy (major). Section 4 evaluates six layers with simulations initialized at one training stage. Figure 4.1 compares only second-order loss changes, while Appendix D.6 acknowledges unexplained first order changes and different recovery behavior. Quantitative predictions from previously unused checkpoints, tested across runs, would better establish whether the simulator reliably identifies future spikes.

  3. Quantitative simulation accuracy is limited at larger batch sizes (moderate). Section 3.3’s simulator reproduces qualitative spike behavior, but its quantitative accuracy varies across settings. At batch size 64, simulations spend 36–38% of steps above threshold versus 3.5% for PLRF (Appendix C, Table 2). The authors acknowledge this limitation; the main paper should clarify when accurate spike statistics can be expected.

  4. The spectral explanation needs stronger validation in the Transformer (moderate). Section 3.4 proposes a plausible mechanism supported by Section 2’s synthetic experiments. However, Section 4 changes β2\beta_2, which affects several aspects of optimization, without isolating the contribution of spectral concentration. More targeted evidence would strengthen the connection between the proposed mechanism and the observed Transformer behavior.

  5. Generality beyond one Transformer configuration remains untested (minor). Section 4 evaluates one 1.3B Transformer on C4, without comparisons across architectures, model sizes, or training-data budgets. The synthetic sensitivity experiments do not establish broader real-model generality. Given the case-study framing, this is a minor limitation, but broader applicability claims should remain qualified.

  6. The simulator’s value for selecting preventive interventions is unclear (moderate). The optimizer guidance presented in the Introduction is not evaluated through interventions selected using Section 3.3’s simulator. Instead, the schedule is fitted to observed metrics (Appendix D.5). Comparing simulator-selected settings with direct metric-based tuning and tuned constants on independent training continuations would clarify its added diagnostic value.

  7. The optimizer schedule’s practical advantage is not established (moderate). The proposed schedule improves validation loss by only 0.0001 over the best tested constant β2\beta_2, without reported uncertainty (Appendix D.5). This does not establish a reproducible advantage over a well-tuned constant setting. Practical comparisons should also account for the calibration costs reported in Table 1.

  8. Missing targeted ablations limit attribution of the results (moderate). Sections 2–4 include useful sensitivity experiments, but additional controls are needed to separate Gaussian sampling from the gradient-moment approximation, spectral shape from overall scale and proximity to instability, and the proposed β2\beta_2 rule from simpler schedules with comparable ranges and average decay. Matched comparisons across seeds would clarify which mechanisms drive the simulated behavior and optimization benefits (Section 3.3; Appendix D.5).

Questions for authors
  1. Under what conditions can the dependence terms in Section 3.2, Eq. (5), be neglected? Could you provide an error bound and empirical checks of the full second-moment approximation, beyond the scalar check in Eq. (6)?

  2. How do Figure 4.1’s second-order comparisons support prediction of actual validation-loss spikes? Could you report predictive results across additional checkpoints and runs, including missed spikes and false alarms, and clarify the discrepancies in first-order changes and recovery behavior?

  3. Under which batch-size conditions does the simulator accurately reproduce spike statistics? Could you report errors in spike magnitude, frequency, and duration across independent synthetic runs to help identify its reliable operating range?

  4. Could you provide spectral-shape comparisons controlling trace, sharpness, and distance to instability to help isolate the role of spectral concentration in Figure 2.1? Which observations from the Transformer experiments support the same mechanism?

  5. Which assumptions limit applicability beyond the Transformer configuration in Section 4? Could you clarify the supported scope across architectures, model sizes, and data settings, including any available broader evidence and conclusions currently limited to this case study?

  6. Does the simulator improve intervention choices beyond direct metric-based tuning? Where feasible, could simulator-selected settings be compared with tuned constants and simpler indicators through short, matched continuations from an existing checkpoint, considering both stability and calibration cost?

  7. Is the schedule’s 0.0001 improvement consistent across independent training runs? Could you report the mean and standard deviation of final validation loss for the proposed schedule, the best fixed β2\beta_2, and a simple schedule with comparable range and average decay?

  8. Which approximation causes the errors in Table 2? Could you compare actual minibatch gradients, Gaussian gradients with exact state-dependent mean and covariance, and Gaussian gradients using the paper’s approximated moments, under matched settings, and report their spike frequency, magnitude, and duration?

  9. How far in advance can the simulator provide a useful warning? Could you report prediction quality across forecast horizons, indicate when recalibration becomes necessary, and compare warning times with simpler instability indicators?

  10. Can the simulator identify which layers contribute most to instability? If feasible, could interventions on predicted high-risk layers be compared with matched interventions on other layers to assess whether the layer-wise diagnosis supports effective, targeted prevention?

5 comments
AuthorsOpenReview ↗

Re: Critical Weakness #1

Appendix D.3 validates only the scalar relation in Eq. (6), leaving the full second-moment approximation insufficiently supported.

We present a "per-eigenmode verification" in this colab: https://colab.research.google.com/drive/19fOfDvr6TAp9a7PhZ-wo1Ndm8-cetVNB?usp=sharing

Combining this with the already existing scalar verification (i.e., verification of relation Eq.(6) in Appendix D.3), we should have covered the full second-moment approximation that is asked by Reviewer PJ6M, which should be enough to theoretically support our simulation algorithm.

dependence terms can remain significant when output gradients align with input eigenvectors

We would like to discuss a bit further on this intuition. Using the notation in the paper, let A=UDiag(λ)V⊤\mathbf{A}=\mathbf{U}\textbf{Diag}(\sqrt{\boldsymbol{\lambda}})\mathbf{V}^{\top} be the singular value decomposition of A\mathbf{A}. It is true that the "dependence terms" can "remain significant" when output gradients align with eigenvectors in V\mathbf{V}. This can be seen by taking the expectation of Eq.(11) in the paper, in Appendix B.2. Where you have probably get confused, is that there are two sets of "eigenvectors", U\mathbf{U} and V\mathbf{V}, in the above singular value decomposition. By conflating U\mathbf{U} and V\mathbf{V} in one concept as "input eigenvectors", you thought the per-eigenmode version of Prediction 5 is against your intuition -- But, note that Eq.(5), the per-eigenmode version of Eq.(6), is purely based on the eigen decomposition defined by U\mathbf{U}. Because U\mathbf{U} is irrelevant to V\mathbf{V}, it almost never "aligns with output gradients". This is why when Eq.(6) is empirically verified, we can almost be sure that the per-eigenmode version Eq.(5) is likely true. The "dependence terms" can "remain significant" in the sense that, its overall norm indeed occupies a non-zero portion of E(∥∇BW∥2)\mathbb{E}(\lVert\nabla_{\texttt{B}}\boldsymbol{W}\rVert^2), which causes errors that are already checked when we verify Eq.(6) in Appendix D.3 -- you have already witnessed the most essential step. But that "dependence terms" cannot really affect the eigenmodes defined by U\mathbf{U}. Please take a deeper thought on this one, and if you have other related questions, I'm here and happy to answer them.

AuthorsOpenReview ↗

Full Reply to Questions and Critical Weaknesses

Question #1 (related to Critical Weakness #1): Under what conditions can the dependence terms in Section 3.2, Eq. (5), be neglected? Could you provide an error bound and empirical checks of the full second-moment approximation, beyond the scalar check in Eq. (6)?

The colab presented above, https://colab.research.google.com/drive/19fOfDvr6TAp9a7PhZ-wo1Ndm8-cetVNB?usp=sharing , should answer this question. We have provided empirical measurements of the error. Theoretically, for the case of random feature models one can prove that the error tends to 0 as the dimension goes to infinity. We have provided a sketch of the proof in the paper as well (Appendix B.2).

Question #2 (related to Critical Weakness #2): How do Figure 4.1’s second-order comparisons support prediction of actual validation-loss spikes? Could you report predictive results across additional checkpoints and runs, including missed spikes and false alarms, and clarify the discrepancies in first-order changes and recovery behavior?

Is the colab https://colab.research.google.com/drive/19qjgh8Je16FcZJrYnNREN3QRj4iFwUWe?usp=sharing presented in our top-level comment what you ask for? The colab combined with our discussion in the paper should have documented most we know about the proposed simulation. It is what it is. We do not intend to provide a "recommended use-case" for the proposed simulation algorithm in the paper; one probably needs several extra components for empirical applications, which are strictly out of scope of this work. If merely for the empirical usage of indicating unhealthy training, several metrics that the simulation relies on, such as the distribution of the eigenvalues and the power-law exponent alpha, should be predictive already.

Question #3 (related to Critical Weakness #3): Under which batch-size conditions does the simulator accurately reproduce spike statistics? Could you report errors in spike magnitude, frequency, and duration across independent synthetic runs to help identify its reliable operating range?

The theory suggests that the condition is B≪D\texttt{B}\ll\sqrt{\texttt{D}}. Specific to our example, D=1024\texttt{D}=1024, so the batch-size B≪32\texttt{B}\ll 32 is the condition. This kind of explains why at B=64\texttt{B}=64 the simulation shows different spiking statistics. The condition B≪D\texttt{B}\ll\sqrt{\texttt{D}} appears in several places in our theory (e.g., on Page 4, 20), and we will make it more explicit in our revision of the paper.

Question #4 (related to Critical Weakness #4): Could you provide spectral-shape comparisons controlling trace, sharpness, and distance to instability to help isolate the role of spectral concentration in Figure 2.1? Which observations from the Transformer experiments support the same mechanism?

Is Figure D.2 (in Appendix D.2) what you are asking?

Question #5 (related to Critical Weakness #5) Which assumptions limit applicability beyond the Transformer configuration in Section 4? Could you clarify the supported scope across architectures, model sizes, and data settings, including any available broader evidence and conclusions currently limited to this case study?

Question #6 (related to Critical Weakness #6) Does the simulator improve intervention choices beyond direct metric-based tuning? Where feasible, could simulator-selected settings be compared with tuned constants and simpler indicators through short, matched continuations from an existing checkpoint, considering both stability and calibration cost?

Question #9 (related to Critical Weakness #6) How far in advance can the simulator provide a useful warning? Could you report prediction quality across forecast horizons, indicate when recalibration becomes necessary, and compare warning times with simpler instability indicators?

Question #10 (related to Critical Weakness #6) Can the simulator identify which layers contribute most to instability? If feasible, could interventions on predicted high-risk layers be compared with matched interventions on other layers to assess whether the layer-wise diagnosis supports effective, targeted prevention?

I cannot tell you that. (Seriously!) The paper is a case-study, and these topics are strictly beyond the scope. If the reviewer really wants to know, we need to figure out a way to collaborate, or the reviewer uses their own resources to find out.

Does the simulator help issue interventions when there are training spikes? Not really. No intervention really works when there are already training spikes. The simulator does help understand normal training and can be informative in figuring out the best hyperparameters; but that also requires many other components to work together.

AuthorsOpenReview ↗

Full Reply to Questions and Critical Weaknesses (cont.)

Question #7 (related to Critical Weakness #7) Is the schedule’s 0.0001 improvement consistent across independent training runs? Could you report the mean and standard deviation of final validation loss for the proposed schedule, the best fixed, and a simple schedule with comparable range and average decay?

Please see the report of our additional experiments on different beta2 settings, presented above in a top-level comment.

Question #8 (related to Critical Weakness #1) Which approximation causes the errors in Table 2? Could you compare actual minibatch gradients, Gaussian gradients with exact state-dependent mean and covariance, and Gaussian gradients using the paper’s approximated moments, under matched settings, and report their spike frequency, magnitude, and duration?

I think one misunderstanding that the reviewer might have, is that the discrepancies in spiking statistics as reported Table 2, are somehow caused by approximation errors in Prediction 5 -- they are not.

Any approximation errors in Prediction 5 will directly cause the simulation to deviate from the average case behavior of the loss. But as you can see, the average case behavior matches (i.e., the underlying flows of the curves match, when ignoring spikes), even at batch-size=64. So there are really no approximation error from Prediction 5, for high-dimensional random feature models. We showed that the empirical error is mostly less than 5% for Transformer LM; but it should be less than 1% for PLRF.

What could cause the discrepancies in spiking statistics, is the short-term gradient noise distribution being not Gaussian. This is the only reason. The two points I wrote in the paper, are meant to be the two factors that make the accurate theoretical calculation of this distribution difficult -- I don't know how to calculate the short-term gradient noise distribution at all. The comparison we present in the paper, is really the best we can do for this observation. There is no further decomposition into separate factors.

Reviewer_hk89OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a theoretical framework to study "training spikes", the sudden jumps in loss seen when training large language models. They replicate the training spikes in simpler linear regression model on random features with power law-scaled inputs (PLRF) with SGD or Adam.

They claim that training spikes happen under two conditions: the input data is heavy-tailed (a large power-law exponent α), and the learning rate is close to the point where training would diverge. The reasoning is that gradient noise normally averages out in high dimensions ("concentration of measure"), but this breaks down when the eigenvalues are too heavy-tailed, leading to training spikes.

To support the theory, they build a simulation that treats each layer of a real Transformer as one of the simple models. On a 1.3B Transformer model where for Adam, beta_2 is set close to 1, the simulation successfully predicts training spikes and finally they propose an appropriate beta_2 schedule, which gives a slightly lower final loss.

Critical strengths
  • Core observation is interesting and easy to check, a simple, convex model can show spikes that look like real LLM spikes. this challenges the common view that spikes can only occur in complex curvature of deep networks
  • The explanation, that noise stops averaging out when a few directions dominate, is intuitive and gives something testable.
  • The simulation is very cheap
  • Code is provided for the simple-model experiments.
Critical weaknesses
  • Introduction mentions the motivation about training spikes signalling instability, poor hyperparameters or worse final model, but these claims aren't supported with citations. The experiments also only partly address them showing spikes are linked to a non-default beta_2, and a single spiky run (beta_2 = 0.992, Table 3) ends slightly worse. As a result, its not clearly established why studying training spikes matter beyond the fact that they happen.
  • Final losses for the runs with training spikes v/s the normal run aren't reported and in Fig 1.1a, the curves end up looking similar. One related evidence in Table 3, where a run with training spikes ends slightly worse (beta_2 = 0.992), but that run uses a different beta value, so it can't separate spikes from different hyperparameter
  • the term "heavy-tailed" is never defined and is used confusingly, as the inputs are ordinary gaussians, so the data isn't heavy tailed in the usual sense. larger alpha value is paired with heavier-tailed which is not the usual interpretation. the paper should define the term precisely and mention whether it refers to data or some other entity.
  • The benefit of linking training spikes to “heavy-tailed data” isn’t argued in the main text. The paper claims to be the first to make this link, but the practical payoffs appear only in Appendix A and aren’t demonstrated.
  • Large α and a near-divergence learning rate are stated as observations from Fig 2.1 without explanations, with reasoning only mentioned in section 3.4 without any reference to it. Also, only two values of alpha (0.3 and 0.9) are tested, so the claim threshold of alpha >0.5 is not tested directly, this needs a test on a tighter window like 0.4-0.6. The regression target is also not specified in the main text, and the loss scales of the two alpha values differ by about 100×, which makes them hard to compare.
  • Turning a transformer layer into a simpler model is based on several assumptions which are not directly tested.
  • the case study uses one model, one checkpoint and two simulation runs and the bumps in Fig 4.1 are small and judging by the eye, they don't seem to occur when the real spikes do.
  • In Table 3, the proposed schedule scores 2.4404, while a plain constant beta_2=0.982 scores 2.4405. this result is on a single run, with no seed variation accounting for noise, weakening the "improves performance" claim.
  • Overall the presentation of the paper is not clear, terms are introduced without any explanation, important results listed in the abstract are only present in the appendix.
  • Appendix A argues for the similarity with related works “from an aesthetics perspective” and dismisses prior work on visual grounds, in an informal tone (“I’m sorry, but they are not similar”). A quantitative comparison would be more convincing, and the tone should be revised.
Questions for authors
  • Define "heavy-tailed" precisely and mention whether it means data or something else?
  • what are the final validation loss for Spiky 1 and 2 runs compared to Normal? is there evidence that spike themselves cause worse results?
  • In Table 3, how many seeds are behind the results, what is the seed to seed variation?
3 comments
AuthorsOpenReview ↗

Re: Question #1

In this paper, the term "heavy-tailed distribution" is used interchangeably as "power-law distribution with exponent (corresponding to 2 * alpha in the PLRF notation) close to, or larger than, 1". It means the data.

The data of PLRF is defined as (i−αxi)(i^{-\alpha}x_i); that "i−αi^{-\alpha}" factor makes it power-law distributed. Yes, xix_i is normal Gaussian, so you are right, this is also a multi-variate anisotropic Gaussian distribution, of which the covariance matrix has eigenvalues distributed as a power-law. If one wants to be precise, one could say the spectrum of the data is power-law distributed. (Citation: https://arxiv.org/abs/2210.16859 Especially on Page 18, where it discusses the range of power-law exponents of PLRF, and says "This is actually a rather compact range and implies that natural datasets have surprisingly heavy tails.")

It is kind of unclear to the authors what the reviewer is complaining about, regarding this topic. Maybe the reviewer could explain to us first what their understanding of "heavy-tailed" is, so that we can better resolve any potential miscommunication?

AuthorsOpenReview ↗

Re: Question #2 and #3

Please see our report of additional experiments on the validation loss of different beta2 settings, in the top-level comment above.

AuthorsOpenReview ↗

Re: Critical Weaknesses

Introduction mentions the motivation about training spikes signalling instability, poor hyperparameters or worse final model, but these claims aren't supported with citations.

Training spikes "causing" worse performance is a folklore. We never intend to claim or prove that. But it is also true that training spikes are often thought to be correlated with worse performance -- at least, the surges in validation loss means that for a positive portion of time, the resulted checkpoint is really bad; only at steps where the model is not spiking, the checkpoint might become useful. This is the "instability" that worries people.

Regarding why training spikes are often correlated with worse final performance, https://arxiv.org/abs/2304.13013 posits that extra "recovery time" required by the spikes might slow down the learning. From our perspective, on the other hand, training spikes are indicators of heavy-tailed data distributions, and since Paquette et al. (2024) has already shown that different power-law exponents of the data can lead to different scaling parameters, it follows that training spikes are correlated with performance under scaling.

Our paper reads: "Empirically, these spikes are considered pathological because the increase in loss may indicate training instability, the hyperparameters being sub-optimal, or the final model less predictive." These are not claims we made. Also, all the four citations we give in this paragraph explicitly discussed how to "avoid" training spikes. Which means it is generally considered as something pathological.

Final losses for the runs with training spikes v/s the normal run aren't reported

Please find them in our top-level comment.

the term "heavy-tailed" is never defined and is used confusingly

We have discussed this in the "Re: Question #1" post above.

only two values of alpha (0.3 and 0.9) are tested, so the claim threshold of alpha >0.5 is not tested directly, this needs a test on a tighter window like 0.4-0.6.

I believe the colab https://colab.research.google.com/drive/1Sqme66YAZ_g6LcejknGiHtNAwQ3hHVp2?usp=sharing in our top-level comment, "Quantitative spiky metric and DNN simulation demo", has fully addressed this issue.

The regression target is also not specified in the main text

Not sure what you mean; I believe the regression target is given in Eq.(1), on Page 2 of the paper?

the loss scales of the two alpha values differ by about 100×, which makes them hard to compare.

The loss scales are proportional to the arbitrarily chosen hyper-parameter cc. So the absolute scales for different alphas are not supposed to be comparable. But the shape of loss trajectories can certainly be compared.

Turning a transformer layer into a simpler model is based on several assumptions which are not directly tested.

To be clear, Assumption 4 and Assumption 6 are tested in the paper in several ways. In Appendix D.2 we plot the evolution of eigenvalue distributions across layers and training steps, which is essentially testing Assumption 4, i.e., the "distribution of data" does not vary too much during training. As the results show, I would say that for Normal training this picture is reasonably acceptable; for Spiky training it is less accurate.

Not to mention that Prediction 5, an implication from Assumption 4, is directly verified in Appendix D.3.

Assumption 6 is partially tested in Appendix D.4, through our factorization of the "equivalent batch factor". The "third factor", measuring how far a DNN deviates from a Random Feature model, turns out to be relatively constant throughout training. This is essentially what Assumption 6 brings to us.

the case study uses one model, one checkpoint and two simulation runs and the bumps in Fig 4.1 are small and judging by the eye

I believe this has been addressed by our top-level comment above, "Quantitative spiky metric and DNN simulation demo".

In Table 3, ...

This should have been addressed by our top-level comment above, "More numbers for the validation loss of different beta2 settings".

Overall the presentation of the paper is not clear, terms are introduced without any explanation

Can you be more specific? I have specifically paid my attention to avoiding unexplained terms and making the paper self-contained. Is the term "heavy-tailed" one of your examples? Do you have other examples? It would be helpful if you let us know.

Reviewer_HGrcOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper shows that power-law random feature (PLRF) models trained with SGD or Adam produce random loss spikes near the stability threshold when alpha > 0.5, where alpha measures the "heavy tailed-ness" of the data distribution. It maps each DNN layer to a random feature model and builds a cheap gradient simulator. The simulator is applied to a 1.3B Transformer and used to derive a beta2 schedule.

Critical strengths

It is a clean and surprising observation that a convex quadratic model produces seed-dependent spikes, and code is provided. The paper separates data-order randomness from initialization randomness well. The simulator runs in seconds on a CPU, and the paper is open about where it fails.

Critical weaknesses

Overall while I find this type of theoretical exploration useful, I do find that a major weakness of the work resides in a somewhat self-fulfilling prophecy. The authors posit that the spike problem is important, and recurrent, whereas in my own practical practice it does seem to come rather from pathologies and lack of common-sense hacks that are immensely popular in the community, such as gradient clipping. The paper adopts somewhat an odd defensive tone (e.g. p. 17 which somewhat foresees this criticism in an overall fairly oddly worded appendix) but if the point is rather to rederive theoretically an optimization approach that recreates these spikes without offering a practical takeaway (something the authors resist, see "That said, the empirical recipes explored in this work such as β2 schedule, are intended as “proof of concept” for the theory, rather than alternatives to replace previous methods") then I feel I am a bit lost and unsure who this work is intended for. Optimization folks? Then it seems a bit light on content to create a synthetic setting in which gradient descent periodically diverges. If it's LLM practitioner, I think the lack of realistic experiment and suggested fix will puzzle them.

Detailed comments

  1. The theory is heuristic. Thm 3 is a first-order Taylor expansion, and the sense in which the model is "approximated" is never quantified. The claim that concentration fails for alpha > 0.5 needs to be reconciled with the deterministic-equivalent results of Paquette et al. (2024).
  2. Assumptions 4 and 6 (i.i.d. Gaussian latents, isotropic gradient covariance) carry the argument but are never checked on the Transformer. The batch factor B is fitted to the data, not predicted.
  3. The language-model evidence is one model, six layers and two seeds, with spikes judged by eye. The simulator also produces bumps for a run that does not spike, and it overestimates spikes by up to 10x at B = 64.
  4. The gain from the beta2 schedule is negligible (final loss 2.4404 vs 2.4405, single seed). Work on heavy-tailed gradient noise is not discussed.
Questions for authors

Can you report quantitative spike-detection rates across many checkpoints and seeds, against a lambda_max or update-RMS baseline? Can you check Assumptions 4 and 6 on the Transformer? Does the beta2 result hold across seeds and model sizes?

5 comments
AuthorsOpenReview ↗

Re: Critical Weakness #1 (Partial)

The claim that concentration fails for alpha > 0.5 needs to be reconciled with the deterministic-equivalent results of Paquette et al. (2024).

Alright. The reviewer probably couldn't quite understand the last step of the arguments that concentration fails for alpha > 0.5; (so they brought up something they also couldn't quite understand and feared that they might be related;) This is on us. I have created a colab to add more explanation: https://colab.research.google.com/drive/14VPkL2GJs17p_3jykV5Pm0S1XmE39Ga9?usp=sharing As you can see, it is actually quite elementary. Please let us know if you have more questions.

The deterministic-equivalent results of Paquette et al. (2024) is purely about the expectation of gradient noise; it never discusses concentration (i.e., convergence to expectation). When I brought up the spiky PLRF examples to Paquette et al. (2024), I believe they were also surprised. Then, people started to realize that concentration needs more conditions. So, theoretically, this is rather a new result, than a rephrase of existing theory.

The deterministic-equivalent technique is based on two ingredients:

(1) The limit spectrum of the data, i.e., (i−αxi)(i^{-\alpha}x_i) mapped by the feature matrix A\mathbf{A}. When the entries of A\mathbf{A} are i.i.d. Gaussian, this spectrum can be precisely calculated. For the purpose of this paper, however, we only need to know that the spectrum, after linear map by A\mathbf{A}, is still asymptotically power-law distributed of the same exponent. The exact form of the spectrum is not necessary. In other words, the deterministic-equivalent results of Paquette et al. (2024) depends on the specific condition that the entries of A\mathbf{A} are i.i.d. Gaussian. We do not need this condition for the concentration arguments. (Btw., https://arxiv.org/abs/2210.16859 has a detailed explanation on how to calculate the spectrum precisely, based on Feynman diagrams; just in case you are interested.)

(2) It applies Cauchy's integral formula (known as "resolvent" in spectral analysis) to sum values across (infinitely many) eigenmodes, which requires that the summands are holomorphic functions. This means any operation used in the optimizer has to be analytic -- things like clipping cannot be accurately handled by this method. We are not using it in this paper. (Fortunately, because it's f**king hard mathematically intense. It took me casually two months to roughly understand it)

AuthorsOpenReview ↗

Full Reply to Critical Weaknesses and Questions

I am a bit lost and unsure who this work is intended for.

This paper is mainly intended as a theoretical contribution. As I mentioned in the previous post, PLRF with α>0.5\alpha>0.5 not having concentration is a surprise to Paquette et al. (2024) as well, and since concentration is not discussed in Paquette et al. (2024), this paper adds something new to the theoretical picture, certainly more than just "rederiv[ing] theoretically an optimization approach".

More generally, I believe having examples of quadratic models spiking, raises questions regarding the empirical implications of almost all the worst-case theoretical bounds. The situation is, given a PLRF model with α>0.5\alpha>0.5, and by setting a large enough learning-rate, every stochastic optimizer is going to spike. So, if a worst-case bound is correct and applies to quadratic models (it almost always does), it should have already taken into account all the possible spikes. Do you feel one could say, "my optimizer is better because the largest spike it can achieve is smaller"? As now we know, depending on the eigenvalue distribution, there is a positive probability that the spikes can be arbitrarily large, no matter what optimizer you adopt! It certainly raises doubts against the general principle of using worst-case bounds as a guide to optimizer designs, doesn't it?

I'm not saying worst-case bounds are completely useless. Indeed, the worst-case bound for SGD is a necessary ingredient in Paquette et al. (2021) for rigorously proving the average-case behavior, because it serves as an "a priori estimate" before proving the exact form of the average case. However, I do believe that spiking PLRF examples have illustrated how "unrealistic" a "worst-case" bound needs to be, in order to cover quadratic models with arbitrary eigenvalues.

Empirically, I believe this paper provides a rigorous link between training spikes and expected model performance. So far, people have observed that training spikes are often correlated with sub-optimal quality of the final model, but couldn't find out exactly why -- as long as there are no other bugs. From our perspective, the training spikes are indicators of heavy-tailed data distribution, and since Paquette et al. (2024) has already established that different power-law exponents of the data can lead to different scaling parameters, it follows that training spikes are correlated with performance under scaling. So I believe the takeaway message for practitioners is: If you see spikes in training, the performance of your scaling ladder might indeed be affected; if you see spikes suddenly appear in one scale but not in smaller scales, the spiky training at that scale has something fundamentally different (i.e., power-law exponent), in terms of distribution of activation tensors.

These thoughts aside, I'm writing this paper because I want YOU to understand the mechanism -- as this is how I understand it. May this serve as a defense for my rather heuristic style in writing mathematical arguments.

The theory is heuristic. Thm 3 is a first-order Taylor expansion, and the sense in which the model is "approximated" is never quantified.

Well, it is linear approximation. Everyone knows how to "quantify" linear approximations (Lipschitz constants, etc.), don't they? I'm not writing it in that style because I don't think the theory or empirical experiments have developed to the point where any of those matters. Not yet.

Assumptions 4 and 6 (i.i.d. Gaussian latents, isotropic gradient covariance) carry the argument but are never checked on the Transformer.

Assumption 4 and Assumption 6 are tested in the paper in several ways. In Appendix D.2 we plot the evolution of eigenvalue distributions across layers and training steps, which is essentially testing Assumption 4, i.e., the "distribution of data" does not vary too much during training. As the results show, I would say that for Normal training this picture is reasonably acceptable; for Spiky training it is less accurate.

Not to mention that Prediction 5, an implication from Assumption 4, is directly verified in Appendix D.3.

Assumption 6 is partially tested in Appendix D.4, through our factorization of the "equivalent batch factor". The "third factor", measuring how far a DNN deviates from a Random Feature model, turns out to be relatively constant throughout training. This is essentially what Assumption 6 brings to us.

The language-model evidence is one model, six layers and two seeds, with spikes judged by eye. The simulator also produces bumps for a run that does not spike

The "judged by eye" issue should have been addressed by our top-level response, Quantitative spiky metric and DNN simulation demo, above.

AuthorsOpenReview ↗

Full Reply to Critical Weaknesses and Questions (cont.)

it overestimates spikes by up to 10x at B = 64.

Our theory suggests that B≪D\texttt{B}\ll\sqrt{\texttt{D}} is a condition for the simulation to be accurate. Specific to our example, D=1024\texttt{D}=1024, so the batch-size B≪32\texttt{B}\ll 32 is the condition. This kind of explains why at B=64\texttt{B}=64 the simulation shows different spiking statistics. The condition B≪D\texttt{B}\ll\sqrt{\texttt{D}} appears in several places in our theory (e.g., on Page 4, 20), and we will make it more explicit in our revision of the paper.

Does the beta2 result hold across seeds and model sizes?

We address multiple random seeds in our top-level response, "More numbers for the validation loss of different beta2 settings", above. I cannot tell you anything across model sizes. That topic is strictly beyond the scope of this paper.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Quantitative spiky metric and DNN simulation demo

Thanks for the comments. Since all reviewers are not satisfied with our arguments based on eyeballing the training trajectories, and want quantitative evaluation of "spikiness" instead, we would like to demonstrate such a quantitative evaluation, and share it early so that we can (hopefully) get some feedback. We will address each review in detail later.

We need a bit of extra setup, in order to have a metric for spikiness that does not depend on hand-picked thresholds, or the scale and slope of the loss. Our proposal, SpikyQ90, is demonstrated in this colab: https://colab.research.google.com/drive/1Sqme66YAZ_g6LcejknGiHtNAwQ3hHVp2?usp=sharing

In general, it is quite difficult to distinguish "normal fluctuations due to random data sampling" from "loss spikes". The introduction of our paper clearly points out one distinction, which is based on the settings of identical data shuffle and different model initialization seeds. However, this criterion does not directly apply to simulated loss trajectories, because it is unclear what "identical data shuffle" means in our simulation. Nevertheless, in the above colab we demonstrate that, purely based on visual statistics of loss trajectories, by running multiple experiments across model initialization, learning-rate or power-law exponent alpha, one can still clearly discover "phase transitions" between non-spiky and spiky runs. We believe this demo can establish that "spiky training" is indeed something different from "normal training", and it is quantitatively detectable.

Then, we present our DNN simulation demo in this colab: https://colab.research.google.com/drive/19qjgh8Je16FcZJrYnNREN3QRj4iFwUWe?usp=sharing . It demonstrates that the simulation of block_0_ffn_0 is more like "normal fluctuation" in the Normal setting, but closer to "spiky training" in both Spiky1 and Spiky2.

We hope these new results can resolve some of the reviewers' concerns.

AuthorsOpenReview ↗

More numbers for the validation loss of different beta2 settings

We would like to report more experiments to be added to Table 3, the validation loss of training a Transformer LM under different beta2 settings.

First, the "Normal (beta2=0.95)" run which is already in the paper, and the two Spiky runs since Reviewer hk89 is asking:

  • Normal (beta2=0.95): 2.4421
  • Spiky1: 2.5465
  • Spiky2: 2.5622

Then, we have re-run "beta2=0.982" and "beta2-schedule", with 3 model initialization seeds under the same data shuffle. The mean and standard deviation are:

  • beta2=0.982: 2.4405 +/- 0.0014
  • beta2-schedule: 2.4405 +/- 0.0011

"beta2-schedule" still appears to be slightly better than "beta2=0.982" in numbers, but this cannot be counted as statistically significant. We are running experiments with different data shuffle seeds as well, and will update the results once they are done.


UPDATE

We have re-run "beta2=0.982" and "beta2-schedule" with different data-shuffle seeds. Each data-shuffle uses 3 model initialization seeds, and we report the mean and standard deviation.

A different data-shuffle:

  • beta2=0.982: 2.4436 +/- 0.0017
  • beta2-schedule: 2.4430 +/- 0.0020

Another data-shuffle:

  • beta2=0.982: 2.4423 +/- 0.0007
  • beta2-schedule: 2.4398 +/- 0.0019

=======

Some more details: It seems there are two "slots" of validation losses, one is around 2.442 and one is around 2.438. Both "beta2=0.982" and "beta2-schedule" can randomly converge to a number near one of these two values.

=======

One note we want to make, is that the "beta2=0.982" setting is also obtained from observing the training metrics as we propose, rather than the baseline. "beta2=0.982" does significantly outperform the baseline "beta2=0.95".

We are not trying to establish that setting beta2 to a non-trivial schedule significantly outperforms any constant beta2 setting. Not at this scale. We believe our narration in the paper has carried this nuance rather clearly.

AuthorsOpenReview ↗

Results updated with different data-shuffle seeds

We have updated results with different data-shuffle seeds, as promised.

Best,

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.