acceptodds
Under review as a conference paper at ICLR 2027

Spiking, Fast and Slow: The Anatomy of Loss Spikes in Language Model Pretraining

Abstract

Loss spikes in language model pretraining can waste substantial compute and derail otherwise healthy runs. A primary driver of loss spikes is the growth and divergence of output logits in the final language modeling (LM) head. In this work, we divide these training instabilities into two types: fast spikes that abruptly increase loss before recovering, and slow terminal divergences that emerge only after prolonged training, which we call time-bomb spikes. We build on observations that these instabilities stem from fundamental properties of the softmax cross-entropy loss and its interaction with Adam’s preconditioner. We observe that this interaction leads to LM head weight growth that may be harmless under exact arithmetic but can be destabilizing under finite-precision training and propose a simple optimization adjustment that ameliorates the issue. Furthermore, we show that various mitigation techniques from the literature (including our own) largely act on the shared underlying problem of logit growth by reducing quantization error in the LM head. Our work connects fast and time-bomb spikes and an array of prior methodological proposals through a common lens of analysis: controlling last-layer logit growth reduces finite-precision rounding error which is essential for stabilizing long pretraining runs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
loss spikes, LLM pretraining, training instabilities, long horizon training

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_fT3COpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies two types of loss spikes in the pre-training of LMs. These loss spikes are mainly caused by the divergence of logits output by the LM head. The work identifies that Adam's coordinate-wise preconditioning disrupts the zero-sum structure of cross-entropy gradients along the vocabulary dimension, leading to a continuous drift in the mean of output embeddings and the scale of logits. This drift is benign under exact computation. It amplifies quantization errors in BF16 precision, ultimately triggering training divergence. Based on this insight, the paper proposes using SGD with momentum exclusively for the LM head.

Critical strengths
  • This work gives a classification of phenomena and experimental analysis regarding fast spikes and time-bomb spikes.

  • The study draws connections between Adam preconditioning, LM-head mean shift, and finite-precision errors, offering an insightful explanation of the underlying mechanisms.

  • The proposed LM-head SGD-M modification is simple to implement and demonstrates robust stability across various model scales and learning rates.

Critical weaknesses
  • While the paper posits a unified mechanism for fast spikes and time-bomb spikes, the evidence regarding fast spikes relies primarily on correlational observations and mitigation experiments. It has not yet been directly proven that the mean shift and finite-precision errors induced by Adam are the root causes.

  • Although BF16 error accumulation is strongly correlated with loss divergence and FP32 training is indeed more stable, the study lacks direct causal validation (such as controlled numerical error injection or fine-grained precision comparisons), leaving the precise causal chain not fully established.

  • Many key experiments rely on settings involving exceptionally high learning rates, the absence of weight decay, and over-training (roughly 12 times the Chinchilla-optimal compute). Furthermore, the experiments concerning time-bomb spikes and precision were conducted primarily on models ranging from 28M to 404M parameters, so it remains unclear whether the conclusions generalize to standard large-scale pre-training.

  • The LM-head SGD-M approach is relatively simple but requires separate tuning of learning rate and momentum. The advantages of this method across a wider range of architectures, tied embedding configurations, and practical large-scale model training remain to be verified.

Questions for authors

Please refer to the Critical Weaknesses part.

1 comment

Thank you for the response. The new centering and controlled rounding experiments address my main concern about the causal role of finite precision in time-bomb spikes, and the revised framing clarifies the scope of that claim. I have decided to raise my rating to 3.

My remaining concerns are whether the mechanism generalizes to standard large-scale pretraining and whether LM-head SGD-M remains effective across architectures, particularly with tied embeddings. I encourage the authors to investigate these questions in future versions.

Reviewer_49NfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The authors reproduce pretraining divergence at small scale and distinguish two types of instability: fast spikes and time-bomb spikes. Comparing against prior mitigations such as z-loss and mu-centering, they argue that the fundamental fix is to resolve the zero-sum violation in the LM head so that the logit norm does not diverge. They attribute this violation to Adam's preconditioner and show that it can be resolved by training only the LM head with momentum SGD. Overall, the paper claims to identify the root cause of loss spikes and to provide a stable remedy.

Critical strengths
  • The paper is systematic and well-structured. It categorizes the loss spikes commonly encountered in pretraining, analyzes each case, and ultimately proposes a remedy, which makes it practically useful. In particular, revisiting existing mitigations strengthens the authors' proposal.
  • Studying time-bomb spikes in the overtraining regime is quite important. Modern LLMs are trained far beyond the compute-optimal TPP, yet many pretraining studies are conducted only at the Chinchilla-optimal budget. Some divergences appear only when the training budget is extended in this way, and discovering such a failure after substantial compute has already been spent may, in the worst case, require retraining from scratch.
  • Although limited, the experiments extend up to the 1.4B scale, which gives some sense that the results have been scaled up.
Critical weaknesses
  • It is reasonable that, under low precision, and in particular in bf16, a growing logit norm blurs the relative differences between logits. However, why this leads to divergence remains observational. The fact that the instability disappears with fp32 only shows that bf16 is involved. It does not show that the pathway is the loss of resolution.
  • Introducing SGD-M adds another learning rate to tune, which may be a practical burden. The sensitivity of SGD-M to its learning rate is unknown. In LLM pretraining, learning rates are typically tuned through small-scale sweeps and then extrapolated, so having one more learning rate to determine is a practical concern. It would be helpful to provide at least a simple observation on the relationship between the SGD-M and AdamW learning rates.
  • For fast spikes, although the paper compares mitigations in the setting without learnable RMSNorm gains, that setting is not common, so there appears to be limited practical solution. Admittedly, fast spikes do not seriously harm convergence, but since learnable RMSNorm gains are effectively standard, it remains unclear how to handle the loss spikes that still occur in modern LLM pretraining. Moreover, as the authors themselves note, recent LLMs all use learnable gains, so the motivation for including this analysis in the paper is not clear.
Questions for authors
  • In Table 2, SGD-M with momentum 0.9 generally underperforms the other methods. Line 346 also states that using beta1=0.99 for the AdamW head narrows the gap to SGD-M. In Figure 7, SGD-M appears to achieve the lowest loss. Does SGD-M use momentum 0.99 while the other methods use beta1=0.9? If beta1 is matched, does this difference persist? Otherwise, the gain from beta1 may be confounded with the effect of the method.
  • In the Related Work, the authors cite the Coupled Adam of Stollenwerk & Stollenwerk (2025), which shares Adam's second moment across the vocabulary axis. Since this method also preserves the zero-sum structure, it seems a natural baseline. Did the authors try it?
Reviewer_Lvf9OpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper explores the source of loss spikes in LM pretraining, focusing on the impact of the LM head and logit distribution. It explores different axes of this question, including data-related causes, modeling choices, LM head optimization, and precision. Overall, the authors propose a detailed analysis of both fast spikes and "time-bomb" spikes, and present promising and simple solutions.

Critical strengths
  • Insightful analysis and methods: This paper grounds improvements in concrete observations and thorough analysis of the behavior of logits and the LM head throughout training. As a result, the proposed fixes are principled and appear to mitigate most of the spikes in early model training. Showing the connections between precision and what they refer to as "band-aid" mitigations is valuable. Some experiments offer promising conclusions for practical setups, such as the momentum SGD on the LM head, or exposing that batch-skipping and token-skipping are not sufficient to avoid spikes.
  • Convincing connections between logits growth and performance: The authors draw a strong and consistent connections between logit growth and spikes or performance. This paves the way for deeper (or more theoretical) analysis on this topic.
Critical weaknesses
  • Concerns about the generalization of results: The experiments are intentionally designed to trigger spikes (high learning rates, small batch size). Although this is necessary for the purpose of the paper, it raises concerns about the validity of observations in a broader setup, as the causes for spikes in this exaggerated setup might not be the same as for spikes in usual pretraining setups. [1] also raise the question of model size / hidden dimension for output anisotropy-based training divergences, which is not explored here.
  • Missing experiments / baselines: the experimental contributions are not extremely thorough. For instance, in the token-skipping and batch-skipping experiments, more fine-grained analysis could be performed, with more or less aggressive rules for skipping. A more detailed analysis of the specific tokens and documents that trigger such spikes could be interesting. Comparing SGD-M with Stollenwerk et al. might also be insightful. Finally, a clear set of guidelines for practitioner, as well as a side-by-side comparison of the combined fixes in a reasonable setup (i.e. not too extreme LR), could clarify the completeness of the study.
  • Writing & Coherence: This is not fundamentally critical, but the paper is a bit chaotic with its writing; it reads like an enumeration of results at time. It would benefit for a more structured writing, and a consistent narrative throughout the paper.

[1]: Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck, Godey et al., COLM 2024

Questions for authors
  • Why did you use 1e-4 as a weight decay? Usual pretraining setups use either 0 or values within 1e-2 to 1e-1. It is also common to exclude the LM head from weight decay, which would be insightful to comment on in the paper.
  • Do you have an intuition whether embedding weight tying would have any impact on your observations?
  • Do you have results for LM head SGD-M in more usual setups (usual LR and batch size)?

Typos

L475: Instabiliites

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.