acceptodds
Under review as a conference paper at ICLR 2027

Understanding and Mitigating AdamW Loss Spikes

Abstract

Loss spikes can disrupt AdamW training despite gradient clipping. Mismatched first- and second-moment time scales can let momentum outpace its normalization, amplifying the adaptive update (). We study how its magnitude and allocation affect immediate loss. Across three GPT-2 355M runs, at the unit threshol , the median fraction of coordinates newly crossing the threshold is 0.571% at 19 captured loss-spike onset states versus 0.0033% at nine ordinary states (over ). Globally shrinking the adaptive update lowers immediate loss at all 19 states, while selectively limiting new tail entrants improves further at equal norm. Tensor-relative history helps allocate attenuation, and replay favors sustained over onset-only control. These findings motivate , a lightweight yet effective combination of coordinate clipping and a tensor-relative growth constraint, with negligible optimizer-state overhead and no extra forward or backward passes. Across three GPT-2 774M runs per configuration, it reduces post-warmup spike episodes from 17 to zero and improves mean validation loss from 2.916 to 2.856. Half-rate and global norm-matched controls retain spikes despite similar validation quality. Dual control also achieves zero post-warmup spikes in the tested larger NanoGPT and Qwen3 models, supporting transfer across architectures.

Primary area:
interpretability and explainable AI
Keywords:
Loss Spikes, Adam, Moment Mismatch

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9qboOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies empirically the reasons behind loss spikes when training LLMs with AdamW. and propose a lightweight control mechanism to reduce the number of spikes by combining coordinate-wise clipping and rescaling tensors. The global coordinate-wise clipping threshold is motivated through estimating a stationary reference proxy for the coordinate tail.

Empirical results on GPT-2 77M and and transferred results on Qwen3-0.6B, Qwen3-1.7B and NanoGPT-1.5B shows that the proposed control mechanism also reduces the number of spikes there.

Critical strengths
  • The paper addresses an important issue of loss spikes in training LLMs using AdamW.
Critical weaknesses

This papers is overall poorly written and very hard to follow:

  • It uses terminology without clearly defining them, including those essential for this paper. While it is intuitive to say what constitutes of a spike, it is not clear how this works counts spikes and measures the area per spike. Other terminologies, which are key to understanding the paper, such as "continued control" (stated in the contributions) and the resulting "coordinate-and-tensor rule" are only mentioned in the contributions section and remain vague and unclear.
  • The results in the 774M controls and ablations are in my opinion inconclusive. While the table does show that combining clipping and EMA reduces the number of spikes and the area per spike (however this is being measured) and achieves a minimally lower validation loss, the row on halving the learning rate shows that the LR appears to be misspecified and chosen to large.
Questions for authors
  • Why do you measure the are under the loss spikes? I understand that this relates to how "persistent" the loss spike is, but it does not seem to be predictive of the final validation loss (at least based on Table 1 and Table 2.

  • Table 1 shows that B5 has no loss spikes, yet it achieves the worst validation loss. Can you comment on this?

  • In your scope you mention that you deliberately use a peak learning rate of 10^-3 and the first 16K steps of a 50K-step cosine schedule. Can you elaborate why you only consider 16K of the 50K-step schedule? What about even larger learning rates? Why do you consider 10^-3 to be the right LR to test?

Reviewer_RxY2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is unreadable. Besides the discrepancies between submitted title ("Understanding and Mitigating AdamW Loss Spikes") and the actual one appeared in the paper ("UNDERSTANDING AND CONTROLLING ADAMW LOSS SPIKES"), the submitted abstract (with a \method{} placeholder unfilled) and the actual one (the proposed method is called "Dual-Scale Update Control"), it feels like the main paper has been aggressively shortened at the last minute in order to satisfy the page limit, possibly by a generative AI without sufficient human supervision. As a result, most of the sentences are stripped of modality words, so it is difficult to tell if a sentence is a claim made by the authors, a finding derived from experiments, or a topic brought up for discussion. What's worse, the organization of the paper also does not explicitly separate background, the proposed method, derivation of the method, and findings of the experiments. What I can tell, from Algorithm 1 presented in Appendix G, is that the paper seems to propose a quite complicated clipping method for AdamW, that is supposed to mitigate loss spikes. The algorithm is also difficult to understand because there is only one line of a somewhat mysterious comment, and I cannot actually link the pseudo-code to the description of the algorithm (Section 5.2 on Page 7).

To be more specific, let me take the Conclusion section and try to decipher it with my best effort. It says:

"AdamW’s moment kernels bound update amplitude. Shared-state interventions link displacement and allocation to immediate loss; replay motivates sustained coordinate and tensor control. In full training, Dual suppresses strict episodes where half-rate and global norm-matched controls do not, with competitive validation. Ablations and transfer tests support its benefit under the tested contracts."

The first sentence, "AdamW’s moment kernels bound update amplitude.", I think means "the ratio between the 1st- and 2nd-moment estimates, used in Adam, is supposed to be bounded, in order to achieve stable training." I have no idea what "moment kernels" mean, but this is my best guess.

The second sentence, "Shared-state interventions link displacement and allocation to immediate loss; replay motivates sustained coordinate and tensor control." I think "displacement" means "optimizer update" in this paper. I still don't understand what "allocation", "replay", "sustained coordinate and tensor control" means in the paper.

The third sentence, "In full training, Dual suppresses strict episodes where half-rate and global norm-matched controls do not, with competitive validation." So "Dual" means the proposed method, Dual-Scale Update Control, while "half-rate" and "global norm-matched controls" should be some baseline methods. I have no idea what they are, or what "suppresses strict episodes" means.

The fourth sentence, "Ablations and transfer tests support its benefit under the tested contracts." What is "tested contracts"?

Critical strengths

N/A. I think it's a clear rejection.

Critical weaknesses

This paper is unreadable.

Questions for authors

N/A.

Reviewer_bxrCOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the mechanisms underlying loss spikes during AdamW training and proposes Dual-Scale Update Control, a lightweight method for improving training stability. The central motivation is that conventional gradient clipping does not necessarily prevent loss spikes because AdamW transforms gradients through first- and second-moment estimates with different temporal dynamics. When these moment estimates evolve at mismatched rates, the resulting adaptive updates can become unusually large and contribute to abrupt increases in training loss. The paper first distinguishes the theoretical update amplitude permitted by AdamW's moment coefficients from the actual parameter updates realized during training. Through experiments involving three GPT-2 355M training runs, the authors observe substantially more newly recruited large-update coordinates at captured loss-spike onset states than at ordinary states. Controlled interventions from identical optimizer states further demonstrate that reducing the adaptive update magnitude can lower immediate loss, while selectively attenuating newly recruited large-update coordinates can provide additional benefits at a matched adaptive-update norm. The authors also investigate tensor-level update allocation and show that an intervention applied only at spike onset is less effective over subsequent optimization steps than sustained control. Building on these observations, the authors introduce Dual-Scale Update Control, which combines coordinate-wise clipping with a tensor-relative update-growth constraint. The first mechanism limits unusually large coordinate-level adaptive updates, while the second constrains the mean absolute update magnitude of each tensor relative to a lagged exponential moving average of its historical update scale. The method retains AdamW's native moment recurrences and learning-rate schedule, adds only modest optimizer-state overhead, and does not require additional forward or backward passes. The proposed method is evaluated through controlled interventions, component ablations, alternative optimizer configurations, complete training runs, and transfer experiments across GPT-2, NanoGPT, and Qwen3 models. In the primary GPT-2 774M experiment, the default method reduces post-warmup loss-spike episodes from 17 to zero under the paper's strict detection protocol, while improving mean validation loss from 2.9160 ± 0.0305 to 2.8558 ± 0.0043. Half-learning-rate and global norm-matched controls retain more detected episodes despite achieving comparable validation performance. Additional experiments report no detected post-warmup episodes in the tested guarded NanoGPT and Qwen3 runs, providing preliminary evidence of transfer across architectures and numerical precisions. Overall, the paper presents a useful combination of optimizer analysis, controlled state-replay experiments, and practical training interventions. Its main contribution is the demonstration that loss-spike mitigation can benefit from controlling both individual large updates and tensor-level update growth, rather than relying solely on gradient clipping or global update scaling. However, the evaluation deliberately uses a relatively high peak learning rate to elicit loss spikes, and the primary comparisons involve only three independent training seeds per configuration. Furthermore, the reported absence of spikes depends on the chosen detection protocol, and some transfer experiments change multiple factors simultaneously. These limitations leave the generality of the proposed method across normally tuned training regimes and longer training schedules insufficiently established.

Critical strengths
  1. A well-motivated investigation combining mechanism analysis and controlled interventions. The authors distinguish the update amplitude permitted by AdamW's moment coefficients from the realized parameter displacement and its effect on loss. The use of shared optimizer states enables more controlled comparisons of update magnitude and allocation, reducing the confounding effects that arise when comparing independently evolving training trajectories. This strengthens the empirical motivation for the proposed intervention.
  2. A practical mitigation method supported by ablations and full-training comparisons. Dual-Scale Update Control combines coordinate-wise clipping with a tensor-relative growth constraint, addressing two complementary aspects of adaptive-update behavior. The authors compare the method with component ablations, half-learning-rate training, and global norm-matched controls. The reported reduction in detected spike episodes, together with competitive validation performance and transfer experiments, provides encouraging evidence that the method can improve stability under the tested training conditions without requiring additional forward or backward passes.
Critical weaknesses
  1. Limited evidence for generalization beyond the deliberately stressful training setting. The paper explicitly uses a peak learning rate of 10⁻³ and evaluates the first 16,000 steps of a 50,000-step cosine schedule to elicit more loss spikes. This is useful for studying failure mechanisms, but it does not establish that the proposed method offers comparable benefits under normally tuned learning rates, longer schedules, or training regimes in which loss spikes are less frequent. Since the practical significance of the method depends on its performance under realistic training conditions, broader evaluation would strengthen the paper's claims.
  2. Limited statistical evidence and sensitivity to evaluation protocols. The primary full-training comparisons use three independent seeds per configuration, while the numerous captured spike-onset states are nested observations rather than independent training replicates. In addition, the reported zero-spike result depends on the strict detection protocol; the paper notes that a broader detector still identifies two post-warmup episodes for the default configuration. Some transfer comparisons also involve changes in architecture, tokenizer, or numerical precision, and certain baseline runs do not reach the common endpoint. These factors make it difficult to determine the robustness and magnitude of the gains across a broader range of settings.
Questions for authors
  1. Clarification request — Robustness to the loss-spike detection protocol The default Dual-Scale Update Control configuration is reported to eliminate post-warmup episodes under the strict detector, while the broader detector retains two episodes. How sensitive are the main conclusions to the choice of relative-rise threshold, absolute-rise threshold, and robust-score threshold? Could the authors provide a systematic sensitivity analysis and clarify which conclusions remain consistent across detection protocols?
  2. Clarification request — Transfer evaluation and comparison fairness In the transfer experiments, some comparisons change multiple factors simultaneously, including architecture, tokenizer, or numerical precision. In addition, some conventional runs diverge before reaching the common endpoint, limiting paired validation comparisons. Could the authors clarify the extent to which the observed improvements can be attributed to the proposed update-control method itself and provide matched-endpoint or otherwise appropriately controlled comparisons where feasible?
  3. Suggestion for future work — Evaluation under normally tuned training regimes The current experiments deliberately use a relatively high learning rate to produce sufficient loss-spike events. Have the authors evaluated the method under more conventional, carefully tuned learning-rate schedules or longer training runs? If not, additional experiments in these settings would help establish whether the method is broadly useful beyond the tested stress conditions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.