- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies two types of loss spikes in the pre-training of LMs. These loss spikes are mainly caused by the divergence of logits output by the LM head. The work identifies that Adam's coordinate-wise preconditioning disrupts the zero-sum structure of cross-entropy gradients along the vocabulary dimension, leading to a continuous drift in the mean of output embeddings and the scale of logits. This drift is benign under exact computation. It amplifies quantization errors in BF16 precision, ultimately triggering training divergence. Based on this insight, the paper proposes using SGD with momentum exclusively for the LM head.
Critical strengths
-
This work gives a classification of phenomena and experimental analysis regarding fast spikes and time-bomb spikes.
-
The study draws connections between Adam preconditioning, LM-head mean shift, and finite-precision errors, offering an insightful explanation of the underlying mechanisms.
-
The proposed LM-head SGD-M modification is simple to implement and demonstrates robust stability across various model scales and learning rates.
Critical weaknesses
-
While the paper posits a unified mechanism for fast spikes and time-bomb spikes, the evidence regarding fast spikes relies primarily on correlational observations and mitigation experiments. It has not yet been directly proven that the mean shift and finite-precision errors induced by Adam are the root causes.
-
Although BF16 error accumulation is strongly correlated with loss divergence and FP32 training is indeed more stable, the study lacks direct causal validation (such as controlled numerical error injection or fine-grained precision comparisons), leaving the precise causal chain not fully established.
-
Many key experiments rely on settings involving exceptionally high learning rates, the absence of weight decay, and over-training (roughly 12 times the Chinchilla-optimal compute). Furthermore, the experiments concerning time-bomb spikes and precision were conducted primarily on models ranging from 28M to 404M parameters, so it remains unclear whether the conclusions generalize to standard large-scale pre-training.
-
The LM-head SGD-M approach is relatively simple but requires separate tuning of learning rate and momentum. The advantages of this method across a wider range of architectures, tied embedding configurations, and practical large-scale model training remain to be verified.
Questions for authors
Please refer to the Critical Weaknesses part.
Thank you for the response. The new centering and controlled rounding experiments address my main concern about the causal role of finite precision in time-bomb spikes, and the revised framing clarifies the scope of that claim. I have decided to raise my rating to 3.
My remaining concerns are whether the mechanism generalizes to standard large-scale pretraining and whether LM-head SGD-M remains effective across architectures, particularly with tied embeddings. I encourage the authors to investigate these questions in future versions.