- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies empirically the reasons behind loss spikes when training LLMs with AdamW. and propose a lightweight control mechanism to reduce the number of spikes by combining coordinate-wise clipping and rescaling tensors. The global coordinate-wise clipping threshold is motivated through estimating a stationary reference proxy for the coordinate tail.
Empirical results on GPT-2 77M and and transferred results on Qwen3-0.6B, Qwen3-1.7B and NanoGPT-1.5B shows that the proposed control mechanism also reduces the number of spikes there.
Critical strengths
- The paper addresses an important issue of loss spikes in training LLMs using AdamW.
Critical weaknesses
This papers is overall poorly written and very hard to follow:
- It uses terminology without clearly defining them, including those essential for this paper. While it is intuitive to say what constitutes of a spike, it is not clear how this works counts spikes and measures the area per spike. Other terminologies, which are key to understanding the paper, such as "continued control" (stated in the contributions) and the resulting "coordinate-and-tensor rule" are only mentioned in the contributions section and remain vague and unclear.
- The results in the 774M controls and ablations are in my opinion inconclusive. While the table does show that combining clipping and EMA reduces the number of spikes and the area per spike (however this is being measured) and achieves a minimally lower validation loss, the row on halving the learning rate shows that the LR appears to be misspecified and chosen to large.
Questions for authors
-
Why do you measure the are under the loss spikes? I understand that this relates to how "persistent" the loss spike is, but it does not seem to be predictive of the final validation loss (at least based on Table 1 and Table 2.
-
Table 1 shows that B5 has no loss spikes, yet it achieves the worst validation loss. Can you comment on this?
-
In your scope you mention that you deliberately use a peak learning rate of 10^-3 and the first 16K steps of a 50K-step cosine schedule. Can you elaborate why you only consider 16K of the 50K-step schedule? What about even larger learning rates? Why do you consider 10^-3 to be the right LR to test?