- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates how brief early-training interventions affect long-term outcomes in unstable Post-LN Transformers. The authors show that modifying only 25 updates can prevent functional collapse after 5,000 steps and further examine learning-rate schedules, transfer across model configurations, and optimizer-state effects.
Critical strengths
- The authors explore several useful controls and report negative results and limitations quite transparently.
- I appreciate the controlled experimental design, particularly the paired runs that share the same initial state and differ only during a short intervention window.
Critical weaknesses
-
I found the paper difficult to follow despite careful reading. The many internal experiment identifiers (R5, R6, R8, R9, etc.) make it hard to track the results. More importantly, I struggled to see how the three contributions connect. For example Sec. 6 reports very different effective ranks for healthy RAMP and NEUTRAL runs, but it remains unclear what these diagnostics tell us about why either intervention works.
-
The novelty and broader significance are not entirely clear to me. [1] already studied how large early updates and parameter rotations affect warmup in GPT training, while [2] investigated how warmup leads to more stable optimization. The successful ramps here seem consistent with these findings. Given that the short interventions fail in several larger configurations while Warmup500 remains more reliable, I am not sure what additional insight this work establishes.
-
Some key mechanisms remain unresolved. In Sec. 4, the equal-dose UP, CONSTANT, and REVERSE schedules have very different initial learning rates, so their outcomes may simply reflect the importance of suppressing the earliest updates. In Sec. 5.2, the failed m-only transplant has a much larger update proposal norm than the healthy full-state transplant (14.20 vs. 0.59). As the authors acknowledged that this makes it difficult to distinguish insufficient moment information from instability caused by mismatched optimizer states.
[1] Kosson et al. "Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training" NeurIPS 2024
[2] Kalra & Barkeshli. "Why Warmup the Learning Rate? Underlying Mechanisms and Improvements" NeurIPS 2024
Questions for authors
Q1. Could the authors compare schedules with similar initial update magnitudes but different later ordering, to clarify whether temporal organization matters beyond early-step suppression?
Q2. Would the m-only transplant still fail if its initial continuation update were norm-matched or clipped?
Q3. What do the authors see as the main new insight beyond existing warmup research (e.g. two papers indicated in weaknesses), particularly given the limited transfer and largely descriptive release-state analysis?