- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Using 175 training runs, this paper compares normalization placement, LayerNorm versus RMSNorm, and QK normalization in decoder-only transformers. Results show that performance rankings and stability depend on learning rate, depth, and training regime.
Critical strengths
- Single-axis comparisons help distinguish the contributions of RMSNorm and QK normalization within each residual layout.
- Seed pairing, closure calibration, and detailed run records improve confidence in the reported comparisons.
Critical weaknesses
- Novelty is modest. Most findings refine established observations about normalization and learning-rate sensitivity. Evidence from one width and one corpus leaves the broader significance of these refinements unclear.
- Learning rates are selected using one seed at 5k steps, then used to compare architectures at 20k steps. Some selections lie at the grid boundary or are barely distinguishable from neighboring rates. These experiments establish rankings under the chosen rates, but not rankings after adequate tuning.
- Loss close to unigram entropy does not demonstrate context-independent predictions. Reduced gradient norms also do not establish that a model cannot escape the plateau. Calling this a warm-up trap requires direct measurements of prediction distributions and interventions on warm-up or recovery.
- Cross-block comparisons remain difficult to interpret. Identical configurations differ by up to 0.018 nats for reasons that cannot be reconstructed, and depth-48 parity has not been checked under the canonical final-norm convention.
- Readability suffers from frequent block identifiers, implementation-history details, and qualifications. Readers must repeatedly consult the appendices to determine which comparisons are controlled and which conclusions remain provisional.
Questions for authors
Typos
Appendix G states that every collapse is complete by step 250, despite the unsettled run documented in Appendix B. Add the exception.