- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies effective learning-rate (ELR) scheduling in weight-normalized (WN) training. The authors first demonstrate that differences in parameter norms lead to different ELR trajectories under WN and weight decay (WD), and that matching ELR largely aligns their training loss dynamics. Motivated by these observations, they derive an inverse-square-root LR schedule from an idealized directional-alignment model under WN and propose its practical affine adaptation, AIR. Experiments on MoE pretraining show that AIR improves training and evaluation losses over cosine scheduling by approximately 0.01 at two model scales. The authors further investigate the importance of late-stage LR scheduling.
Critical strengths
1. Useful empirical insights.
The ELR-matching experiments and stage-wise schedule interventions provide insights into WN/WD training dynamics and the role of late-stage LR scheduling.
2. Practical improvements.
AIR achieves lower training and evaluation losses than cosine scheduling at two model scales, along with improvements in downstream performance.
Critical weaknesses
1. Unclear connection between ELR matching and AIR design.
Although ELR matching motivates the paper, AIR is derived independently by optimizing directional alignment under WN, where ELR is proportional to the nominal LR. The paper does not establish how AIR under WN relates to the naturally occurring ELR trajectories of standard WD training under cosine or WSD schedules. Moreover, there is no direct comparison of WN + AIR against these standard WD configurations, leaving the practical implications of the ELR-based motivation unclear.
2. Limited novelty of the ELR-matching observation.
D’Angelo et al. (NeurIPS 2024, Figure 7) already demonstrate that matching ELR trajectories can reproduce training loss dynamics in language model pretraining. Although the present paper studies a different setting (WN vs. WD), the authors should clarify what new insights their ELR-matching experiments provide beyond these existing results.
Questions for authors
- How do the actual ELR trajectories and final training/evaluation losses of WN + AIR compare with WD + Cosine and WD + WSD under comparable training budgets and hyperparameter tuning? Do these comparisons support the ELR-based motivation for AIR?
- Can the authors theoretically compare the ELR trajectories induced by WD + WSD/Cosine and WN+inverse-square-root, at least under simplified toy models like linear regression. In particular, under what conditions would these ELR trajectories be expected to be similar or different?
3 comments
Kindly Following Up
Thank you again for your time and thoughtful review. As the discussion deadline approaches, we are writing to kindly follow up on our response and revised manuscript. We have carefully addressed all of your questions and concerns, including the direct comparisons with standard WD recipes (WD + Cosine and WD + WSD) and their actual ELR trajectories, the distinction from D'Angelo et al. (2024), and the theoretical toy-model and linear-regression comparison of induced ELR trajectories.
We hope our response and revisions have resolved your concerns, and we would greatly appreciate your feedback and re-evaluation. Please feel free to let us know if you have any further questions.
Thanks for your response. My concerns are addressed, but the mathematical details are not checked carefully. I raise my score.
Thank you for your thoughtful review and constructive questions. We have added direct comparisons with standard WD recipes, actual ELR trajectories, and theoretical clarifications, with pointers below.
1. Connection Between ELR Matching and AIR, and Comparison with WD Recipes (W1 & Q1)
Because changing ηk under WD alters subsequent norms r>k, fixing rk≡R under WN decouples ELR (ηk/rk) from norm drift so designing the nominal LR schedule under WN directly designs the ELR schedule (Sec. 2.3). To address W1 and Q1 directly, we added comparisons against standard WD recipes and their actual ELR trajectories in Table 1 (P7) and Appendix C.5 (Table 6 and Fig. 9, P24–25). At 128B tokens (Table 6, P24), WN + AIR achieves substantially lower training and evaluation losses (1.730 / 2.174) than both WD + Cosine (1.753 / 2.205) and WD + WSD (1.763 / 2.203), gaining +1.94 downstream points over WD + Cosine (Table 1). At 500B tokens (Table 1), WN + AIR lowers evaluation loss from 1.857 to 1.832 and gains +4.05 points over WD + Cosine (57.78 vs. 53.73). Their actual ELR curves (App. C.5, Fig. 9, P25) directly support our motivation: WD + Cosine's growing norm drives ELR down too fast early on and slows late progress, and WD + WSD plateaus before a late drop, whereas WN + AIR sustains smooth convex 1/t ELR annealing to reach the lowest loss.
2. New Insights Beyond D'Angelo et al. (NeurIPS 2024, Fig. 7) (W2)
We have refined Related Work and added an explicit discussion of this distinction (Sec. 6, L511–513, P10). First, the setting and mechanism differ: D'Angelo et al. (2024) study unregularized vs. WD training around low-precision (bfloat16 vs. float32) stability via one-way global ELR matching (which diverges in bfloat16 without WD), whereas we perform bidirectional, blockwise ELR matching between WN and WD. Second, our new insight goes beyond post-hoc explanation to establish ELR scheduling: we first show that matching ELR collapses WN and WD training dynamics (Fig. 2a), then leverage WN where LR is proportional to ELR to derive the optimal schedule (Sec. 3), and finally verify in reverse that the derived schedule still works under WD when matched in ELR (Fig. 2b), establishing that schedule design should directly target ELR rather than nominal LR.
3. Theoretical Comparison of WD + WSD/Cosine vs. WN + 1/t in Toy Models (Q2)
We have added a theoretical comparison of the ELR trajectories induced by WD + WSD/Cosine and WN + 1/t under simplified toy models (including linear regression) in Appendix C.5 (Eqs. 38–44, P25–26). In short, WD + WSD/Cosine and WN + 1/t induce a similar 1/t ELR decay early on before parameter norms equilibrate, but diverge once WD norms approach their equilibrium r⋆(t)∝η(t)/λ (where equilibrium ELR scales as η⋆eff(t)∝λη(t), Eq. 39):
These exact regimes hold in noisy linear regression (Eqs. 43–44) and explain Figures 1(b,c) and 9(b,c): WD mimics 1/t ELR decay only transiently during early norm growth before plateauing or turning concave, whereas fixing rk≡R under WN removes norm lag so AIR directly sustains calibrated convex 1/t ELR annealing across the full horizon.
All of the comparisons, clarifications, and theoretical analyses above have been updated in the revised version. We sincerely appreciate your insightful feedback, hope our responses and revisions have addressed your concerns, and please feel free to let us know if you have any further questions.