- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper analyzes the last-iterate convergence properties of learning rate schedules in geodesically convex spherical settings. Building on top of a last-iterate bound for rotational coefficient schedules, it shows that the bound for linear decay approaches the optimum bound after amplitude tuning. The theoretical analysis also sheds light on understanding the HyperBall algorithm's behavior.
Empirical experiments on a 124M Llama on the FineWeb dataset for 20K updates show that: 1. with Hyperball, the validation loss trend matches the normalized bound; 2. without Hyperball, warmup-stable-decay and cosine decay are competitive with each other. Although the theory predicts cosine over WSD for the nominal on-the-sphere profiles, which contradicts the empirical evidence, in a different setting of measured unconstrained rotational trajectories the theory and practice align with each other.
Critical strengths
The paper is well-organized and the theoretical results are novel. The last-iterate convergence bound for learning rate schedules in Hyperball-style optimizers is an open question, and this paper provides a detailed analysis, along with an optimal schedule under the given bound and its connection to practical learning rate schedules. The analysis can be useful for guiding and developing new optimizers in different real-world applications.
Critical weaknesses
The major weakness of the paper lies in significance and quality.
-
Significance: although the paper provides a theoretical analysis in the spherical setting and connects it to real-world practices, no new practically usable optimizers have been produced based on the theory. A truly valuable theory is expected to guide empirical practices, especially in developing better methodologies. To compensate for this weakness, it is recommended to provide more empirical comparisons between the baseline schedules and bound-optimal schedules.
-
Significance: In lines 435-439, the bound favors cosine over WSD in nominal on-the-sphere profiles, which contradicts the empirical observation. This makes this theory setting-dependent and hurts its usefulness in real-world practice. How do engineers know which theoretical setting to apply when choosing the learning rate schedule for a coming large-scale pre-training run? To alleviate this issue, it is recommended to provide empirical evidence that helps distinguish settings based on empirical statistics, e.g., by examining the local structure of the loss [1] and determining which setting to apply.
-
Significance: The empirical experiments are conducted on only Llama 124M, trained on 2.6B tokens (20,000 updates, sequence length 512, batch size 256). This is a very small-sized experiment. To improve the empirical significance, it is recommended to add more model families, with larger model sizes.
-
Quality: It is highly recommended to verify the theoretical assumptions in the empirical settings to show that the adopted pre-training setting aligns with the theory.
-
Quality: In Theorem 2.3, cap containment is assumed, not derived.
-
Novelty: AI tools are used to help develop theoretical models or conceptual frameworks, formulate mathematical claims, and provide critical ingredients for proving mathematical claims. As AI is nowadays sufficiently strong to tackle even Millennium-Prize-level problems, the key novelty of theory papers now lies not in strong results, but in the core technical insights and intuition. It would be nice if the authors could explain what the key technical advancement of the theorem derivation is compared with past analysis in last-iterate learning rate schedules and optimization in spherical settings.
Reference
[1]: Yao, Zhewei, et al. "Pyhessian: Neural networks through the lens of the hessian." 2020 IEEE international conference on big data (Big data). IEEE, 2020.
Questions for authors
When one is designing proper learning-rate schedules when pre-training a new model, how can the engineer/researcher make the decision based on the provided theory?
5 comments
[...continues]
Theoretical assumptions and novelty
It is highly recommended to verify the theoretical assumptions in the empirical settings. [...] Cap containment is assumed, not derived.
We extend the theorem to hemispherical caps (Appendix B.2). In the audited trajectories, all 60 monitored matrices remain within their initialization-centered hemispheres at every update, including warmup; the largest observed angle is 88.9° (Figure 15, Appendix E.4).
We also investigate the scale invariance assumption in the empirical settings. Across all four optimizer configurations, typical matrices become less sensitive to local rescaling while their raw gradients become more nearly tangent (Figures 16–17, Appendix E.5). These measurements support the geometric picture motivating our analysis.
As clarified in Section 4, the pretraining experiments test the framework's predictive power beyond the idealized convexity and oracle assumptions.
It would be nice if the authors could explain what the key technical advancement of the theorem derivation is compared with past analysis in last-iterate learning rate schedules and optimization in spherical settings.
Our contribution connects learning-rate scheduling in LLMs with geodesically convex optimization, establishing a last-iterate guarantee for arbitrary rotational schedules in a setting compatible with nontrivial scale invariance. Normalization makes the effective step depend on the sampled gradient; controlling this effect gives the one-step inequality needed for the analysis (Appendix B.3). We then solve the scheduling problem exactly, proving a unique bound-optimal profile, its finite-horizon recursion, and its linear-decay limit (Appendix C). Combining this result with models of norm dynamics recovers schedule comparisons missed by predictions based directly on learning rates.
Conclusions
We hope that these clarifications, together with the additional experiments included in the manuscript have adequately addressed your concerns and reservations. In light of this new evidence, we would be grateful if you could reconsider your score. We remain at your disposal for any further questions or requests.
[2/2]
Dear Reviewer,
With the discussion period closing soon, we would appreciate a brief reply on whether our rebuttal addresses your main concerns. We’re happy to clarify any remaining points before the deadline.
Thank you again for your time.
Authors Response
Thank you for your constructive review. We are glad to read that you found our paper well-organized and that you appreciated the novelty of our theoretical results, which seek to answer an open question in the literature.
We are particularly grateful for your remarks on the practical significance of our work: they were very helpful in guiding us to improve the manuscript, in which now -- we hope -- you will find clear evidence of the practical benefits of our theoretical analysis. In what follows, we summarize the changes we made to the manuscript in response to your comments.
Practical significance
Following your suggestion, we now demonstrate two practical uses of the framework:
Constructing better schedules for existing optimizers. Using our theory, we derive new LR schedules for Adam and Muon that achieve lower final validation loss than independently tuned linear, cosine, and WSD in all three evaluation seeds and across both Adam and Muon (Section 4.3, Figure 4; Appendix E.8).
The theory specifies the target rotational profile, and the norm model determines the learning rates needed to produce it as each matrix's norm evolves. All candidate schedules are computed before training from initialization, matrix dimensions, and optimizer settings. The temporal profiles and relative rates across matrices follow from this construction; only one overall peak is tuned per optimizer. Compensating for norm growth produces learning rates that keep rising while the target rotation decreases.
Choosing between established schedules before training. We predict the rotational motion induced by candidate learning-rate schedules and use the bound to compare them before launching a training run. For example, the framework correctly favors WSD over cosine for unconstrained Muon, where predictions based directly on learning rates (Schaipp et al., 2025) give the opposite ordering (Section 4.2, Figures 1 and 3).
Agreement between predicted and measured norm growth and rotation supports the mechanism behind these comparisons (Figures 18 and 22; Appendices E.6–E.7). No training measurements enter the predictions.
We now report three-seed results and add 720M Muon/MuonH experiments on 16.25B tokens. The larger runs confirm the smaller-scale schedule comparisons, and the predicted norm and rotational profiles consistently agree with measurements (Section 4.2, Appendix E.7).
We agree that the original wording needed clarification: the predicted WSD–cosine ordering actually agrees with the experiments both with Hyperball and without norm control.
In particular, under Hyperball, our experiments agree with the prediction: cosine and linear outperform WSD. Without norm control (Adam and Muon), norm growth cools rotation during WSD's constant-rate phase. Our model predicts this cooling, and evaluating our bound on the predicted rotational profiles recovers WSD's advantage over cosine that is observed in the experiments. Instead, predictions based directly on learning rates, as in Schaipp et al. (2025), miss this reversal.
Figures 1 and 3 now make the distinction explicit across optimizers and model sizes. The profiles are predicted from the training setup, without using measured rotations.
[1/2, continues below]