acceptodds
Under review as a conference paper at ICLR 2027

From Learning Rates to Rotational Schedules: Finite-Horizon Theory on the Sphere

Abstract

How should step sizes be scheduled across a finite training horizon? Cosine and warmup-stable-decay schedules are widely adopted in large-scale pretraining, yet existing convergence theory for such schedules is largely Euclidean and does not capture the rotational dynamics of normalized neural networks. In this work, we formulate schedule design directly in the space of rotational motion. Working within a geodesically convex spherical setting, we derive a last-iterate convergence guarantee for arbitrary rotational schedules, i.e. sequences of angular step sizes, that separates the sum of the scheduled step sizes from their distribution over time. We recover the same finite-horizon schedule-dependency as in Euclidean theory, and we solve it globally: the bound-optimal schedule is unique, and approaches linear decay as the horizon grows. Finally, we connect this result to Hyperball, where fixed parameter norm and normalized updates make the learning rate act directly as the rotational schedule. We find that our theoretical rotational bound predicts schedule performance in practice, evaluated on language model pretraining runs optimized with Hyperball. Consistent with the bound prediction, we find that cosine and linear schedules achieve comparable performance, both ahead of warmup-stable-decay and constant schedules, thus providing practical insight into the schedule design of Hyperball-style optimizers.

Primary area:
optimization
Keywords:
optimization, deep learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_z3R7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes the last-iterate convergence properties of learning rate schedules in geodesically convex spherical settings. Building on top of a last-iterate bound for rotational coefficient schedules, it shows that the bound for linear decay approaches the optimum bound after amplitude tuning. The theoretical analysis also sheds light on understanding the HyperBall algorithm's behavior.

Empirical experiments on a 124M Llama on the FineWeb dataset for 20K updates show that: 1. with Hyperball, the validation loss trend matches the normalized bound; 2. without Hyperball, warmup-stable-decay and cosine decay are competitive with each other. Although the theory predicts cosine over WSD for the nominal on-the-sphere profiles, which contradicts the empirical evidence, in a different setting of measured unconstrained rotational trajectories the theory and practice align with each other.

Critical strengths

The paper is well-organized and the theoretical results are novel. The last-iterate convergence bound for learning rate schedules in Hyperball-style optimizers is an open question, and this paper provides a detailed analysis, along with an optimal schedule under the given bound and its connection to practical learning rate schedules. The analysis can be useful for guiding and developing new optimizers in different real-world applications.

Critical weaknesses

The major weakness of the paper lies in significance and quality.

  1. Significance: although the paper provides a theoretical analysis in the spherical setting and connects it to real-world practices, no new practically usable optimizers have been produced based on the theory. A truly valuable theory is expected to guide empirical practices, especially in developing better methodologies. To compensate for this weakness, it is recommended to provide more empirical comparisons between the baseline schedules and bound-optimal schedules.

  2. Significance: In lines 435-439, the bound favors cosine over WSD in nominal on-the-sphere profiles, which contradicts the empirical observation. This makes this theory setting-dependent and hurts its usefulness in real-world practice. How do engineers know which theoretical setting to apply when choosing the learning rate schedule for a coming large-scale pre-training run? To alleviate this issue, it is recommended to provide empirical evidence that helps distinguish settings based on empirical statistics, e.g., by examining the local structure of the loss [1] and determining which setting to apply.

  3. Significance: The empirical experiments are conducted on only Llama 124M, trained on ∼\sim 2.6B tokens (20,000 updates, sequence length 512, batch size 256). This is a very small-sized experiment. To improve the empirical significance, it is recommended to add more model families, with larger model sizes.

  4. Quality: It is highly recommended to verify the theoretical assumptions in the empirical settings to show that the adopted pre-training setting aligns with the theory.

  5. Quality: In Theorem 2.3, cap containment is assumed, not derived.

  6. Novelty: AI tools are used to help develop theoretical models or conceptual frameworks, formulate mathematical claims, and provide critical ingredients for proving mathematical claims. As AI is nowadays sufficiently strong to tackle even Millennium-Prize-level problems, the key novelty of theory papers now lies not in strong results, but in the core technical insights and intuition. It would be nice if the authors could explain what the key technical advancement of the theorem derivation is compared with past analysis in last-iterate learning rate schedules and optimization in spherical settings.

Reference

[1]: Yao, Zhewei, et al. "Pyhessian: Neural networks through the lens of the hessian." 2020 IEEE international conference on big data (Big data). IEEE, 2020.

Questions for authors

When one is designing proper learning-rate schedules when pre-training a new model, how can the engineer/researcher make the decision based on the provided theory?

5 comments
AuthorsOpenReview ↗

Authors Response

Thank you for your constructive review. We are glad to read that you found our paper well-organized and that you appreciated the novelty of our theoretical results, which seek to answer an open question in the literature.

We are particularly grateful for your remarks on the practical significance of our work: they were very helpful in guiding us to improve the manuscript, in which now -- we hope -- you will find clear evidence of the practical benefits of our theoretical analysis. In what follows, we summarize the changes we made to the manuscript in response to your comments.


Practical significance

No new practically usable optimizers have been produced based on the theory. A truly valuable theory is expected to guide empirical practices, especially in developing better methodologies. [...] How can the engineer/researcher make the decision based on the provided theory?

Following your suggestion, we now demonstrate two practical uses of the framework:

  • Constructing better schedules for existing optimizers. Using our theory, we derive new LR schedules for Adam and Muon that achieve lower final validation loss than independently tuned linear, cosine, and WSD in all three evaluation seeds and across both Adam and Muon (Section 4.3, Figure 4; Appendix E.8).

    The theory specifies the target rotational profile, and the norm model determines the learning rates needed to produce it as each matrix's norm evolves. All candidate schedules are computed before training from initialization, matrix dimensions, and optimizer settings. The temporal profiles and relative rates across matrices follow from this construction; only one overall peak is tuned per optimizer. Compensating for norm growth produces learning rates that keep rising while the target rotation decreases.

  • Choosing between established schedules before training. We predict the rotational motion induced by candidate learning-rate schedules and use the bound to compare them before launching a training run. For example, the framework correctly favors WSD over cosine for unconstrained Muon, where predictions based directly on learning rates (Schaipp et al., 2025) give the opposite ordering (Section 4.2, Figures 1 and 3).

    Agreement between predicted and measured norm growth and rotation supports the mechanism behind these comparisons (Figures 18 and 22; Appendices E.6–E.7). No training measurements enter the predictions.

The empirical experiments are conducted on only Llama 124M, trained on ∼ 2.6B tokens (20,000 updates, sequence length 512, batch size 256). This is a very small-sized experiment.

We now report three-seed results and add 720M Muon/MuonH experiments on 16.25B tokens. The larger runs confirm the smaller-scale schedule comparisons, and the predicted norm and rotational profiles consistently agree with measurements (Section 4.2, Appendix E.7).

The bound favors cosine over WSD in nominal on-the-sphere profiles, which contradicts the empirical observation.

We agree that the original wording needed clarification: the predicted WSD–cosine ordering actually agrees with the experiments both with Hyperball and without norm control.

In particular, under Hyperball, our experiments agree with the prediction: cosine and linear outperform WSD. Without norm control (Adam and Muon), norm growth cools rotation during WSD's constant-rate phase. Our model predicts this cooling, and evaluating our bound on the predicted rotational profiles recovers WSD's advantage over cosine that is observed in the experiments. Instead, predictions based directly on learning rates, as in Schaipp et al. (2025), miss this reversal.

Figures 1 and 3 now make the distinction explicit across optimizers and model sizes. The profiles are predicted from the training setup, without using measured rotations.


[1/2, continues below]

[...continues]

Theoretical assumptions and novelty

It is highly recommended to verify the theoretical assumptions in the empirical settings. [...] Cap containment is assumed, not derived.

We extend the theorem to hemispherical caps (Appendix B.2). In the audited trajectories, all 60 monitored matrices remain within their initialization-centered hemispheres at every update, including warmup; the largest observed angle is 88.9° (Figure 15, Appendix E.4).

We also investigate the scale invariance assumption in the empirical settings. Across all four optimizer configurations, typical matrices become less sensitive to local rescaling while their raw gradients become more nearly tangent (Figures 16–17, Appendix E.5). These measurements support the geometric picture motivating our analysis.

As clarified in Section 4, the pretraining experiments test the framework's predictive power beyond the idealized convexity and oracle assumptions.

It would be nice if the authors could explain what the key technical advancement of the theorem derivation is compared with past analysis in last-iterate learning rate schedules and optimization in spherical settings.

Our contribution connects learning-rate scheduling in LLMs with geodesically convex optimization, establishing a last-iterate guarantee for arbitrary rotational schedules in a setting compatible with nontrivial scale invariance. Normalization makes the effective step depend on the sampled gradient; controlling this effect gives the one-step inequality needed for the analysis (Appendix B.3). We then solve the scheduling problem exactly, proving a unique bound-optimal profile, its finite-horizon recursion, and its linear-decay limit (Appendix C). Combining this result with models of norm dynamics recovers schedule comparisons missed by predictions based directly on learning rates.


Conclusions

We hope that these clarifications, together with the additional experiments included in the manuscript have adequately addressed your concerns and reservations. In light of this new evidence, we would be grateful if you could reconsider your score. We remain at your disposal for any further questions or requests.


[2/2]

Dear Reviewer,

With the discussion period closing soon, we would appreciate a brief reply on whether our rebuttal addresses your main concerns. We’re happy to clarify any remaining points before the deadline.

Thank you again for your time.

Reviewer_zE66OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks how step sizes should be distributed over a finite horizon when the parameters of a scale-invariant network effectively move on a sphere. It replaces the learning rate by a rotational coefficient sts_t (for GD, st=ηt/∥xt∥2s_t = \eta_t/\lVert x_t\rVert^2, Prop. 2.2) and studies the normalized tangent update x^t+1=(x^t−stg~t)/∥x^t−stg~t∥\hat x_{t+1} = (\hat x_t - s_t \tilde g_t)/\lVert \hat x_t - s_t \tilde g_t\rVert on a geodesically convex spherical cap of radius θcap<π/4\theta_{\mathrm{cap}} < \pi/4. Theorem 2.3 gives a last-iterate bound D22M+(1+θcap)G2M2CT(ω)\frac{D^2}{2M} + \frac{(1+\theta_{\mathrm{cap}})G^2 M}{2} C_T(\omega), where M=∑tstM = \sum_t s_t and CT(ω)=ωT+∑k=1T−1ωk2/∑t=k+1TωtC_T(\omega) = \omega_T + \sum_{k=1}^{T-1} \omega_k^2 / \sum_{t=k+1}^{T} \omega_t is — as the paper states — the same allocation functional as in the Euclidean analysis of Defazio et al. (2023) and Schaipp et al. (2025). The proof combines a spherical one-step inequality (from Zhang & Sra's comparison, with the normalization absorbed into the factor 1+θcap1+\theta_{\mathrm{cap}}) with the Defazio/Schaipp tail-averaging argument. Section 3 minimises CTC_T exactly: the optimal cost obeys Cn⋆=2(1+Cn−1⋆−1)C_n^\star = 2(\sqrt{1+C_{n-1}^\star} - 1), the optimal allocation follows a recursion (Cor. 3.3), CT⋆∼4/TC_T^\star \sim 4/T, and the optimum converges pointwise to linear decay (Prop. 3.4), which itself attains 4/T+O(log⁡T/T2)4/T + O(\log T/T^2).

Empirically, the paper uses Hyperball, which fixes each matrix's Frobenius norm and normalizes the update, so that under tangency st=ηts_t = \eta_t and G=1G = 1. On 124M Llama / FineWeb (20 000 steps, single seed), with AdamH and MuonH, separately tuned linear and cosine schedules finish comparably and ahead of WSD and constant, matching the CTC_T ranking. Without Hyperball (Adam/Muon, weight decay 0), WSD becomes competitive; evaluating CTC_T on measured post-warmup turning-angle profiles reproduces the WSD-below-cosine ordering (Fig. 3, Table 2).

The exact solution of the allocation problem is clean, the recursion is a nice small result, and the paper is careful in its appendix about what it does not establish. My reservations are that the theoretical content is very close to the existing Euclidean analysis, that the hypotheses under which the bound holds demonstrably fail in the regime where it is tested, and that the empirical evidence (one seed, one scale, small gaps, a retrospective diagnostic) is thinner than the abstract's "predicts schedule performance in practice" suggests.

Critical strengths
  1. An exact, unique solution of the finite-horizon allocation problem. Theorem 3.2 and Corollary 3.3 reduce the minimisation of CTC_T over the simplex to a one-dimensional recursion, Cn⋆=2(1+Cn−1⋆−1)C_n^\star = 2(\sqrt{1+C_{n-1}^\star} - 1), with the optimal allocation built from the shorter tails. The paper notes that this "provides an analytic solution to the constant-envelope functional optimized numerically by Schaipp et al. (2025, App. C.3)" (p. 7).

  2. Under Hyperball the bound makes a sharp, testable prediction. Because Hyperball fixes G=1G = 1 and makes st=ηts_t = \eta_t exactly under tangency, the bound determines not only the ranking of schedules but the ratio of their optimal peak learning rates, independent of the unknown DD and θcap\theta_{\mathrm{cap}}.

Critical weaknesses
  1. The theorem's hypotheses fail in the experiments it is used to explain, by margins large enough that the agreement cannot be read as confirmation of the theory.

    (a) The cap hypothesis is contradicted by the paper's own tuned learning rates. Theorem 2.3 requires x^1∈C\hat x_1 \in C, i.e. D=dS(x^1,x^⋆)<θcap<π/4D = d_S(\hat x_1, \hat x^\star) < \theta_{\mathrm{cap}} < \pi/4 (eq. 2.5, p. 4). Under Hyperball G=1G = 1 exactly (p. 43), so the bound-optimal amplitude is M⋆=D/(1+θcap)CTM^\star = D/\sqrt{(1+\theta_{\mathrm{cap}})C_T} and the bound-optimal peak rate for the executed linear schedule is 0.00695⋅D/1+θcap0.00695 \cdot D/\sqrt{1+\theta_{\mathrm{cap}}}. Inverting at the best tested peak rates in Table 1 gives D/1+θcap≈2.04D/\sqrt{1+\theta_{\mathrm{cap}}} \approx 2.04 rad (AdamH) and 1.44 rad (MuonH); cosine and WSD give 1.35–2.09 rad. Since 1≤1+θcap<1.341 \le \sqrt{1+\theta_{\mathrm{cap}}} < 1.34, every one of these implies D>π/4≈0.785D > \pi/4 \approx 0.785 rad, typically by a factor of 2–3; even the smallest value (MuonH constant, 0.82 rad) exceeds the bound. That is unsurprising: a randomly initialised high-dimensional matrix is ≈π/2\approx \pi/2 from any fixed target direction. If the theory is taken at face value, either the tuned rates are far from bound-optimal (and then the "prediction" is not what selects them), or the iterates start well outside any cap on which the theorem holds. Fig. 11a avoids the issue by plotting against the dimensionless G1+θcaps1/DG\sqrt{1+\theta_{\mathrm{cap}}} s_1/D (p. 44), so the absolute scale is never confronted with data. (Caveats: an upper bound's minimiser need not coincide with the empirical optimum, and with ≈ 60 matrices on a product of spheres the relevant DD is a root-mean-square angle; neither changes the order of magnitude.) Relatedly, assumption (v) — that all iterates "remain in C almost surely" (p. 14) — is a trajectory assumption on the stochastic process. Because a single step turns by arctan⁡(st∥g~t∥)\arctan(s_t\lVert\tilde g_t\rVert), almost-sure containment implicitly requires an almost-sure bound on st∥g~t∥s_t\lVert\tilde g_t\rVert, which is stronger than the second-moment assumption (iv). The only instance where it is verified (Prop. D.1) uses "the exact-gradient tangent oracle" (p. 40), not a stochastic one.

    (b) The oracle hypothesis cannot hold for the directions Hyperball uses. The theorem needs a fresh oracle with E[g~t∣Ft]=∇SL(x^t)\mathbb{E}[\tilde g_t \mid \mathcal F_t] = \nabla_S L(\hat x_t) (2.8). Under Hyperball the oracle is the unit-norm normalised Adam or Muon direction vtv_t (p. 43). A unit-norm vector can be an unbiased estimate of ∇SL\nabla_S L only if the noise grows as the gradient shrinks, so that ∥Evt∥\lVert\mathbb{E} v_t\rVert tracks ∥∇SL∥\lVert\nabla_S L\rVert; momentum (β1=0.8\beta_1 = 0.8 for AdamH, 0.95 Nesterov for MuonH, p. 43) makes vtv_t depend on earlier draws, so its conditional mean is an average of past gradients at past points; and Newton–Schulz orthogonalisation is a nonlinear map that biases the direction. The main text addresses this only conditionally — "If the normalized direction is also conditionally unbiased for a geodesically convex loss and the trajectory stays in the cap, Theorem 2.3 applies with G = 1" (p. 8) — and does not say that the condition is false here.

    (c) The turning-angle agreement does not test either hypothesis. The measured angles follow arctan⁡stHB\arctan s_t^{\mathrm{HB}} (p. 8). But by (E.1), the turning angle depends only on ηt\eta_t and ct=⟨x^t,vt⟩c_t = \langle \hat x_t, v_t\rangle, and for matrices with hundreds of thousands of entries ctc_t is near zero for almost any update direction not aligned with the weights. The agreement confirms the geometric identity tan⁡Δϕt=ηt\tan\Delta\phi_t = \eta_t under tangency — which is a useful sanity check of the Hyperball implementation — but it carries no information about unbiasedness, convexity, or containment. The text's "The angle agreement establishes Hyperball as a setting in which the learning-rate schedule directly prescribes a rotational coefficient schedule" (p. 8) is accurate; the abstract's step from there to "our theoretical rotational bound predicts schedule performance in practice" (p. 1) is not supported by it.

    Taken together, the bound is applied in a regime where none of its non-trivial hypotheses (cap, unbiasedness, freshness, containment) holds. That is common practice in the Schaipp et al. line of work and can be acceptable — but then the paper should present the experiments as evidence that the functional CTC_T is a useful heuristic for Hyperball, not as a test of the spherical theory, and should explain why the spherical machinery adds anything over applying the Euclidean bound to ηt\eta_t.

  2. The theoretical novelty relative to Defazio et al. (2023) and Schaipp et al. (2025) is modest, and the empirical claims are stated more strongly than the evidence supports.

    (a) Theory. The schedule-dependent part of the bound is identical to the Euclidean one; the paper says so ("the resulting time-distribution cost CTC_T matches the Euclidean one found by Schaipp et al. (2025)", p. 2) and the tail argument is taken over from Defazio et al. "summing the one-step inequality over successive tails" (p. 5). Any one-step inequality of the form st(Lt−L⋆)≤12(dt2−dt+12)+cst2G2s_t(L_t - L^\star) \le \tfrac12(d_t^2 - d_{t+1}^2) + c s_t^2 G^2 yields this same functional, so the coincidence "despite using different geometry, assumptions, and update rules" (p. 5) is expected rather than surprising. The new spherical ingredient is the one-step inequality with the 1+θcap1+\theta_{\mathrm{cap}} constant. Under Hyperball, where st=ηts_t = \eta_t and G=1G = 1, the resulting prediction is exactly the Euclidean Defazio/Schaipp bound evaluated on ηt\eta_t, and for the unconstrained runs the diagnostic is that same functional evaluated on tan⁡Δϕt\tan\Delta\phi_t, i.e. on the effective learning rate times the gradient norm (p. 46) — so I do not see what the claim that the prediction works "thanks to the scale-invariant construction absent in previous Euclidean analyses of Schaipp et al. (2025)" (p. 2) rests on. The headline prescription, "decay rotational coefficients approximately linearly over the horizon" (p. 2), is the prescription of Defazio et al., whose abstract states that "When considering only worst-case analysis, our theory predicts that the optimal choice is the linear decay schedule". Missing related work on exact/optimal last-iterate step sizes — Zamani & Glineur (SIAM J. Optim. 2025; arXiv 2307.11134), who construct a horizon-dependent step-size sequence attaining the exact last-iterate lower bound, and Jain, Nagaraj & Netrapalli (2019) — should be discussed alongside the recursion.

    (b) Empirics. All runs are a single seed at one scale (124M, 20 000 steps): "The single-seed results support the observed ordering without estimating its variability across seeds" (p. 46). Read from the figures, the Hyperball endpoint gap between linear/cosine and WSD is ≈ 0.02–0.03 nats, which is plausibly real; but the unconstrained comparison on which the "inverted WSD–cosine ordering" rests (Fig. 3, final 10%) differs by only ≈ 0.005–0.01 nats — the range where seed-to-seed variation at this scale is typically comparable. The diagnostic is retrospective: it is computed from the measured angles of the already-selected runs and "does not identify the theorem's stochastic-gradient coefficients or establish its oracle assumptions" (p. 46). And it gets one of the three pairwise comparisons wrong: "For Adam, linear retains a lower diagnostic value than WSD, although the plotted validation loss favors WSD" (p. 46). Against this, the introduction says "Our bound is able to predict the observed loss ordering in both settings (Fig. 3)" (p. 2), and the abstract says the bound "predicts schedule performance in practice" (p. 1). The Hyperball ordering itself is confirmatory: the paper notes that Hägele et al. (2026, Sec. 4.1.5) "find linear decay better than the tested WSD variants under magnitude–direction decoupling" (p. 14).

Questions for authors
  1. What is the measured angular distance between each matrix's initial and final weights under AdamH/MuonH? My inversion of the bound-optimal amplitude at your tuned peak rates implies D≈1.4D \approx 1.4–2.12.1 rad, far above π/4\pi/4. If the measured distance is also ≳π/4\gtrsim \pi/4, in what sense does Theorem 2.3 apply to these runs?

  2. What is the seed-to-seed standard deviation of final validation loss for at least one configuration, relative to the ≈ 0.005–0.01 nat gaps in Fig. 3 and the ≈ 0.02–0.03 nat gaps under Hyperball? The unconstrained "inverted ordering" claim rests entirely on the former.

  3. What, concretely, does the spherical analysis predict that the Euclidean bound applied to ηt\eta_t (Hyperball) or to the effective learning rate (unconstrained) does not? If the answer is "nothing different, but now under valid hypotheses", please reconcile this with Weakness 1.

4 comments
AuthorsOpenReview ↗

Authors Response

We sincerely thank you for the time and care you put into your review. Your thoughtful comments were very helpful in clarifying our presentation and in strengthening the evidence for our claims. We are glad that you found the paper clear and appreciated the exact, unique solution of the scheduling problem.

We have revised the manuscript to address your concerns, and we summarize the changes below, together with our responses to your questions.


Assumptions and Positioning

The hypotheses under which the bound holds demonstrably fail in the regime where it is tested. [...] [T]he bound is applied in a regime where none of its non-trivial hypotheses (cap, unbiasedness, freshness, containment) holds. That is common practice in the Schaipp et al. line of work and can be acceptable — but then the paper should present the experiments as evidence that the functional CTC_T is a useful heuristic for Hyperball, not as a test of the spherical theory[.]

We recognize that the original wording did not sufficiently distinguish between the theoretical guarantees and the empirical evaluation. Thanks to your comments, we have now clarified our positioning and the scope of both the theoretical and empirical claims.

As you suggest, our study follows the line of work of Schaipp et al. in using an idealized convex analysis to motivate predictions for nonconvex pretraining. The theoretical setup should therefore be intended as a simplifying model of the actual optimization process. The revised abstract, introduction, and Section 4 make clear that the experiments evaluate the predictive value of the rotational interpretation beyond the idealized convexity and oracle assumptions.

Despite the discrepancy between the theorem's assumptions and the empirical setting, we believe that our results (even more so now with the additional evidence we have collected; see below) provide strong evidence that the rotational, geodesically convex theory remains a useful model for understanding and predicting scheduling behavior in practice. We discuss what this interpretation adds to the prior Euclidean functional below.

The cap hypothesis is contradicted by the paper's own tuned learning rates. [...] If the theory is taken at face value, either the tuned rates are far from bound-optimal (and then the "prediction" is not what selects them), or the iterates start well outside any cap on which the theorem holds. [...] What is the measured angular distance between each matrix's initial and final weights under AdamH/MuonH?

Your comment prompted us to revisit the cap restriction. The original θcap<π/4\theta_{\mathrm{cap}}<\pi/4 condition was unnecessarily strong: the revised Theorem 2.3 allows θcap≤π/2\theta_{\mathrm{cap}}\leq\pi/2, with a fixed cap center that need not coincide with the minimizer. Appendix B.2 gives the geometric derivation.

We also audited AdamH and MuonH with linear, cosine, and WSD schedules, monitoring all 60 matrices at initialization and after every update, including warmup. All observed trajectories remain inside their initialization-centered hemispheres. Across these six runs, the RMS initialization-to-final angle across matrices ranges from 80.6° to 87.7°, and the largest angle observed at any update is 88.9° (Figure 15, Appendix E.4).

Regarding the amplitude inversion, identifying DD from an empirically selected learning rate would require that rate to also minimize the theoretical bound. We do not assume this equality; peak rates are tuned by validation loss.


[1/3, continues below]

[...continues]


Theoretical Novelty

The theoretical novelty relative to Defazio et al. (2023) and Schaipp et al. (2025) is modest

We agree that the Defazio/Schaipp last-iterate argument is an important ingredient of our proof. However, we believe that the main contribution of our work lies in connecting learning-rate scheduling in LLMs with geodesically convex optimization and models of norm dynamics, and in the predictive power that this connection provides. For normalized tangent updates, the effective exponential-map step depends on the sampled gradient norm. Controlling this effect gives the spherical one-step progress inequality; we then apply their last-iterate argument to obtain a guarantee for arbitrary rotational schedules in a setting compatible with nontrivial scale invariance (Appendices B.3–B.4). Neither Defazio et al. nor Schaipp et al. consider this setting, nor prior work on Riemannian optimization merges the scale-invariant and geodesically convex perspectives to study learning-rate scheduling in LLMs. We have clarified this contribution in the introduction and Section 2.

Furthermore, we also solve the shared scheduling functional exactly, proving a unique finite-horizon optimizer and its linear-decay limit. Schaipp et al. optimize this functional only numerically and leave global optimality open.

Finally, thank you for suggesting Jain, Nagaraj, and Netrapalli (2019) and Zamani and Glineur (2025). We have incorporated both in Section 3 and Appendix A, situating our result relative to their Euclidean last-iterate guarantees.


[2/3, continues below]

[...continues]


Empirical Evidence

The empirical evidence (one seed, one scale, small gaps, a retrospective diagnostic) is thinner than the abstract's "predicts schedule performance in practice" suggests. The diagnostic is retrospective: it is computed from the measured angles of the already-selected runs. What, concretely, does the spherical analysis predict that the Euclidean bound applied to learning rates or to the effective learning rate does not?

Following your comments, we have substantially expanded the evaluation and clarified the scope of the empirical claims.

Predicting schedule performance

Regarding your question about the shared functional with Schaipp et al., our work identifies a limitation of their learning-rate interpretation and shows how to account for it. Norm growth changes the rotational motion induced by a learning-rate schedule, so comparisons based directly on nominal learning rates can miss the observed ordering. Our spherical analysis gives a formal basis for applying the scheduling functional to rotational coefficients.

The retrospective diagnostic has been replaced by rotational profiles fully predicted from schedule shapes, matrix dimensions, and fixed optimizer settings alone. Using a norm model we predict update magnitudes and norm growth, from which we calculate the normalized rotational profiles. No training measurements enter these predictions. Figures 18 and 22 compare the predicted and measured dynamics.

Without Hyperball, the bound on nominal learning rates favors cosine, whereas our predicted rotational profiles favor WSD, matching the validation loss (Figures 1 and 3). For Adam, the new prediction also places WSD ahead of linear, resolving the earlier diagnostic's mismatch.

Under Hyperball, the learning rate directly controls rotation, so the learning-rate and rotational evaluations coincide and agree with the observed WSD–cosine ordering, as you point out.

As you noted, under Hyperball the bound also predicts relative preferred peak rates independently of the unknown problem scale. The expanded learning-rate sweeps show the same pattern: linear and cosine prefer similar rates, while WSD and constant prefer smaller ones (Figure 13).

The Hyperball ordering itself is confirmatory.

We agree that earlier empirical work of Hägele et al. favors linear over tested WSD variants under norm control. Our analysis gives a finite-horizon account of this comparison, together with the exact bound-optimal schedule. The revised evaluation then tests the same rotational interpretation without norm control, predicts the reversed WSD–cosine ordering, and uses it to construct improved schedules.

Evidence across seeds and model sizes

For the primary schedule comparisons, we now report three evaluation seeds with tuned peak rates selected on a separate tuning seed. For Muon without Hyperball, WSD finishes about 0.0065 nats below cosine, with a paired standard deviation below 0.001 across the three seeds. Figure 3 shows the corresponding comparisons for the other settings.

We also add 720M Muon/MuonH experiments on approximately 16.25B tokens, using the same tuning and three-seed evaluation procedure. The larger runs reproduce the WSD–cosine reversal between training with and without norm control (Figure 1, Appendix E.7).

Constructing learning-rate schedules

Finally, we use the framework to construct learning-rate schedules targeting the bound-optimal rotational profile. All candidate schedules are computed before training from initialization, matrix dimensions, and optimizer settings. The target profile and relative rates across matrices follow from the construction; one overall peak is tuned per optimizer. For both Adam and Muon, the measured rotations follow the prescribed decay, and the selected schedules outperform independently tuned linear, cosine, and WSD in all three evaluation seeds (Section 4.3, Figure 4; Appendix E.8).


Conclusions

We hope that these clarifications and the additional experimental evidence have addressed your concerns. We are grateful for the opportunity to make the contribution and its relationship to practice clearer, and remain at your disposal for any further questions or suggestions.


[3/3]

Dear Reviewer,

With the discussion period closing soon, we would appreciate a brief reply on whether our rebuttal addresses your main concerns. We’re happy to clarify any remaining points before the deadline.

Thank you again for your time.

Reviewer_gjY4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors formulate the rotational scheduling problem on a geodesically convex spherical cap, motivated by recent training methods for normalized networks with scale-invariance. Building on previous results on convex last-iterate analysis, they extend the analysis to the geodesically convex case and propose an analytical solution to the optimal scheduling problem. They empirically verify that these theoretical results can be applied to language model training using the HyperBall optimizer [Wen et al., 2026].

Critical strengths
  1. The paper extends previous theoretical results to the geodesically convex case, covering non-trivial scale-invariant losses.
  2. They support their results with experiments using the recently proposed HyperBall optimizer.
Critical weaknesses
  1. The overall writing of this paper could be improved. For example, since the theoretical results in this paper are largely based on the previous work of Schaipp et al. (2025), the authors could introduce the previous results in more detail to improve readability. Also, some figures are repeatedly referenced in the main text, even though they are located in the appendices.
Questions for authors
  1. The proposed optimal schedule and the ranking among WSD, linear, and cosine schedules are based on a theorem establishing an upper bound on last-iterate convergence for a geodesically convex loss. It is quite surprising that these results transfer to language model pretraining, where the loss is nonconvex and the upper bound need not be tight. Do you have any intuition as to why this theory-to-practice transfer occurs?
3 comments
AuthorsOpenReview ↗

Authors Response

Thank you for your thoughtful review and helpful suggestions, which we have used to improve the clarity of the manuscript.

We have rewritten several paragraphs to make our contributions clearer and easier to follow. We have also improved the presentation of the derivations, to explain the steps more directly.


Clarity and prior work

We agree that the relationship to Schaipp et al. needed a clearer introduction. The introduction and Section 2.2 now explain their convex SGD result, its empirical scheduling predictions, and how our spherical analysis changes the interpretation of its coefficients. Appendix A expands the discussion of related work.


Figures and Appendices

The newest version of the manuscript is careful to keep the main results and figures in the main text. Figures 1 and 3 now compare predictions based on learning rates and rotations with pretraining results at 720M and 124M, respectively. Figure 4 shows the constructed learning-rate schedules, their measured rotational profiles, and their validation losses. Supporting details remain in the appendices.


Question 1: Theory-to-practice transfer

Do you have any intuition as to why this theory-to-practice transfer occurs?

Thank you for this interesting question! We share your surprise at the agreement, especially with the new experimental evidence which makes it even more compelling.

Convex analysis has already shown empirical value as a proxy for scheduling in nonconvex pretraining. In particular, Schaipp et al. demonstrate that a convex last-iterate bound reproduces scheduling behavior and guides learning-rate transfer.

One possible intuition is that these bounds capture a basic tradeoff between making progress and reducing fluctuations near the endpoint. Wen et al.'s river-valley analysis provides a complementary picture: training makes progress along the valley while oscillating across it, and cooldown suppresses the oscillations, revealing earlier progress.

We suspect that our improved predictive power comes from using a description closer to the geometry of normalized networks. Under scale invariance, parameter directions determine the loss, while norms affect how learning rates change those directions. Rotational steps account for this dependence, making the scheduling theory more representative of the effective optimization dynamics. Understanding the precise conditions for transfer beyond convexity remains an interesting direction for future work.


Conclusions

We hope that the changes we have made in response to your comments have improved the clarity of the work, and thus addressed your main concern. In case you have any further questions or suggestions, we remain at your disposal.

Thank you for your response. I encourage the authors to discuss related work on loss landscapes, as mentioned in the response to Question 1. Since my initial evaluation was positive, I will maintain my current score.

Dear Reviewer,

Thank you for following up and for supporting our paper’s acceptance.

We appreciate your suggestion and will definitely add a discussion in the final version, connecting our findings on rotational cooling to the river-valley interpretation of the pretraining loss landscape.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We sincerely thank all reviewers for their careful reading and constructive suggestions. We are glad that they recognized the value of the geodesically convex extension (gjY4), the clean exact solution of the scheduling problem (zE66), and its potential to guide optimization in practice (z3R7).

Your comments helped us clarify the contribution and strengthen its experimental support. The revised manuscript emphasizes the predictive and practical value of the rotational interpretation, while distinguishing the empirical predictions from the formal guarantee. The main updates are:

  • Predictions before training. We replace the retrospective diagnostic with rotational profiles predicted from schedule shapes, matrix dimensions, and optimizer settings. Accounting for norm growth allows our rotational bound to predict the WSD–cosine ordering observed experimentally, and missed by the learning-rate predictions of Schaipp et al. (Figures 1 and 3).

  • Stronger experimental evidence. We now report three independent evaluation seeds at rates selected on a separate tuning seed, and add 720M Muon/MuonH pretraining on 16B tokens. The larger runs consistently reproduce the schedule comparisons observed at 124M (Section 4.2; Appendix E.7).

  • Practical schedule construction. Guided by the theory, we construct learning-rate schedules targeting the bound-optimal rotational profile. For both Adam and Muon, they achieve lower final validation loss than independently tuned linear, cosine, and WSD schedules in all three evaluation seeds (Section 4.3, Figure 4).

  • Refined theory and clearer presentation. We extend the theorem to hemispherical caps and add containment and local scale-sensitivity measurements. We also clarify the relationship to prior work and the distinction between formal guarantees and empirical predictions, improve the derivations, and bring the central comparisons into the main text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.