acceptodds
Under review as a conference paper at ICLR 2027

Your LR Schedule May Not Be Principled: Effective Learning Rate Scheduling in Weight-Normalized Training

Abstract

Weight-normalized (WN) training is emerging as a compelling alternative to conventional weight decay (WD), with recent studies reporting substantial pretraining loss gains. We show that these gains are driven primarily by the induced effective learning rate (ELR) trajectory rather than by normalization alone. Matching ELR trajectories largely collapses the training loss dynamics of WN and WD. *Schedule design should therefore target ELR rather than the nominal learning rate (LR) alone.* Building on this principle, we propose **AIR** (**A**ffine **I**nverse-square-**R**oot decay), which combines a theoretically derived decay with an affine adjustment to satisfy prescribed peak and terminal LRs over a finite training horizon. We further identify late-stage ELR scheduling as a key source of its gains, because a shared late schedule attenuates the effect of earlier LR choices. Experiments on 128 B-token Mixture-of-Experts pretraining demonstrate that AIR outperforms cosine and other candidate schedules in training loss, downstream performance, and hyperparameter robustness, reducing terminal training loss by approximately 0.01 relative to cosine at matched cumulative LR. Applying the same ELR trajectory to WD reproduces the corresponding training dynamics, showing that the benefits extend beyond fixed-norm training and establishing ELR as the principled scheduling object.

Primary area:
optimization
Keywords:
Weight Normalization, Effective Learning Rate Scheduler, Hyperball Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zQtzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies effective learning-rate (ELR) scheduling in weight-normalized (WN) training. The authors first demonstrate that differences in parameter norms lead to different ELR trajectories under WN and weight decay (WD), and that matching ELR largely aligns their training loss dynamics. Motivated by these observations, they derive an inverse-square-root LR schedule from an idealized directional-alignment model under WN and propose its practical affine adaptation, AIR. Experiments on MoE pretraining show that AIR improves training and evaluation losses over cosine scheduling by approximately 0.01 at two model scales. The authors further investigate the importance of late-stage LR scheduling.

Critical strengths

1. Useful empirical insights.

The ELR-matching experiments and stage-wise schedule interventions provide insights into WN/WD training dynamics and the role of late-stage LR scheduling.

2. Practical improvements.

AIR achieves lower training and evaluation losses than cosine scheduling at two model scales, along with improvements in downstream performance.

Critical weaknesses

1. Unclear connection between ELR matching and AIR design.

Although ELR matching motivates the paper, AIR is derived independently by optimizing directional alignment under WN, where ELR is proportional to the nominal LR. The paper does not establish how AIR under WN relates to the naturally occurring ELR trajectories of standard WD training under cosine or WSD schedules. Moreover, there is no direct comparison of WN + AIR against these standard WD configurations, leaving the practical implications of the ELR-based motivation unclear.

2. Limited novelty of the ELR-matching observation.

D’Angelo et al. (NeurIPS 2024, Figure 7) already demonstrate that matching ELR trajectories can reproduce training loss dynamics in language model pretraining. Although the present paper studies a different setting (WN vs. WD), the authors should clarify what new insights their ELR-matching experiments provide beyond these existing results.

Questions for authors
  1. How do the actual ELR trajectories and final training/evaluation losses of WN + AIR compare with WD + Cosine and WD + WSD under comparable training budgets and hyperparameter tuning? Do these comparisons support the ELR-based motivation for AIR?
  2. Can the authors theoretically compare the ELR trajectories induced by WD + WSD/Cosine and WN+inverse-square-root, at least under simplified toy models like linear regression. In particular, under what conditions would these ELR trajectories be expected to be similar or different?
3 comments

Thank you for your thoughtful review and constructive questions. We have added direct comparisons with standard WD recipes, actual ELR trajectories, and theoretical clarifications, with pointers below.

1. Connection Between ELR Matching and AIR, and Comparison with WD Recipes (W1 & Q1)

Because changing ηk\eta_k under WD alters subsequent norms r>kr_{>k}, fixing rk≡Rr_k \equiv R under WN decouples ELR (ηk/rk\eta_k / r_k) from norm drift so designing the nominal LR schedule under WN directly designs the ELR schedule (Sec. 2.3). To address W1 and Q1 directly, we added comparisons against standard WD recipes and their actual ELR trajectories in Table 1 (P7) and Appendix C.5 (Table 6 and Fig. 9, P24–25). At 128B tokens (Table 6, P24), WN + AIR achieves substantially lower training and evaluation losses (1.730 / 2.174) than both WD + Cosine (1.753 / 2.205) and WD + WSD (1.763 / 2.203), gaining +1.94 downstream points over WD + Cosine (Table 1). At 500B tokens (Table 1), WN + AIR lowers evaluation loss from 1.857 to 1.832 and gains +4.05 points over WD + Cosine (57.78 vs. 53.73). Their actual ELR curves (App. C.5, Fig. 9, P25) directly support our motivation: WD + Cosine's growing norm drives ELR down too fast early on and slows late progress, and WD + WSD plateaus before a late drop, whereas WN + AIR sustains smooth convex 1/t1/\sqrt{t} ELR annealing to reach the lowest loss.

2. New Insights Beyond D'Angelo et al. (NeurIPS 2024, Fig. 7) (W2)

We have refined Related Work and added an explicit discussion of this distinction (Sec. 6, L511–513, P10). First, the setting and mechanism differ: D'Angelo et al. (2024) study unregularized vs. WD training around low-precision (bfloat16 vs. float32) stability via one-way global ELR matching (which diverges in bfloat16 without WD), whereas we perform bidirectional, blockwise ELR matching between WN and WD. Second, our new insight goes beyond post-hoc explanation to establish ELR scheduling: we first show that matching ELR collapses WN and WD training dynamics (Fig. 2a), then leverage WN where LR is proportional to ELR to derive the optimal schedule (Sec. 3), and finally verify in reverse that the derived schedule still works under WD when matched in ELR (Fig. 2b), establishing that schedule design should directly target ELR rather than nominal LR.

3. Theoretical Comparison of WD + WSD/Cosine vs. WN + 1/t1/\sqrt{t} in Toy Models (Q2)

We have added a theoretical comparison of the ELR trajectories induced by WD + WSD/Cosine and WN + 1/t1/\sqrt{t} under simplified toy models (including linear regression) in Appendix C.5 (Eqs. 38–44, P25–26). In short, WD + WSD/Cosine and WN + 1/t1/\sqrt{t} induce a similar 1/t1/\sqrt{t} ELR decay early on before parameter norms equilibrate, but diverge once WD norms approach their equilibrium r⋆(t)∝η(t)/λr_\star(t) \propto \sqrt{\eta(t)/\lambda} (where equilibrium ELR scales as η⋆eff(t)∝λη(t)\eta_\star^{\mathrm{eff}}(t) \propto \sqrt{\lambda\eta(t)}, Eq. 39):

  • When they are similar (2λ∫0tη(s) ds≪12\lambda\int_0^t \eta(s)\,\mathrm{d}s \ll 1 and r0≪r⋆r_0 \ll r_\star): Because r(t)2r(t)^2 relaxes toward r⋆(t)2r_\star(t)^2 with a lag (Eq. 40), early in the WSD stable phase or near the Cosine peak, r(t)2≈r02+U2η02tr(t)^2 \approx r_0^2 + U^2\eta_0^2 t grows linearly with step tt, so delayed norm growth under WD passively induces shifted 1/t1/\sqrt{t} ELR decay, matching WN + 1/t1/\sqrt{t} (Eq. 41).
  • When they differ (2λ∫0tη(s) ds≳12\lambda\int_0^t \eta(s)\,\mathrm{d}s \gtrsim 1): Once norms equilibrate, WD + WSD flattens onto a constant equilibrium plateau ∝λη0\propto \sqrt{\lambda\eta_0} during its stable phase before its late drop, whereas WD + Cosine turns into a concave square-root-cosine profile ηWD+Coseff(t)∝ηcos(t)∝cos⁡(πt2K)\eta_{\mathrm{WD+Cos}}^{\mathrm{eff}}(t) \propto \sqrt{\eta_{\mathrm{cos}}(t)} \propto \cos(\frac{\pi t}{2K}) rather than strictly convex 1/t1/\sqrt{t} decay (Eq. 42).

These exact regimes hold in noisy linear regression (Eqs. 43–44) and explain Figures 1(b,c) and 9(b,c): WD mimics 1/t1/\sqrt{t} ELR decay only transiently during early norm growth before plateauing or turning concave, whereas fixing rk≡Rr_k \equiv R under WN removes norm lag so AIR directly sustains calibrated convex 1/t1/\sqrt{t} ELR annealing across the full horizon.

All of the comparisons, clarifications, and theoretical analyses above have been updated in the revised version. We sincerely appreciate your insightful feedback, hope our responses and revisions have addressed your concerns, and please feel free to let us know if you have any further questions.

AuthorsOpenReview ↗

Kindly Following Up

Thank you again for your time and thoughtful review. As the discussion deadline approaches, we are writing to kindly follow up on our response and revised manuscript. We have carefully addressed all of your questions and concerns, including the direct comparisons with standard WD recipes (WD + Cosine and WD + WSD) and their actual ELR trajectories, the distinction from D'Angelo et al. (2024), and the theoretical toy-model and linear-regression comparison of induced ELR trajectories.

We hope our response and revisions have resolved your concerns, and we would greatly appreciate your feedback and re-evaluation. Please feel free to let us know if you have any further questions.

Thanks for your response. My concerns are addressed, but the mathematical details are not checked carefully. I raise my score.

Reviewer_jRrKOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper investigates ELR as an explanation for the performance difference between WN and WD. It proposes AIR, a shifted 1/(t)1/\sqrt(t) lr schedule motivated by analyzing the alignment with a fixed target direction. Empirical evaluations show the benefits of AIR over several existing decay methods and shed light on late stage scheduling.

Critical strengths
  • The paper through 1) rescaling WN blocks with WD norm and unchanged lrs, and 2) constant norm WN blocks with ELR adjusted lrs close the WN-WD gaps.
  • AIR outperforms other methods on large scale MoE experiments demonstrating its effectiveness.
  • The authors shed light on late-stage scheduling and highlight its importance in closing the gap between the baselines and AIR.
Critical weaknesses
  • Inverse-square-root decay has been widely studied in convex optimization,particularly in the context of non smooth SGD [1] and weight averaging [2]. In fact, [2] already evaluate ηαα+t\eta\sqrt{\frac{\alpha}{\alpha + t}} with weight averaging for anytime LLM pretraining. AIR adds an affine endpoint adjustment, however, it is not clear if this adjustment results in the gains.
  • Matching cumulative lrs is not a strong basis for comparison, ideally, all methods should have their peak learning rates tuned.
  • Experiments with AdamW and dense models would help strengthen the paper.
  • For peak lr sensitivity, ideally the peak lr should be changed keeping the schedule hyperparameters constant.
  • The 1(t)\frac{1}{\sqrt(t)} schedule derivation does not require any ELR assumptions. The connection between the two is not very clear. The two contributions should be listed separately.
  • The theory analysis assumes aa and bb as which seems to be a very unrealistic assumption. Also, qkTukq_k^Tu_k need not be zero for large correlated updates, and the necessary condition mk2+a22b≤1m_k^2 + \frac{a^2}{2b} \le 1 also need not hold in that case.

[1] Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes. Ohad Shamir and Tong Zhang. ICML, 2013. [2] Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging. Alexandru Meterez, Pranav Ajit Nair, Depen Morwani, Cengiz Pehlevan, and Sham Kakade. 2026.

Questions for authors
  • Can the authors independently sweep peak LR for cosine and AIR, allowing cumulative LR to differ? Does AIR retain its advantage?
  • Can the authors run a version of the lr sensitivity experiments where the peak is the only quantity that is changed, the shapes remain fixed.
  • Can the authors empirically validate the assumption on aa, bb, ν\nu and uku_k.
  • Does the affine endpoint transformation retain the theoretical optimality derived on the unconstrained version.
3 comments

Thank you for your thorough review and insightful questions. We have revised the manuscript with specific pointers below.

1. Connection Between ELR and the 1/t1/\sqrt{t} Derivation (W5)

Following your suggestion, we listed ELR alignment and the AIR derivation as two separate contributions (Sec. 1, L98–104, P2) and clarified their connection in Section 2.3. Since ELR (ηkeff=ηk/rk\eta_k^{\mathrm{eff}} = \eta_k / r_k) governs training dynamics (Sec. 2.2), deriving the nominal LR schedule under WN (rk≡Rr_k \equiv R) is mathematically equivalent to deriving the ELR schedule (ηkeff=ηk/R\eta_k^{\mathrm{eff}} = \eta_k / R). Carrying ηk/rk\eta_k / r_k directly under WD (App. A.1, Eq. 23) is far more cumbersome due to evolving rkr_k, whereas multiplying rkWDr_k^{\mathrm{WD}} back to control WD at AIR's ELR reproduces WN's loss gains (Fig. 2b), confirming that ELR indeed dominates training dynamics.

2. Relaxing Theoretical Conditions and Empirical Validation (W6 & Q3)

We generalized Section 3.1–3.2 so the derivation no longer requires θk⊤uk=0\mathbf{\theta}_k^\top\mathbf{u}_k = 0 or constant a,ba, b. Spherical normalization cancels radial updates automatically (Eqs. 5–7, App. B.1), completing the square gives the general optimum for time-varying (a(t),b(t))(a(t), b(t)) without constant-drift saturation (Eqs. 12–13, App. B.4), and constant b(t)b(t) with slowly varying a(t)a(t) recovers shifted 1/t1/\sqrt{t} decay (Eqs. 14–15, App. B.5). For Q3, MoE measurements (Fig. 3, P5 and App. B.6) empirically confirm these conditions, showing nearly orthogonal updates, constant b(t)b(t) (<1%<1\% variation), and a 1.0%–1.9% median match to (1+t/α)−1/2(1 + t/\alpha)^{-1/2}.

3. Horizon-Free Derivation, Finite-Horizon Annealing, and Affine Optimality (W1 & Q4)

We have added citations and discussions of [1, 2] in Section 3.3 (L268–275, P5–6) and Related Work (Sec. 6, L508–510, P10). Because stepwise dynamics do not depend on a cutoff KK, deriving a decay law in a horizon-free setting and adapting it to bounded data is standard practice: [1, 2] and Su (2025) derive 1/t1/\sqrt{t} under horizon-free analyses, while bounded training adapts non-zeroing shapes via weight averaging [1, 2], cooldown (Ma & Chen, 2026), or the same affine map (Loshchilov & Ginsburg, 2026, Xiao et al., 2026). At step KK, raw 1/t1/\sqrt{t} stays near p/3≫ηmin⁡p/3 \gg \eta_{\min} with large noise, so weight averaging in [1, 2] (like LAWA) and our affine adjustment to ηmin⁡\eta_{\min} both serve as finite-horizon annealing for fair comparison.

For Q4, enforcing η(K)=ηmin⁡\eta(K) = \eta_{\min} departs from the unconstrained optimum, yet the affine map uniquely preserves relative decay-rate ratios across steps (Sec. 3.3, Eq. 18, App. B.7) and matches the finite-horizon tail-sum schedule (Defazio et al., 2024) uniquely for 1/t1/\sqrt{t} (App. B.9). Moreover, sharing the exact same affine map across all 12 shapes in Table 1 isolates the gain of 1/t1/\sqrt{t} itself (App. C.3, Table 5).

4. Peak-LR Tuning, Fixed-Shape Sensitivity, and Additional Settings (W2-W4, Q1, Q2)

In schedule-aware scaling laws (Tissue et al., 2025), the area under the LR curve (∑kηk\sum_k \eta_k) captures the total cumulative parameter-update amount, which stays closely aligned at the optimum across schedules for a given model and token budget. Matching cumulative LR is thus a principled setting that keeps candidate optima comparable while greatly reducing search cost. Furthermore, our newly added WN cosine baseline peak-LR search (App. C.1, Fig. 8, P23) ensures an optimal cosine baseline, which already suffices to prove AIR's gain over the baseline (and Cycloid at θ1=π\theta_1 = \pi shares AIR's exact peak, terminal, and cumulative LRs yet trails by about 0.0050.005 loss in Sec. 5.1, Fig. 6). Yet we also agree with (i) independently sweeping peak LR for Cosine and AIR (Q1), (ii) fixed-shape peak-LR sensitivity sweeps (W4, Q2), and (iii) evaluating Adam and dense models (W3). While all of these settings require dedicated hyperparameter sweeps across costly full pretraining runs for valid comparison, they are underway and we will update our response as soon as the results are available.

All of the refinements, verifications, and discussions above have been updated in the revised version. We sincerely appreciate your constructive comments, hope our responses and revisions have addressed your concerns, and please feel free to let us know if you have any further questions.

I thank the authors for their detailed response, however, several signifcant changes have been made to the paper requiring a reevaluation. I have increased my score to 2.

Thank you very much for taking the time to read our response and for your constructive feedback.

As noted in our global response, all revisions were made directly in response to the reviewers' suggestions while keeping our core contributions, proposed AIR schedule, and main conclusions unchanged. The updates mainly focus on polishing the writing, refining the logical flow and derivations, and adding available supporting experimental results.

We hope these clarifications and improvements help further address your concerns, and we kindly look forward to a more positive evaluation. Thank you again for your time and thoughtful review.

Reviewer_5yQWOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper claims to have the following contributions:

  • The authors identify the induced ELR trajectory as the key driver of WN's loss advantage over WD.
  • They further proposed AIR, an affine 1/sqrt(t) schedule and demonstrate that AIR can reduce the final loss by approximately 0.01 relative to cosine at matched cumulative LR.
Critical strengths
  • This paper successfully demonstrates that the proposed schedule outperforms a large family of baselines in large-scale training.
Critical weaknesses
  • Many parts of the paper are poorly written. For example, I cannot really understand Section 2. The caption of Figure 2 is barely readable and I don't know what information it really wants to convey. What does "arrows indicate the ELR target" mean? Where are the arrows? Which one is the ELR target? Is ELR target a scalar or a curve? Section 2.2 is also very dense. The second paragraph starts with "for the reverse matching", is this mentioned before? Why should people suddenly care about "reverse matching" which does not have a clear meaning? Section 3 is even worse. Where does Equation 6 come from? All subsequent math derivations jump so quickly that I can only understand them after chatting with an AI. Overall, I believe ICLR should enforce strict standards for writing quality. Papers are meant to be read by humans. I will insist on rejecting this paper if the writing quality is not improved in the rebuttal phase.
  • Why is "a" (Equation 7) constant in the theory part? This is an important question because if it is not constant, then AIR is not optimal. If this AIR schedule is not theoretically justified, then we should put a higher standard on the experiments. For example, the authors should validate AIR on more architectures and datasets.
Questions for authors

Due to the poor writing quality, I cannot really understand Section 2, but Wen et al. (2026) already emphasize that it is the ELR viewpoint that motivates Hyperball. There are also many recent studies such as https://arxiv.org/abs/2607.22444 (although this paper is also barely readable), https://hy.tencent.ai/research/elr and https://arxiv.org/abs/2608.27370 that propose similar viewpoints. In particular, the first two studies already showed that replaying the ELR trajectory makes muonH's and muon's performances the same. I understand that these can be regarded as concurrent works according to ICLR's rules, but it would be good if the authors could discuss how their works relate to these studies.

4 comments

Thank you for your careful review and constructive feedback. We have thoroughly revised the manuscript to improve both its clarity and theoretical generality, with specific pointers to the revised PDF below.

1. Clarifying Section 2, Figure 2, and Section 3 Derivations (W1)

We have rewritten Section 2 and updated Figure 2 to make these concepts explicit. Specifically, each parameter block has a scalar ELR ηkeff:=ηk/rk\eta_k^{\mathrm{eff}} := \eta_k / r_k at step kk (Sec. 2.1, Eq. 3), while an ELR target is the complete per-step curve {ηkeff}k≥0\{\eta_k^{\mathrm{eff}}\}_{k \ge 0} recorded in a baseline run (Sec. 2.2, L135–136, P3). Revised Section 2.2 now introduces both matching directions at the beginning (L133–135, P3): matching WN to WD's ELR validates that WN reproduces WD's loss dynamics (Eq. 4), while reverse matching (WD matching WN ELR) validates that an ELR trajectory derived under WN (such as AIR) transfers back to WD to reproduce its improved loss dynamics (L145–149, P3, Sec. 2.3, and App. A.3). In Figure 2, we removed the unclear "arrows" wording, explicitly labeled every curve in the legend, and rewrote the caption to clarify that each curve plots training loss minus the WD baseline (Fig. 2).

For Section 3, we have refined and expanded the derivations by adding clear prerequisite explanations and intermediate steps so they are much easier to follow (Sec. 3.1–3.2, Eqs. 5–15, paired step by step with App. B.1–B.5).

2. Relaxing the Constant-aa Condition and Validating AIR (W2)

We have refined and generalized our derivation in Section 3.2 so that it no longer requires a(t)a(t) to be constant. After deriving the general optimum for time-varying (a(t),b(t))(a(t), b(t)) (Sec. 3.2, Eqs. 12–13, App. B.4), we now only require a(t)a(t) to be non-vanishing and slowly varying to obtain the same shifted 1/t1/\sqrt{t} decay as before (with constant curvature b(t)b(t), Eqs. 14–15, App. B.5). We have also added empirical verification in MoE pretraining (Fig. 3, P5 and App. B.6) confirming that these relaxed conditions hold in practice, collapsing the optimal schedules onto (1+t/α)−1/2(1 + t/\alpha)^{-1/2} within 1.0%–1.9% median error across Cosine, Linear, and AIR. Beyond our two-scale MoE evaluation (128B and 500B tokens across 11 benchmarks, Sec. 4, Table 1), we agree that validating AIR across more architectures and optimizers (such as dense models and Adam) will further strengthen the paper, and we will update our response as soon as those results are available.

3. Relation to Wen et al. (2026) and Concurrent ELR Studies (Q1)

We have added this discussion to Related Work (Sec. 6, L503–505 and L514–520, P10). Specifically, Wen et al. (2026) use the ELR viewpoint to explain why weight decay works and to motivate Hyperball, but do not show that Hyperball's own gains over WD still come from its induced ELR trajectory, which we directly establish (Sec. 2.2). Concurrent studies likewise observe that matching ELR aligns WN and WD loss dynamics, while focusing on different aspects: Xiao et al. (2026) study angular step-size attribution, Liu et al. (2026) examine the scope of ELR collapse and ELR-based functional scaling laws, Zhou et al. (2026) explore angular-update heuristics and empirical schedules, and Luo et al. (2026) build low-cost pretraining recipes with heuristic shifted 1/t1/\sqrt{t} plus linear cooldown. Yet none of these works derives an optimal ELR schedule analytically. We instead derive 1/t1/\sqrt{t} ELR decay directly from directional-alignment dynamics under WN (Sec. 3.1–3.2), adapt it via an affine transformation into AIR for finite-horizon pretraining (Sec. 3.3, outperforming e−te^{-t} and 1−t1 - \sqrt{t} in Table 1), and attribute most of AIR's gain to late-stage scheduling (Sec. 5).

All of the detailed clarifications and discussions above have been updated in the revised version. We sincerely appreciate your time and thoughtful feedback, which have helped us greatly improve the paper. We hope our responses and the revised version have addressed your concerns, and please feel free to let us know if you have any further questions.

  1. Thanks for updating the paper. I can feel that the authors spent effort to improve the writing quality, which is good. Still, I don't think the math part is smooth enough for humans to read, especially the part deferred to appendix B.2.

    If the authors want to know why it is still not good enough, I would suggest the authors invite a friend to sit down and read your math part word by word. It is a trick that works for me.

    A simple example: "To simplify Eq. (13) without requiring constant a(t)" In the current version, there is no other place mentioning that a(t) may be required to be a constant. This is a condition that only appears in the last version. This kind of wording can distract the readers' attention and feed the readers' "cache" in the human brain with useless information. When really useful information comes, the readers may no longer have the attention or the cache to read it.

    I'm 100% sure that if the authors have read what they wrote, they will remove this "without requiring constant a(t)". Again, papers are meant to be read by humans. Maybe some other people are more open with reading papers with AI, but I will insist on rejecting this paper because it is just so hard to understand the math part without chatting with AI.

  2. Good. But still I do not fully understand your derivation of AIR.

  3. Several minor issues:

    • Wen et al. (2026)'s view is that Hyperball directly controls the ELR. So of course when you match the ELR of Muon with MuonH, you may see a similar training curve. I don't think it is fair to say "do not show that Hyperball's own gains over WD still come from its induced ELR trajectory".
    • "Xiao et al. (2026) study angular step-size attribution". I don't think "angular step-size attribution" is an existing terminology.
    • "Zhou et al. (2026) explore angular-update heuristics and empirical schedules." Very vague. What does this summary even mean?

    But I don't view these minor issues as a weakness of this paper.

Overall, I do like the idea and experiments in this paper. If the writing quality is ok and I could fully understand the derivation, then I will surely vote for acceptance. But the current writing is still not good, and I believe ICLR should enforce strict standards for writing quality.

Thank you so much for taking the time to carefully read our revised paper and for sharing such thoughtful, constructive feedback. We sincerely appreciate the effort you put into reviewing our paper in such detail, as well as your kind and supportive words about our core idea and empirical results.

We will continue to carefully revise the paper during this final stage of the rebuttal and afterward. In particular, we will work on:

  • Section 3 and the math derivations (including Appendices): Making the exposition smoother and more readable, adding intermediate steps so a human reader can follow easily.
  • Derivation and motivation of AIR: Explaining more clearly how the directional-alignment analysis leads to the 1/t1/\sqrt{t} schedule and why we use the affine calibration for finite-horizon training.
  • Related Work: Making our summaries of related work more accurate and fair.

We also appreciate and will follow your suggestion to invite a friend to read through the math section with us from a reader's perspective. Since a thorough pass like this takes real effort and time, we will upload an updated revision as soon as available.

Thank you again for your detailed review, support, and constructive suggestions.

Thank you again for your patience, detailed review, and helpful suggestions. Following your feedback, we have carefully revised the paper and uploaded an updated PDF. Here are the main changes:

  1. Clearer structure and smoother math flow (Section 3, Appendix B):

    • Added short overview sentences at the start of Sections 3.1–3.3 to lay out the full logic upfront:
      • Section 3.1: Starts from WN directional increments, defines the alignment mkm_k between θ^k\hat{\mathbf{\theta}}_k and the optimal direction v\mathbf{v}, and derives its dynamics with tangential drift a(t)a(t) and curvature b(t)b(t): m˙(t)=a(t)η(t)−b(t)η(t)2m(t).\dot m(t) = a(t)\eta(t) - b(t)\eta(t)^2 m(t).
      • Section 3.2: Maximizes m(t)m(t) by completing the square to derive the optimal ELR schedule η⋆(t)\eta^\star(t): η⋆(t)=a(t)2b(t)m02+τ(t)≈η01+t/α,\eta^\star(t) = \frac{a(t)}{2b(t)\sqrt{m_0^2 + \tau(t)}} \approx \frac{\eta_0}{\sqrt{1 + t/\alpha}}, which reduces to 1/t1/\sqrt{t} decay under two empirically verified conditions: (i) nearly constant b(t)b(t), and (ii) non-vanishing, slowly varying a(t)a(t).
      • Section 3.3: Adapts 1/t1/\sqrt{t} decay to prescribed peak and terminal LRs via an affine transformation to obtain AIR.
    • Reordered some equations and added brief explanations between them so the notation and logical flow are much easier to follow.
    • Split Appendix B into short, step-by-step subsections matching the exact order of Section 3 (Appendices B.1–B.11).
    • Also cleaned up leftover wording from earlier drafts (e.g., "without requiring constant a(t)a(t)").
  2. Clearer derivation and motivation of AIR (Sections 3.2–3.3, Appendices B.4–B.11): We start from the unconstrained optimal 1/t1/\sqrt{t} decay as in Eq. 15: q(k)=(1+kα)−1/2.q(k) = \left(1 + \frac{k}{\alpha}\right)^{-1/2}. For finite-horizon pretraining over KK steps, the terminal LR q(K)q(K) stays significantly above ηmin⁡\eta_{\min}, so we apply an affine transformation to obtain AIR (Section 3.3, Appendix B.8):

    \eta_{\mathrm{AIR}}(k) &= sq(k) + d \\ &= \eta_{\min} + (p - \eta_{\min})\frac{q(k) - q(K)}{1 - q(K)}, \end{aligned}$$ satisfying the **peak and terminal LR constraints**: $$\eta_{\mathrm{AIR}}(0) = p, \qquad \eta_{\mathrm{AIR}}(K) = \eta_{\min}.$$ Then, using the **Cauchy–Schwarz inequality**, we prove that among all smooth schedules satisfying these constraints, **this affine map uniquely minimizes relative decay-rate distortion from raw $1/\sqrt{t}$** and **preserves both decay-rate ratios and interval reduction fractions** across steps (Appendix B.10): $$\frac{\dot\eta_{\mathrm{AIR}}(t_1)}{\dot\eta_{\mathrm{AIR}}(t_2)} = \frac{\dot q(t_1)}{\dot q(t_2)}, \qquad \frac{\eta_{\mathrm{AIR}}(i) - \eta_{\mathrm{AIR}}(j)}{p - \eta_{\min}} = \frac{q(i) - q(j)}{1 - q(K)}.$$ Moreover, under convex last-iterate bounds (Defazio et al., 2024), $1/\sqrt{t}$ is the unique decay whose finite-horizon schedule $\phi(t)\int_t^K \phi(u)\,\mathrm{d}u$ reduces exactly to this affine form (Appendix B.11).
  3. More accurate summaries of Related Work (Section 6):

    • Wen et al. (2026): motivate Hyperball directly through explicit effective step-size control and demonstrate improved scale transfer, while focusing on norm constraints and adopting standard schedules rather than designing new ELR schedules.
    • Xiao et al. (2026): analyze an angular ELR and radial-update effects, and test empirical exponential and polynomial decays on dense models.
    • Liu et al. (2026): systematically establish the architectural and timescale conditions for ELR collapse and show that ELR enables functional scaling laws to transfer.
    • Zhou et al. (2026): show that ELR improves loss prediction and cross-scale transfer, and design empirical 1−t1-\sqrt{t} and piecewise-linear schedules inspired by observed angular-update patterns.
    • Luo et al. (2026): incorporate MuonH and ELR-aligned diagnostics into a low-cost continual-pretraining recipe.

Following your suggestion, we indeed invited a friend to read through the math section with us from a reader's perspective. They pointed out many spots in our wording and transitions that we had not considered before, which really helped a lot.

Thank you again for helping us improve the paper. We hope it now reads much more smoothly and that the math derivation is clear, and we look forward to a more positive re-evaluation of our work.

Comments on the paper

3 comments
Area_Chair_24DWOpenReview ↗

Discussion deadline approaching

Dear Reviewers and Authors,

Thank you for providing detailed reviews and the rebuttal.

To Reviewers 5yQW and zQtz: As the author–reviewer discussion deadline approaches, could you please review the authors’ responses and indicate whether your concerns are addressed?

Kind regards,

AC

AuthorsOpenReview ↗

Global Response

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all reviewers for their time and constructive feedback, and have carefully revised our paper accordingly. Importantly, our core principle, proposed schedule (AIR), theoretical result (1/t1/\sqrt{t} decay), and main empirical conclusions remain unchanged. All updates in the revised version are targeted improvements following the reviewers' suggestions to polish the writing and further strengthen our analysis:

  • Core Novelty and Contributions: Going beyond prior and concurrent works that use ELR mainly for post-hoc explanation or empirical heuristics, we establish ELR scheduling through a complete closed loop: we show via bidirectional blockwise matching that ELR governs WN and WD dynamics (Sec. 2), analytically derive 1/t1/\sqrt{t} decay as the optimal ELR schedule and propose AIR under WN (Sec. 3), and validate AIR across 128B- and 500B-token MoE pretraining, outperforming cosine and 11 schedule families, identifying late-stage scheduling as the main source of the gain, and showing that AIR's ELR transfers back to WD (Sec. 4–5, Fig. 2b).
  • Presentation Polishing (especially Sec. 1–3 and Fig. 2, Reviewers 5yQW & jRrK): We polished the writing, logical flow, figure labels, contribution structure, and intermediate steps in Sections 1–3 to explain our ideas and derivations much more clearly, without changing the underlying logic or conclusions.
  • Generalized Derivation and Verification (Sec. 3, Fig. 3, App. B, Reviewers 5yQW & jRrK): We further generalized our derivation by relaxing the orthogonal-update and constant-drift conditions (still yielding the same optimal shifted 1/t1/\sqrt{t} schedule under slowly varying drift), added MoE measurements confirming these conditions within 1.0%–1.9% error (Fig. 3, P5), and proved AIR's rate-preservation uniqueness (Eq. 18, App. B.7, B.9).
  • Added Baseline Search, WD Comparisons, and Toy-Model Analysis (Table 1, App. C.1 and C.5, Figs. 8–9, Reviewers jRrK & zQtz): We added the WN cosine peak-LR search (Fig. 8, P23) ensuring an optimal baseline, which already suffices to prove AIR's gain over the baseline, together with direct comparisons showing WN + AIR outperforms standard WD + Cosine/WSD alongside their actual ELR trajectories (Table 6, Fig. 9, P24–25) and a closed-form toy-model and linear-regression ELR analysis (Eqs. 38–44, P25–26).
  • Expanded Related Work (Sec. 6, P10): We added explicit discussions positioning our work relative to Hyperball and concurrent ELR studies, prior horizon-free 1/t1/\sqrt{t} schedules, and global ELR matching in unregularized training.

We sincerely appreciate the reviewers' thoughtful feedback and the Area Chair's guidance, which have helped us make the paper much clearer and stronger. We hope our responses and updates address the reviewers' concerns, and we kindly look forward to a re-evaluation of our paper. We are happy to provide any further clarification during the discussion period.

Warm regards,

Authors of Submission #24723

AuthorsOpenReview ↗

Final Global Summary

Dear Area Chair and Reviewers,

As the discussion period draws to a close, we sincerely thank the Area Chair for guiding the review process and all reviewers for their time, thoughtful and constructive feedback. Below we briefly summarize our core contributions, our latest polishing revision, and the status of each reviewer's feedback.

1. Core Novelty and Contributions

Going beyond prior and concurrent works that use ELR mainly for post-hoc explanation or empirical heuristics, we establish ELR scheduling through a complete closed loop: we show that ELR governs WN and WD dynamics (Sec. 2), analytically derive 1/t1/\sqrt{t} decay as the optimal ELR schedule and propose AIR under WN (Sec. 3), and validate AIR on large-scale MoE pretraining, identifying late-stage scheduling as the main source of the gain and showing that AIR's ELR transfers back to WD (Sec. 4–5). Throughout all revisions, our core principle, proposed AIR schedule, and main empirical conclusions remain solid and unchanged.

2. Further Writing and Exposition Polishing

Following the latest feedback, we further polished the writing, mathematical flow, and intermediate steps in Section 3 and Appendix B, clarified the motivation and derivation of AIR, and refined our Related Work summaries (Sec. 6), including a reader-perspective pass with a colleague, to make the paper much smoother and easier to follow, without changing the underlying logic or conclusions.

3. Summary of Reviewer Discussions and Status

  • Reviewer 5yQW (Score: 2, supportive of acceptance with writing refinement): Explicitly stated that "I do like the idea and experiments in this paper. If the writing quality is ok and I could fully understand the derivation, then I will surely vote for acceptance." The reviewer's remaining concern focused on the clarity of the math exposition and AIR derivation, which our latest revision already refined.
  • Reviewer jRrK (Score: 1 →\to 2): The reviewer's score increased from 1 to 2 after we generalized the derivation beyond orthogonal updates and constant (a,b)(a, b) with MoE validation, added affine optimality proofs and the WN cosine peak-LR search, and clarified that all revisions keep our core claims, AIR schedule, and main results unchanged.
  • Reviewer zQtz (Score: 2 →\to 3): The reviewer's score increased from 2 to 3 after we added direct comparisons against standard WD + Cosine and WD + WSD with actual ELR trajectories, clarified our distinction from D'Angelo et al. (2024), and provided the toy-model ELR analysis.

We once again thank the Area Chair and all reviewers for their valuable feedback, which has substantially improved the clarity and quality of our paper. We kindly look forward to the final evaluation and will continue to refine our paper for the camera-ready version.

Warm regards,

Authors of Submission #24723

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.