acceptodds
Under review as a conference paper at ICLR 2027

Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining

18 views

Abstract

We establish ELR collapse as a quantitative empirical law in language model pretraining: learning rate (LR) and parameter norm govern loss dynamics primarily through their ratio, the effective learning rate (ELR). When tensor-wise ELRs are matched across runs, their loss trajectories collapse throughout training despite substantially different LRs and parameter norms. Across optimizers, architectures, datasets, and model scales, mean collapse errors are typically a few , below the seed-to-seed variation measured in a representative configuration. Systematic ablations identify normalization design and the timescale of LR–norm variation as key determinants of collapse precision. In standard training, tensor-wise ELRs approximately synchronize, following a shared temporal profile up to fixed amplitudes. Controlled interventions further show that adjusting only the base LR to match an aggregate ELR reproduces the loss dynamics induced by weight decay and Hyperball to high precision. Replacing LR with ELR enables a fitted functional scaling law (FSL) to transfer across norm-control methods without refitting. The resulting FSL also explains delayed acceleration, where norm control initially raises loss but improves it later, and guides the design of an ELR schedule that further lowers final loss. Together, these findings identify ELR as an intrinsic clock rate governing loss dynamics across LR schedules and norm-control methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
effective learning rate, training dynamics, language model pretraining, norm control, weight decay, scaling laws

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Fs6KOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces the concept of effective learning rate (ELR) collapse, based on the observation that the learning rate and weight magnitude jointly determine training dynamics. In particular, training runs with different learning rate and weight norm schedules can follow nearly identical loss trajectories when their effective learning rates, defined as the learning rate divided by the weight norm, match.

Critical strengths

The paper presents an interesting observation regarding the approximate equivalence of training dynamics under different configurations. In particular, the authors establish ELR collapse across architectures, datasets, optimizers, and scales, showing that training trajectories can follow approximately the same temporal profile, with different amplitudes.

The practical implications are also interesting. The authors demonstrate approximate ELR collapse across different architectures, datasets, optimizers, and scales. They further use this observation to improve loss prediction across norm-controlled methods, explain the delayed benefits of weight decay, and support improved learning rate schedule design.

Critical weaknesses

I did not identify any major flaws in the paper. My main reservation concerns the extent of its novelty. The underlying concept appears to have been hinted at in previous work, particularly in connection with exact scale invariance. While the present paper provides interesting empirical observations, I would appreciate a clearer explanation of the magnitude of the conceptual advance over existing results.

Questions for authors

1- Could the authors elaborate on the extent of the novelty of ELR collapse relative to previous results on exact scale invariance?

2- I would appreciate a brief intuition for why the effective learning rate, as defined in Equation (1), depends on the norm of the parameters themselves rather than their changes during training. Intuitively, one might expect the relationship between the learning rate and parameter updates to be relevant to the training dynamics. Is the observed behavior related to some form of sparsification, where many weights approach zero and the weight norm decreases significantly, or is there another explanation?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Fs6K: Novelty and the Definition of ELR

We thank the reviewer for their positive assessment and thoughtful questions.


[W-1 & Q-1] What does ELR collapse add beyond prior results on exact scale invariance?

Our contribution goes beyond exact scale invariance. Exact scale invariance motivates ELR, but does not establish high-precision loss collapse in practical LLM pretraining, where that symmetry is absent. We characterize the precision, scope, and conditions of this empirical law and establish its predictive value through cross-method loss prediction and ELR-schedule design. Please see the global response for the further clarification and additional results.

$ $

[Q2] why the effective learning rate, as defined in Equation (1), depends on the norm of the parameters themselves rather than their changes during training.

We thank the reviewer for ask this insightful question. Consider an optimizer update xk+1=xk−ηkUkx_{k+1}=x_k-\eta_kU_k, with ELR ek:=ηk/∥xk∥e_k:=\eta_k/\|x_k\|. If we understand correctly, you are asking why we use eke_k rather than the angular update size ∥x^k+1−x^k∥≈ek∥Uk∥\|\hat{x}_{k+1}-\hat{x}_k\|\approx e_k\|U_k\| as ELR, which has been adopted in recent work, including Wen et al. (2026); Yang et al. (2026).

ELR appears naturally in the direction dynamics. Writing x^k=xk/∥xk∥\hat x_k=x_k/\|x_k\|, we have

x^k+1=x^k−ekUk∥x^k−ekUk∥=x^k−ekPkUk+O(ek2∥Uk∥2),Pk=I−x^kx^k⊤.\hat x_{k+1}=\frac{\hat x_k-e_kU_k}{\|\hat x_k-e_kU_k\|}=\hat x_k-e_kP_kU_k+O(e_k^2\|U_k\|^2),\qquad P_k=I-\hat x_k\hat x_k^\top.

Thus, eke_k is the LR coefficient in the direction dynamics, while the resulting movement also depends on the update PkUkP_kU_k. This geometric explanation does not require sparsification or parameter norms to decrease. Note the actual angular movement also depends on ∥PkUk∥\|P_kU_k\|; ek∥Uk∥e_k\|U_k\| is an approximation of it.

Matching update size can change the stochastic update rule. For intuition, assume xkx_k lies on the unit sphere and follows

xk+1=N(xk−ekUk),Uk=Uˉ(xk)+ξk,x_{k+1}=\mathcal N(x_k-e_kU_k),\qquad U_k=\bar U(x_k)+\xi_k,

where N(y)=y/∥y∥\mathcal N(y)=y/\|y\|. If two runs use the same update law, matching {ek}\{e_k\} gives the matching dynamics

x~k+1=N(x~k−ekU~k).\tilde x_{k+1}=\mathcal N(\tilde x_k-e_k\tilde U_k).

The two runs can be viewed as the two indenpdent realization of same stochastic iteration. Therefore, it is naturel to expect that there loss dynamics collapse.

Now suppose we instead match the update size before projection. This requires

e~k∥U~k∥=ek∥Uk∥,e~k=ek∥Uk∥∥U~k∥.\tilde e_k\|\tilde U_k\|=e_k\|U_k\|,\qquad \tilde e_k=e_k\frac{\|U_k\|}{\|\tilde U_k\|}.

Then, the matching dynamics therefore become

x~k+1=N(x~k−ek∥Uk∥U~k∥U~k∥).\tilde x_{k+1}=\mathcal N\left(\tilde x_k-e_k\|U_k\|\frac{\tilde U_k}{\|\tilde U_k\|}\right).

To preserve the update distribution, i.e., x~k=dxk\tilde{x}_k\overset{d}{=} x_k, we would need

Uk=d∥Uk∥U~k∥U~k∥.(∗∗)U_k\overset{d}{=}\|U_k\|\frac{\tilde U_k}{\|\tilde U_k\|}.\qquad (**)

This equality generally fails even when UkU_k and U~k\tilde U_k are independent draws from the same distribution at the same state. Consequently, matching update size can change the stochastic dynamics. However, if the update magnitude ∥Uk∥\|U_k\| and direction Uk/∥Uk∥U_k/\|U_k\| are independent, condition (**) holds, and update-size matching preserves the update distribution.

For the two popular optimizers, we have:

  • Muon is a special case. For Muon, the update size ∥Uk∥\|U_k\| is fixed, so the condition (**) holds. ELR eke_k and update size ek∥Uk∥e_k\|U_k\| differ only by a constant factor.
  • Adam shows poorer collapse when matching update size. Adam’s update norms vary, so the two matching criteria are no longer related by a fixed factor. Our added AdamW experiment finds approximately four times higher mean collapse error when matching relative update size than when matching ELR (Figure B.8 in the revised manuscript). The mean collapse errors are shown below.
    Matching criterionMean collapse error
    ELR0.00470
    Relative update size0.01890

In summary, our analysis and experiments support using LR divided by parameter norm as ELR for describing loss dynamics. ELR and relative update size are conceptually distinct quantities, even though they differ only by a constant factor for Muon. We will make this distinction explicit in the revised manuscript. Due to time constraints, this clarification has not yet been incorporated.

Thank you for the detailed responses. The geometric explanation of ELR and the additional experiments address my second question. The predictive and schedule-design results also help clarify the contribution beyond earlier scale-invariance observations, although I still view the novelty as primarily empirical. The corrected seed-to-seed comparison should also be reflected in the manuscript's claims about collapse precision. I maintain my rating.

Reviewer_DyZfOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines whether effective learning rate (ELR), defined as learning rate divided by parameter RMS norm, explains loss dynamics during language model pretraining. Matching tensor-wise ELRs produces closely aligned loss trajectories across different learning-rate and norm schedules. The paper also studies aggregate ELR matching, transfer of functional scaling laws (FSL) across norm-control methods, and ELR-based schedule design.

Critical strengths

I find the main empirical result convincing. The experiments cover dense, MoE, and linear-attention models up to 1B parameters, several datasets, and AdamW, Muon, and Signum. Matching initialization and data order makes the interventions informative. The heterogeneous tensor-wise schedules provide a useful additional test.

The normalization and modulation-frequency ablations are particularly interesting because they identify conditions under which collapse becomes less precise. Reproducing weight-decay and Hyperball loss trajectories by changing the base LR also gives the observation practical relevance.

Critical weaknesses
  1. Novelty needs more precise positioning. D'Angelo et al. (2024) already matched LLM loss trajectories through ELR and explained the late benefit of weight decay through a bias–variance tradeoff. Li et al. (2025b) also studied trajectory invariance systematically. These are cited, but the contribution here is the broader, more precise characterization and the FSL extension. The explanation of delayed acceleration should more clearly acknowledge the earlier account. Alongside Xiao et al. (2026), the authors should also consider MuonMD, which combines sphere-constrained updates with separately learned magnitudes. This is directly relevant to the role of norm control and learnable gains studied here, both for testing the scope of ELR matching and as a schedule-design baseline.

  2. The FSL transfer result needs a stronger comparison. Appendix E replaces Hyperball's instantaneous update norms with fixed scales estimated from an AdamW reference run. Its FSL input is therefore a calibrated multiple of nominal LR, whereas AdamW uses measured parameter-norm trajectories. This is disclosed, but makes the comparison against raw-LR FSL difficult to interpret. An ablation should separate the benefit of calibration from that of tracking time-varying ELR, including sensitivity to the reference run and averaging window.

  3. The schedule improvement needs more careful baseline tuning. Table F.1 includes twelve AdamW configurations spanning weight decay and schedule choice, and the improvement also appears on validation loss. However, the reported setup fixes AdamW's peak LR rather than sweeping it. Muon is included in the collapse experiments but absent from this performance comparison. Peak LR and decay duration need careful tuning, with repeated seeds and the cost of fitting runs reported, before claiming a practical optimization advantage.

  4. The precision claims need better uncertainty estimates and clearer scope. Comparing matched-seed collapse errors against independent-seed variation provides context but does not establish that the collapse precision is reproducible. I would like repeated paired experiments and sensitivity to loss smoothing. The Hyperball aggregate-matching error also excludes training before step 7,500 because of instability. This is a substantial part of the run and limits the claim about reproducing full loss trajectories.

  5. The synchronization interpretation needs a stronger control. A dominant singular component in an uncentered ELR matrix can partly reflect positive means and the shared LR schedule. The rank-one matching experiment is useful, but comparison against a simple shared-LR approximation and analysis of relative norm changes would clarify what additional structure is being discovered.

  6. Reproducibility. I could not find an anonymous repository in the provided paper; the links to modded-nanogpt and nanoGPT are baseline repositories. Please provide anonymous code, exact configurations, and scripts for reproducing the interventions and FSL fits. For an empirical ICLR submission, I expect these materials to be available during review.

Questions for authors

A) How much of the FSL transfer gain remains against a calibrated-LR baseline?

B) Does the optimized schedule retain its advantage after careful baseline tuning and across seeds, including comparisons with Muon and MuonMD?

C) Can the main paired interventions be repeated, with errors reported over the entire intervention period?

4 comments
AuthorsOpenReview ↗

Response to Reviewer DyZf: Part I

We thank the reviewer for their supportive assessment and constructive comments. We address each concern below.

$ $


[W-1] Contributions relative to prior work and the scope of ELR collapse

Novelty and positioning. We have revised the manuscript to acknowledge more explicitly the earlier ELR-related results and explanations of weight decay’s late-stage benefit, including D’Angelo et al. (2024) and Li et al. (2025b). Our contribution is to characterize the precision, scope, and conditions of ELR collapse and test its value for prediction and schedule design, advancing ELR toward a high-precision, predictive coordinate for loss dynamics. Please see the global response for the supporting evidence.

MuonMD: additional collapse and schedule-design experiments. Following the reviewer’s suggestion, we added Hägele et al. [4] to Related Work (Section 2) and tested ELR matching with MuonMD, which learns weight directions and magnitude gains separately. The matched loss trajectories closely align (Appendix B.2.2):

ELR scheduleMean collapse error
Linear-up0.00081
Linear-down0.00051
Sinusoidal0.00141

We also applied FSL-based ELR-schedule design to MuonMD. At the same matrix peak LR, the optimized schedule improves final validation loss from 3.0297 to 2.9982 relative to the best tuned baseline. These results extend both the collapse evidence and the design procedure to MuonMD. Please see W-3 & Q-B and Appendix F.1, Table F.3, for details.

$ $


[W-2 & Q-A] How much of the FSL transfer gain comes from calibration?

Calibration converts AdamH’s nominal LR into the ELR coordinate used by the fitted elr-FSL. The FSL is fitted using AdamW’s ELRs. Applying it to AdamH therefore requires expressing AdamH’s schedule in the same ELR coordinate. This conversion depends on AdamH’s update norms (Section 3). We approximate these norms using fixed scales estimated from an AdamW reference run; the calibration does not use the target Hyperball runs’ loss trajectories, and the FSL is not refitted.

Directly measured ELR gives slighly better predictions. To test whether the transfer accuracy depends on this approximation, we evaluated the same fitted elr-FSL using instantaneous AdamH update norms on two held-out Hyperball runs (Appendix E, Figure E.2):

FSL modelInput on Hyperball runsCosine RMSEWSD-10% RMSE
LR-based FSLNominal LR0.121250.33889
ELR-based FSLCalibrated ELR0.017680.01666
ELR-based FSLInstantaneous ELR0.015390.01576

Instantaneous ELR yields slightly lower errors, confirming that accurate transfer also holds without the fixed-scale approximation. The small difference between the last two rows shows that tracking instantaneous fluctuations provides only a modest additional benefit in these runs.

Fixed calibration enables prediction before training. Instantaneous update norms are unavailable before running AdamH. The reference-based estimate allows us to predict loss from a prescribed schedule in advance. Figure E.1 shows that the calibrated ELR closely tracks the instantaneous ELR profile, supporting its use for prospective prediction and schedule design.

AuthorsOpenReview ↗

Response to Reviewer DyZf: Part II


[W-3 & Q-B] Does the optimized ELR schedule retain its advantage over carefully tuned baselines?

Yes. The improvement persists across Adam, Muon, and MuonMD after baseline tuning. Following the reviewer’s suggestion, we tuned the peak LR for WSD, cosine, and 7111 schedules, and additionally tuned the decay ratio for WSD. We also extended our FSL-based ELR-schedule design to Muon and MuonMD. The optimized schedules improve final validation loss by approximately 0.03 over the best tuned baseline for all three optimizers.

ScheduleAdamMuonMuonMD
Best tuned baseline3.06703.03753.0297
FSL-optimized3.03523.00302.9982

Appendix F of the revised manuscript provides the tuning details and full results.

$ $


[W-4 & Q-C] Are collapse errors reproducible, robust to smoothing, and representative of the full training period?

Paired repetitions across seeds. We repeated the Figure 1 experiment with two additional initialization seeds, running the WSD reference and all three ELR-matched schedules under each seed. We obtained the following collapse errors (Table B.1, Appendix B.2.1; values in units of 10−310^{-3}):

ELR-matched scheduleOriginal seedSeed 17Seed 314159
Linear-up1.793.801.60
Linear-down2.462.851.81
Sinusoidal2.583.263.23

The paired errors remain on the order of a few 10−310^{-3} across all three seeds. For context, our original experiments measured mean seed-to-seed loss discrepancies of 0.0111 for initialization and 0.0162 for data order (Figure B.1). These provide a reference scale; the paired repetitions above directly assess reproducibility.

Sensitivity to loss smoothing. We recomputed the Figure 1 errors over the same training steps with different EMAs (Table B.2, Appendix B.2.1). All values below are in units of 10−310^{-3}.

ELR-matched schedule0.99 (paper)0.980.960.800 (no smoothing)
Linear-up1.792.012.323.265.43
Linear-down2.462.622.853.625.55
Sinusoidal2.582.722.953.765.68

LR-only matching to MuonH. We reanalyzed the runs in Figure 5(b) over all steps:

Training stepsMean absolute loss difference
1–7,4990.01531
7,500–20,4000.00145
1–20,400 (full run)0.00654

The early discrepancy is substantially larger, so the previously reported high-precision agreement applies to the later phase. Despite further investigation during the rebuttal period, we have not yet identified the cause of this early deviation. We have not observed a comparable deviation in our Adam experiments.


AuthorsOpenReview ↗

Response to Reviewer DyZf: Part III

$ $


[W-5] Does ELR synchronization reflect structure beyond the shared LR schedule?

Recall that E=(ηi,keff)∈RK×N\mathbf E=(\eta_{i,k}^{\mathrm{eff}})\in\mathbb R^{K\times N} collects the blockwise ELRs, where KK and NN denote the numbers of blocks and training steps, respectively. Following the reviewer’s suggestion, we compare the following approximations. All errors are measured using the relative Frobenius norm.

  • Mean-ELR approximation. Let sˉ∈RN\bar{\mathbf s}\in\mathbb R^N be the ELR averaged across blocks, with sˉk=K−1∑iEi,k\bar s_k=K^{-1}\sum_i E_{i,k}. We measure ∥∥E−1sˉ⊤∥∥F∥∥E∥∥F.\frac{\|\|\mathbf E-\mathbf 1\bar{\mathbf s}^{\top}\|\|_F}{\|\|\mathbf E\|\|_F}. Here, 1∈RK\mathbf 1\in\mathbb R^K is the all-ones vector. This approximation assigns the same ELR trajectory to every block, without allowing block-specific amplitudes.
  • Shared-LR approximation. To test how much of the rank-one structure can be explained by the shared nominal LR schedule, we measure min⁡a∈RK∥∥E−aη⊤∥∥F∥∥E∥∥F,\min_{\mathbf a\in\mathbb R^K}\frac{\|\|\mathbf E-\mathbf a \eta^\top\|\|_F}{\|\|\mathbf E\|\|_F}, where η=(η1,…,ηN)⊤\eta=(\eta_1,\ldots,\eta_N)^\top. This allows a separate amplitude for each block while fixing the temporal profile to the nominal LR schedule.
  • Inverse-norm rank-one. Let Q^\widehat{\mathbf Q} be the best rank-one approximation to the inverse-norm matrix Q\mathbf Q . We find this rank-one approximation error is roughly 5.8%-6.5% in the two training setup tested below. We then further use this to explain the rank-one property of ELR matrix. Specifically, we multiply it by the shared LR schedule to reconstruct the ELR matrix: $ \widehat{\mathbf E}=(\mathbf{1}\eta^\top)\odot\widehat{\mathbf Q}. $ The reported error is ∥∥E−E^∥∥F∥∥E∥∥F.\frac{\|\|\mathbf E-\widehat{\mathbf E}\|\|_F}{\|\|\mathbf E\|\|_F}.

For comparison, “ELR rank-one (SVD)” reports the best unconstrained rank-one approximation to E\mathbf E. The results below use AdamW with a cosine LR schedule.

ConfigurationELR rank-one (SVD)Mean-ELRShared-LRInverse-norm rank-one (SVD)
WD = 06.30%17.86%44.74%6.84%
WD = 0.16.97%16.32%28.59%8.49%

Interpretation. Shared LR alone gives a substantially poorer fit, while the inverse-norm reconstruction approaches the optimal ELR rank-one fit. This supports approximately proportional inverse-norm evolution across blocks as a major source of ELR synchronization. The larger Mean-ELR errors further highlight the importance of block-specific amplitudes.

$ $


[W-6] I could not find an anonymous repository ... For an empirical ICLR submission, I expect these materials to be available during review.

We thank the reviewer for raising this point. Our implementation is now available in an anonymous repository; see elr-experiments. We will add the link to the next revision of the manuscript.

AuthorsOpenReview ↗

Clarifying the Reference Scale for ELR-Collapse Precision

Thank you again for raising the invaluable question of collapse precision and its reference scale. Following your comment and the AC’s follow-up, we clarified two points:

  • The original seed-variation baseline used fixed RMSNorm gains, whereas the ELR-matching runs use trainable gains. With the updated trainable-gain baseline, ELR-collapse errors and initialization-seed variation are of the same order.
  • Comparing losses on a fixed validation set removes the evaluation-batch fluctuations present in training-loss comparisons across different data orders, providing a clearer reference scale.

We refer to our response to the AC for the detailed comparisons. Your comment helped us make this comparison more consistent, and we will incorporate these results and clarifications into the next manuscript version.

Reviewer_o7mwOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper convincingly establishes the ELR collapse phenomenon, i.e., matching effective learning rates yields nearly identical loss trajectories, in controlled experiments, and further observes time synchronization of tensor-wise ELRs in standard training. It further uses an lr-control approach to reproduce norm-control methods based on the first two observations. With these tools, it verifies that an ELR-based functional scaling law can transfer and guide better ELR scheduling, which promisingly promotes the significance of ELR schedule design. Overall, the paper provides a well-executed empirical study that identifies an intriguing phenomenon and develops meaningful practical applications.

Critical strengths

(1) The paper provides convincing empirical evidence for its claimed results. (2) Its results provide an insightful understanding of training dynamics. ELR collapse and approximate synchronization suggest that LR scheduling and norm control may largely operate through one shared variable, which is an inspiring perspective for optimizers and practical schedule design.

Critical weaknesses

None.

Questions for authors
  1. Could the authors further clarify why an aggregate ELR is used instead of tensor-wise ELRs (Section 6)? What additional complexity would tensor-wise ELR matching introduce?
  2. In Section 8, the proposed schedule is derived from an FSL fitted to several specific training configurations. How generalizable is the optimized ELR schedule beyond the fitted regime?
  3. The optimized schedule is further realized with Hyperball. Could standard AdamW be used to match the same target aggregate ELR schedule and achieve a similar loss? In other words, is explicit norm control necessary to obtain observed improvement?
2 comments
AuthorsOpenReview ↗

Response to Reviewer o7mw: ELR Matching, Schedule Generalization, and AdamW Realization

We thank the reviewer for their strong support of our work and insightful questions.


[Q-1] Why aggregate rather than tensor-wise ELR matching, and what additional complexity would tensor-wise matching introduce?

Aggregate matching separates ELR scheduling from blockwise preconditioning. For block kk, the ELR is ηkeff(t)=ηt∥Wk(t)∥RMS.\eta_k^{\mathrm{eff}}(t)=\frac{\eta_t}{\|W_k(t)\|_{\mathrm{RMS}}}. By changing parameter norms, weight decay can affect both:

  • ELR scheduling: how ELRs evolve during training.
  • Blockwise preconditioning: the relative ELRs across blocks, since their norms can evolve differently.

Tensor-wise matching would reproduce both effects, leaving their respective contributions unresolved. Our base-LR-only experiments in Section 6 show that matching the aggregate ELR largely reproduces weight decay’s effects on loss dynamics. This suggests that ELR scheduling accounts for the main effects, with limited contribution from blockwise preconditioning.

Tensor-wise matching requires maintaining a separate LR schedule for each tensor. This is straightforward to implement and incurs little additional computational overhead, as demonstrated in Section 4.

$ $


[Q-2] Can the optimized ELR schedule generalize beyond the training configurations used for fitting?

The schedule is configuration-dependent; the design procedure applies across configurations. We do not assume that a schedule optimized for one configuration remains optimal for another. FSL parameters depend on the optimizer, architecture, and dataset. For a new configuration, one can refit the FSL and optimize the corresponding ELR schedule using the same fit–optimize–realize procedure.

Additional experiments extend the procedure to Muon and MuonMD. Following the reviewers’ suggestions, we also strengthened the baselines by tuning peak LRs for WSD, cosine, and 7111 schedules, together with the WSD decay ratio. The FSL-optimized schedules improve final validation loss by approximately 0.03 over the best tuned baseline for all three optimizers:

ScheduleAdamMuonMuonMD
Best tuned baseline3.06703.03753.0297
FSL-optimized3.03523.00302.9982

Appendix F reports the baseline tuning, FSL fits, optimized schedules, and final-loss comparisons.

The optimal schedule depends on the training configuration. The recent work (Li et al., COLT 2026) characterizes optimal schedules under FSL asymptotically, showing that optimal schedules indeed depend on the problem parameters and stability constraint. Section 8 demonstrates the effectiveness of ELR-first schedule design. Identifying ELR schedules that remain effective across configurations, including different batch sizes, is an important direction for future work.

$ $


[Q-3] The optimized schedule is further realized with Hyperball. Could standard AdamW be used to match the same target aggregate ELR schedule and achieve a similar loss? In other words, is explicit norm control necessary to obtain observed improvement?

Yes—for the optimized ELR schedule tested here, adjusting AdamW’s base LR gives nearly the same loss as Hyperball. We tested three ways to implement the same ELR schedule:

  • AdamW with base-LR adjustment, following Section 6.
  • Norm control, Wk+1=Norm⁡(Wk−ηkUk)W_{k+1}=\operatorname{Norm}(W_k-\eta_kU_k) (no update size normalization).
  • Hyperball, Wk+1=Norm(Wk−ηkNorm(Uk))W_{k+1}=\mathrm{Norm}(W_k-\eta_k\mathrm{Norm}(U_k)).

The loss curves closely match (Figure F.2), and the final losses differ by only a few times 10−310^{-3} (Table F.1). For convenience, the final losses are reported below:

RealizationFinal training lossFinal validation loss
Adam with Hyperball3.00393.0352
AdamW with LR adjustmnet3.00643.0384
Adam with norm control3.00533.0370

Explicit norm control may expand the stable ELR range. Different realizations may support different ranges of ELR schedules. With small weight decay, AdamW’s parameter norms can grow large, so reaching a high ELR requires a large base LR, which may destabilize training. Explicit norm control may allow higher ELRs while maintaining stability. Understanding this possible advantage is an important direction for practical ELR-based training.

Thank you for your clarifications. They have addressed my previous questions, and I find the work clear and inspiring overall. I will keep my original positive rating.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Global Respone

We thank all reviewers for their constructive feedback. Below, we clarify our contribution and summarize the additional results. Due to time constraints, the main text of the revised manuscript does not yet fully reflect these updates; we will incorporate them into the final version.


1. Our contribution and novelty

ELR collapse as a quantitative empirical law. Building on prior work, we systematically investigate whether ELRs accurately describe loss dynamics in LLM pretraining despite the absence of exact scale invariance. Three levels of evidence support this description:

  • Precision, scope, and conditions. Matched ELRs yield loss discrepancies of only a few times 10−310^{-3} across diverse optimizers, architectures, datasets, and scales. Normalization and modulation-frequency ablations identify conditions under which this precision deteriorates (Section 4).
  • Intervention and prediction. Base-LR-only interventions reproduce the main loss effects of norm control, and an ELR-based FSL predicts trajectories under an unseen norm-control method without refitting (Sections 6–7).
  • Design. Optimizing the ELR schedule improves final loss, and different realizations of that ELR schedule achieve nearly identical losses (Section 8 and the added experiments).

From empirical collapse to prediction and design. These results establish ELR as a high-precision, predictive coordinate for LLM pretraining loss dynamics. They also clarify norm control’s role: the observed loss improvements from weight decay and Hyperball arise primarily through the ELR schedules they induce, while norm regulation may expand the range of schedules that can be realized stably. This provides a scientific basis for studying and designing training directly in ELR coordinates.

$ $

2. Additional evidence for FSL-based ELR-schedule design

The improvement persists after baseline tuning and extends to Muon and MuonMD. Following Reviewer DyZf’s suggestion, we tuned peak LRs for WSD, cosine, and 7111 schedules, as well as the decay ratio of WSD. We also extended the FSL-based design procedure to Muon and MuonMD (Hägele et al., 2026). The optimized schedules improve final validation loss by approximately 0.03 over the best tuned baseline for all three optimizers:

ScheduleAdamMuonMuonMD
Best tuned baseline3.06703.03753.0297
FSL-optimized3.03523.00302.9982

Different realizations of the same optimized ELR schedule achieve nearly identical losses. For Adam, we compared three ways of realizing the same optimized ELR schedule: Hyperball, explicit norm control, and AdamW with base-LR adjustment (the approach used in Section 6). Their final losses differ by only a few times 10−310^{-3}:

RealizationFinal training lossFinal validation loss
Adam with Hyperball3.00393.0352
AdamW with base-LR adjustment3.00643.0384
Adam with norm control3.00533.0370

These results support the optimized ELR schedule as the main source of the observed loss improvement. Appendix F of the revised manuscript provides the tuning details and additional results.

Area_Chair_C25hOpenReview ↗

Discussion deadline approaching

Dear Reviewers and Authors,

Thank you for providing detailed reviews and the rebuttal.

To the reviewers: As the author–reviewer discussion deadline approaches, please review the authors’ responses and indicate whether your concerns are addressed. I would particularly appreciate feedback on the contribution and novelty concerns raised by Reviewers DyZf and Fs6K.

To the authors: After carefully reading the rebuttal, I have a follow-up question about your response to W4 from Reviewer DyZf. Specifically, could you please elaborate on the comparison of collapse errors between the ELR-matched runs and the seed-to-seed and data-order baselines:

  • According to Appendix A.2, the ELR-matching experiments use a joint warm-up phase (lines 855-858), while the seed-to-seed and data-order baselines use the fully independent trajectories. Can this mismatch between the settings explain the higher collapse error for the baselines?
  • Based on the results added to Appendix B.2.1 during the rebuttal, the seed-to-seed collapse error seems to be much lower on average than the initially reported 1.11×10⁻² (most of the 66 pairs correspond to experiments with different seeds). Does this mean that, the collapse errors of ELR-matched pairs are actually of the same magnitude as the seed-to-seed ones, not several times lower?

Kind regards,

AC

AuthorsOpenReview ↗

Response to the AC: Clarifying the Seed-Induced Reference Scale for ELR Collapse

Dear AC,

Thank you for your thoughtful follow-up questions. We understand the concern: cross-seed ELR matching changes both the initialization seed and the ELR realization, yet the collapse error appears smaller than the original seed-to-seed baseline.

After carefully checking the code, we found that the original baseline and ELR-matching experiments used different setups. Our original seed-variation experiments used the official modded-nanogpt codebase, which uses RMSNorm without trainable gains. However, the ELR-matching runs use trainable gains. These are therefore not directly comparable.

 


1. ELR-collapse errors are of the same order as seed-to-seed variation.

We repeated the initialization-seed experiments with trainable gains and explicitly compared same-seed and cross-seed collapse errors. The table reports mean absolute differences of training loss with training EMA 0.99 and unsmoothed validation loss:

ComparisonTrainingValidationValidation (last 500 steps)
ELR matching: same seed0.00230.00350.0044
ELR matching: different init seeds0.00330.00420.0042
Same config: different init seeds0.00330.00320.0035
Same config: different data-order seeds0.01490.00380.0012

Validation-loss difference provides a more reasonable reference. Training losses under different data orders are evaluated on different minibatches, so their differences reflect both model differences and batch-to-batch fluctuations. Evaluating all runs on the same validation set removes the latter. In the data-order experiments, the mean difference is 0.0149 on training batches and 0.0038 on the fixed validation set.

 


2. Shared warm-up improves collapse precision but is not necessary for collapse.

We tested matching from steps 0, 500, and 1,000. Mean absolute validation-loss differences over the same interval, steps 1,001–10,200, are:

ScheduleStart = 0Start = 500Start = 1,000
Linear-down0.00820.00460.0032
Linear-up0.00250.00150.0014
Cosine wave0.00900.00470.0040

A longer shared phase improves collapse precision, so shared warm-up can contribute to the smaller matching errors, as the AC suggested. However, the original comparison also involved different gain configurations. With trainable gains in both settings, seed-to-seed variation and ELR-matching errors are already of the same order.

 


Thank you again for raising these invaluable questions. They helped us make the seed-to-seed comparison more consistent and establish a more scientific reference scale for ELR-collapse precision. We will incorporate these results and clarifications into the next version of the manuscript.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.