- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces the concept of effective learning rate (ELR) collapse, based on the observation that the learning rate and weight magnitude jointly determine training dynamics. In particular, training runs with different learning rate and weight norm schedules can follow nearly identical loss trajectories when their effective learning rates, defined as the learning rate divided by the weight norm, match.
Critical strengths
The paper presents an interesting observation regarding the approximate equivalence of training dynamics under different configurations. In particular, the authors establish ELR collapse across architectures, datasets, optimizers, and scales, showing that training trajectories can follow approximately the same temporal profile, with different amplitudes.
The practical implications are also interesting. The authors demonstrate approximate ELR collapse across different architectures, datasets, optimizers, and scales. They further use this observation to improve loss prediction across norm-controlled methods, explain the delayed benefits of weight decay, and support improved learning rate schedule design.
Critical weaknesses
I did not identify any major flaws in the paper. My main reservation concerns the extent of its novelty. The underlying concept appears to have been hinted at in previous work, particularly in connection with exact scale invariance. While the present paper provides interesting empirical observations, I would appreciate a clearer explanation of the magnitude of the conceptual advance over existing results.
Questions for authors
1- Could the authors elaborate on the extent of the novelty of ELR collapse relative to previous results on exact scale invariance?
2- I would appreciate a brief intuition for why the effective learning rate, as defined in Equation (1), depends on the norm of the parameters themselves rather than their changes during training. Intuitively, one might expect the relationship between the learning rate and parameter updates to be relevant to the training dynamics. Is the observed behavior related to some form of sparsification, where many weights approach zero and the weight norm decreases significantly, or is there another explanation?
2 comments
Thank you for the detailed responses. The geometric explanation of ELR and the additional experiments address my second question. The predictive and schedule-design results also help clarify the contribution beyond earlier scale-invariance observations, although I still view the novelty as primarily empirical. The corrected seed-to-seed comparison should also be reflected in the manuscript's claims about collapse precision. I maintain my rating.
Response to Reviewer Fs6K: Novelty and the Definition of ELR
We thank the reviewer for their positive assessment and thoughtful questions.
Our contribution goes beyond exact scale invariance. Exact scale invariance motivates ELR, but does not establish high-precision loss collapse in practical LLM pretraining, where that symmetry is absent. We characterize the precision, scope, and conditions of this empirical law and establish its predictive value through cross-method loss prediction and ELR-schedule design. Please see the global response for the further clarification and additional results.
$ $
We thank the reviewer for ask this insightful question. Consider an optimizer update xk+1=xk−ηkUk, with ELR ek:=ηk/∥xk∥. If we understand correctly, you are asking why we use ek rather than the angular update size ∥x^k+1−x^k∥≈ek∥Uk∥ as ELR, which has been adopted in recent work, including Wen et al. (2026); Yang et al. (2026).
ELR appears naturally in the direction dynamics. Writing x^k=xk/∥xk∥, we have
x^k+1=∥x^k−ekUk∥x^k−ekUk=x^k−ekPkUk+O(ek2∥Uk∥2),Pk=I−x^kx^k⊤.
Thus, ek is the LR coefficient in the direction dynamics, while the resulting movement also depends on the update PkUk. This geometric explanation does not require sparsification or parameter norms to decrease. Note the actual angular movement also depends on ∥PkUk∥; ek∥Uk∥ is an approximation of it.
Matching update size can change the stochastic update rule. For intuition, assume xk lies on the unit sphere and follows
xk+1=N(xk−ekUk),Uk=Uˉ(xk)+ξk,
where N(y)=y/∥y∥. If two runs use the same update law, matching {ek} gives the matching dynamics
x~k+1=N(x~k−ekU~k).
The two runs can be viewed as the two indenpdent realization of same stochastic iteration. Therefore, it is naturel to expect that there loss dynamics collapse.
Now suppose we instead match the update size before projection. This requires
e~k∥U~k∥=ek∥Uk∥,e~k=ek∥U~k∥∥Uk∥.
Then, the matching dynamics therefore become
x~k+1=N(x~k−ek∥Uk∥∥U~k∥U~k).
To preserve the update distribution, i.e., x~k=dxk, we would need
Uk=d∥Uk∥∥U~k∥U~k.(∗∗)
This equality generally fails even when Uk and U~k are independent draws from the same distribution at the same state. Consequently, matching update size can change the stochastic dynamics. However, if the update magnitude ∥Uk∥ and direction Uk/∥Uk∥ are independent, condition (**) holds, and update-size matching preserves the update distribution.
For the two popular optimizers, we have:
In summary, our analysis and experiments support using LR divided by parameter norm as ELR for describing loss dynamics. ELR and relative update size are conceptually distinct quantities, even though they differ only by a constant factor for Muon. We will make this distinction explicit in the revised manuscript. Due to time constraints, this clarification has not yet been incorporated.