- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the effects of temporary weight decay during LLM pretraining with Muon. The authors show that the loss measured when weight decay is disabled can be misleading: an intervention that initially hurts performance may eventually become beneficial. By varying the start time, duration, and evaluation point, they find that early weight decay keeps certain weight matrices smaller, changing the magnitude of subsequent updates relative to the weights. This provides a possible explanation for why temporary interventions can have lasting effects. However, it remains unclear how far this explanation extends beyond weight decay and why changes in relative update magnitude can help at one training stage but hurt at another.
Critical strengths
-
The paper highlights an important distinction between the loss at the end of a temporary intervention and its eventual effect on training. An intervention that initially appears harmful may become beneficial later.
-
The authors connect weight magnitude to the relative size of subsequent updates, providing a concrete explanation for how temporary weight decay can continue to affect training after it is disabled.
Critical weaknesses
-
The paper raises a general question about temporary optimization interventions, but its explanation relies heavily on the relationship between weight magnitude and relative update size under Muon. Other interventions, such as Gram matrix regularization, can also alter weight magnitudes. It is unclear whether the proposed explanation extends to these interventions.
-
The analysis explains how differences in relative update magnitude persist, but not why they are beneficial early in training and detrimental later. Since larger relative updates do not necessarily improve loss, relative update magnitude alone cannot explain the final outcomes.
Questions for authors
-
Could a learning-rate schedule reproduce the relative update magnitudes observed after temporary weight decay is disabled? If this achieves similar final performance, is temporary weight decay essentially acting as an indirect learning-rate schedule?
-
What properties of the model state determine whether larger relative updates are beneficial or harmful? If temporary changes in learning rate and weight decay produce similar subsequent relative update magnitudes but different final losses, what other state variables explain the difference?
-
Can the eventual crossover between two training runs, and approximately when it occurs, be predicted from the weights, relative update magnitudes, and optimizer state at the end of the intervention, without training both runs to completion?
3 comments
Thanks to the author for the rebuttal. I think this question is a very interesting starting point, and there is much more interesting exploration that can be done on this basis.
Thank you for the encouraging feedback and thoughtful questions. Please let us know if there’s anything else we can clarify. If you feel the revision addresses your concerns, we’d appreciate it if you would consider a higher rating.
Thank you for your thoughtful review.
Does the explanation extend to Gram regularization? Gram regularization also leaves larger relative steps after removal in the matrices it targets. Its raw gradient splits into a radial part, parallel to the weights, and a tangential part, perpendicular to them. Applied after Muon, the radial part nearly matches the full penalty's final loss, within 0.0005 nats; the tangential part alone gives no gain. Copying column norms recovers 62% of the gain, versus 27% for column correlations.
Applied before Muon's orthogonalization, neither part alone reproduces the full penalty. The split of the raw gradient therefore does not describe the update Muon actually applies. We will clarify this.
Q1: Is temporary decay an indirect learning-rate schedule? It acts partly through relative step size. The fixed-norm replay prescribes the measured relative-step schedule without decay and reproduces a comparable gain, mostly after the original decay window. It uses a modified optimizer and averages over the six matrix families, so it does not establish equivalence to a learning-rate schedule in ordinary Muon. Early decay also remains beneficial after tuning the constant learning rate (General response, item 3).
Q2: Which properties determine whether larger relative steps help? We cannot yet predict this. Early and late windows with the same coefficient, duration, and total shrinkage have opposite final effects. At 405M, resetting norms removes most but not all of the advantage. These results show that timing and other parts of the training history matter; they do not identify whether weight directions or momentum explain the remaining gain. The norm mechanism explains persistence, not the full ordering of schedules.
Q3: Can crossover time be predicted from the state at removal? The current analysis cannot predict it. The approximate recurrence describes how relative steps recover after removal, but does not predict loss or crossover time. This remains an open question.