acceptodds
Under review as a conference paper at ICLR 2027

Delayed Effects of Temporary Weight Decay in Muon Pretraining

Abstract

A temporary optimization intervention can increase validation loss when it ends yet reduce it at the final training budget. We demonstrate this with temporary weight decay in Muon-trained decoder-only Transformers with 162M, 405M, and 1.4B parameters. Within each scale, moving a fixed-duration decay window from early to late training reverses its final effect. At 162M, changing window duration alters final loss even at approximately matched total shrinkage. Most strikingly, a schedule selected at 162M and transferred without retuning to 1.4B is 0.26 nats worse than no decay when removed, overtakes it only after 73% of training, and ultimately finishes 0.028 nats better while also outperforming the best sampled constant-decay baseline. Early decay leaves a persistent weight-norm deficit, which keeps relative steps larger after the intervention ends. An approximate norm-dynamics recurrence captures the slow convergence of these relative steps, and fixed-norm replays of the measured trajectories reproduce the performance ordering and a similar loss difference. A separate Gram intervention exhibits related post-removal norm and relative-step dynamics. These results show that temporary optimization schedules have delayed effects long after removal and should therefore be evaluated at the intended final training budget, not only at removal.

Primary area:
optimization
Keywords:
optimization dynamics, Muon, weight decay, early training interventions, language model pretraining

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xeAJOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the effects of temporary weight decay during LLM pretraining with Muon. The authors show that the loss measured when weight decay is disabled can be misleading: an intervention that initially hurts performance may eventually become beneficial. By varying the start time, duration, and evaluation point, they find that early weight decay keeps certain weight matrices smaller, changing the magnitude of subsequent updates relative to the weights. This provides a possible explanation for why temporary interventions can have lasting effects. However, it remains unclear how far this explanation extends beyond weight decay and why changes in relative update magnitude can help at one training stage but hurt at another.

Critical strengths
  • The paper highlights an important distinction between the loss at the end of a temporary intervention and its eventual effect on training. An intervention that initially appears harmful may become beneficial later.

  • The authors connect weight magnitude to the relative size of subsequent updates, providing a concrete explanation for how temporary weight decay can continue to affect training after it is disabled.

Critical weaknesses
  • The paper raises a general question about temporary optimization interventions, but its explanation relies heavily on the relationship between weight magnitude and relative update size under Muon. Other interventions, such as Gram matrix regularization, can also alter weight magnitudes. It is unclear whether the proposed explanation extends to these interventions.

  • The analysis explains how differences in relative update magnitude persist, but not why they are beneficial early in training and detrimental later. Since larger relative updates do not necessarily improve loss, relative update magnitude alone cannot explain the final outcomes.

Questions for authors
  • Could a learning-rate schedule reproduce the relative update magnitudes observed after temporary weight decay is disabled? If this achieves similar final performance, is temporary weight decay essentially acting as an indirect learning-rate schedule?

  • What properties of the model state determine whether larger relative updates are beneficial or harmful? If temporary changes in learning rate and weight decay produce similar subsequent relative update magnitudes but different final losses, what other state variables explain the difference?

  • Can the eventual crossover between two training runs, and approximately when it occurs, be predicted from the weights, relative update magnitudes, and optimizer state at the end of the intervention, without training both runs to completion?

3 comments

Thank you for your thoughtful review.

Does the explanation extend to Gram regularization? Gram regularization also leaves larger relative steps after removal in the matrices it targets. Its raw gradient splits into a radial part, parallel to the weights, and a tangential part, perpendicular to them. Applied after Muon, the radial part nearly matches the full penalty's final loss, within 0.0005 nats; the tangential part alone gives no gain. Copying column norms recovers 62% of the gain, versus 27% for column correlations.

Applied before Muon's orthogonalization, neither part alone reproduces the full penalty. The split of the raw gradient therefore does not describe the update Muon actually applies. We will clarify this.

Q1: Is temporary decay an indirect learning-rate schedule? It acts partly through relative step size. The fixed-norm replay prescribes the measured relative-step schedule without decay and reproduces a comparable gain, mostly after the original decay window. It uses a modified optimizer and averages over the six matrix families, so it does not establish equivalence to a learning-rate schedule in ordinary Muon. Early decay also remains beneficial after tuning the constant learning rate (General response, item 3).

Q2: Which properties determine whether larger relative steps help? We cannot yet predict this. Early and late windows with the same coefficient, duration, and total shrinkage have opposite final effects. At 405M, resetting norms removes most but not all of the advantage. These results show that timing and other parts of the training history matter; they do not identify whether weight directions or momentum explain the remaining gain. The norm mechanism explains persistence, not the full ordering of schedules.

Q3: Can crossover time be predicted from the state at removal? The current analysis cannot predict it. The approximate recurrence describes how relative steps recover after removal, but does not predict loss or crossover time. This remains an open question.

Thanks to the author for the rebuttal. I think this question is a very interesting starting point, and there is much more interesting exploration that can be done on this basis.

Thank you for the encouraging feedback and thoughtful questions. Please let us know if there’s anything else we can clarify. If you feel the revision addresses your concerns, we’d appreciate it if you would consider a higher rating.

Reviewer_CPFuOpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the delayed effects of temporary weight decay during language-model pretraining with Muon. The authors show that an intervention can worsen validation loss when removed but improve it much later. The paper attributes this persistence partly to reduced weight norms, which sustain larger relative update sizes after decay ends. Fixed-norm replay experiments reproduce the performance ordering and a comparable loss difference.

Critical strengths

Clear evidence of delayed effects. The 1.4B experiment provides a compelling example: temporary decay substantially worsens loss at removal, crosses the no-decay baseline only at 73% of training, and ultimately outperforms both no decay and the best sampled constant-decay baseline.


Useful experimental separation of timing and evaluation horizon. The comparisons distinguish intervention start time, duration, total shrinkage, and post-removal training. The reversal in schedule rankings under different evaluation horizons is particularly informative.

Critical weaknesses

The conceptual contribution appears incremental relative to the prior work discussed in the paper. Early regularization having lasting effects, and weight decay influencing effective step size through weight norms, are established ideas. The clearest additional finding is the delayed sign reversal and its consequences for evaluation. This is interesting, but the paper could more sharply establish what understanding it adds beyond the combination of these existing observations.


The mechanism explains persistence more convincingly than benefit. The norm analysis explains how temporary decay can continue to affect optimization after removal. However, it does not predict why the resulting trajectory improves final loss, why late interventions hurt, or which duration should be preferred. The authors acknowledge this distinction, but it remains a substantial limitation: the account describes the persistence of the intervention without explaining the performance ordering across schedules.


The strongest mechanistic evidence has limited coverage. Detailed measurements and replay experiments are concentrated at 162M. The 1.4B experiment demonstrates the phenomenon at a larger scale, but lacks the no-decay relative-step logs needed for the same mechanistic checks. Moreover, many schedule comparisons use a single seed, making the reliability of finer schedule rankings difficult to assess.

Questions for authors

Can the proposed account predict a favorable intervention duration or explain the early-versus-late sign reversal?


Would resetting weight norms to the reference values at removal substantially eliminate the delayed advantage? Such a control could complement the replay experiments by testing the role of the norm deficit within the original Muon setting.

3 comments

Thank you for your constructive review and for suggesting the norm reset.

What the paper adds. Golatkar et al. (2019) show, in image classifiers, that early regularization matters more than late regularization. Kosson et al. (2024) explain how constant decay sets norms and effective step size at equilibrium. We study temporary weight decay in Muon pretraining of language models from 162M to 3.4B parameters, and focus on what happens after decay stops. The loss difference can change sign, and the preferred schedule changes depending on when it is compared. At 405M, the 20%-start window is better at the common final update, but the initialization-start window is better after equal training following removal. Understanding how early optimization choices affect subsequent training can guide the design of pretraining schedules. We show that effects developing after decay ends can change both its benefit and the preferred schedule, and use norm-reset experiments to establish the contribution of weight norms to the final improvement.

The norm reset tests whether the norms left at removal cause the final advantage. The 3.4B experiments are new since submission, added in the General response above.

Persistence versus benefit. The recurrence explains how relative steps approach the reference values after removal; it does not predict loss. Resetting norms shows that they explain most of the measured benefit. The fixed-norm replay reproduces a comparable gain through the measured relative-step schedule, but uses a modified optimizer. The analysis does not predict which schedules help or hurt.

Coverage. We extended the norm-reset experiment to 405M. The paper also reports consistent gains from early decay across five paired comparisons at 162M and three at 405M.

Q1: Can the mechanism predict duration or the sign reversal? The current analysis cannot predict either. The compute-optimal sweeps find an interior optimum; in the 16,000-update 162M sweep, the longest tested window is best. These are experimental findings, not predictions of the recurrence.

Q2: Does resetting norms eliminate the advantage? It removes most of it. Giving the early-decay run no-decay norms removes 97% of the final advantage at 162M and 71% at 405M. Giving the no-decay run early-decay norms recovers 96% and 91%, respectively. Directions and optimizer state are retained at the reset, and both runs continue without decay. The second change initially worsens loss and only later helps. This directly tests the norm deficit in ordinary Muon. See General response, item 1, for the norm-reset setup and the loss changes immediately after rescaling.

Thank you again for suggesting the norm-reset experiment. We hope the results at 162M and 405M address your question about how much of the delayed benefit comes from the weight norms at removal. Please let us know if you have any remaining questions. If you feel we’ve addressed your concerns, we would appreciate it if you would reconsider your rating.

Thank you for the clarification and the additional experiments. To be candid, I still struggle to identify substantial novelty in the main part of the paper, given how extensively the relationship between weight decay, weight norms, and effective step size has already been discussed. I appreciate the distinction between prior work and your focus on post-removal dynamics, but I remain unconvinced that the delayed crossover and schedule-ranking reversals provide a sufficiently substantial conceptual advance.

That said, I find the norm-reset experiments interesting. The bidirectional interventions provide more direct causal evidence than the original analysis, and the extent to which a one-time norm reset removes or recovers the final advantage goes beyond what I would have expected from my existing understanding. This strengthens the empirical contribution, although it does not fully resolve my novelty concerns. I will therefore raise my score to 3.

Reviewer_uG5COpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper discusses the effect of intervention during optimization process, mainly focusing on effect of temporary weight decay in Muon pretraining. They show that a temporary weight decay can lead to improvement over the no decay baseline, and sometimes over the constant weight decay baseline. The loss improvement may not appear immediately after the intervention finish. Instead, the effect is due to the weight decay's effect on weight norm, which influences the effective learning rate.

Critical strengths
  1. This paper provides a in depth analysis on the effect of temporary weight decay, and attribute this effect to the norm dynamics.

  2. The paper is clearly written and easy to understand.

Critical weaknesses
  1. It is unclear to the reviewer why the studied problems is of great significance. For example, the authors didn't show that temporary weight decay has consistent benefit over constant weight decay or weight decay scheduling.
Questions for authors
  1. The reviewer would appreciate an explainataion on the motivation on the problem studied.
3 comments

Thank you for asking us to clarify the motivation and baseline comparisons.

Why study decay timing in Muon? Muon was introduced without weight decay [1]. Liu et al. identify adding weight decay and adjusting update scale as crucial to scaling Muon in Moonlight [2]. This raises a practical question: must decay remain active throughout pretraining, or can applying it early be sufficient?

Our paper studies when decay is applied, how long it lasts, and when its benefit is measured. In the main experiments, constant decay gives little benefit at larger scales: the best sampled constant improves on no decay by only 0.0011 nats at 1.4B, and neither tested constant helps at 3.4B. Early decay improves final loss at both.

The benefit can emerge long after decay ends. At 1.4B, the schedule selected at 162M and transferred without retuning is 0.26 nats worse than no decay at removal, crosses it only after 73% of training, and finishes 0.0264 nats below the best sampled constant. This motivates studying how temporary decay changes subsequent optimization: an initially harmful intervention can improve the outcome of much longer pretraining. Smaller weight norms keep updates larger relative to the weights after decay stops, and the norm-reset experiments show that these norm differences explain most of the final gain at 162M and 405M (General response, item 1).

Does early decay consistently beat constant decay? The first-10% windows beat both no decay and the best sampled constant in the main Muon configurations:

ScaleEarly decay minus the better reference (nats)
162M−0.0058
405M−0.0175
1.4B−0.0237
3.4B, new−0.0112; transferred schedule with 12,320 decay updates: −0.0194

The benefit also holds at 20 times the compute-optimal budget at 162M and after learning-rate tuning at 162M and 405M.

How does it compare with scheduled decay? Both new linear schedules beat the best constant at 162M and 405M; some outperform the early window (General response, item 2). We do not claim that the window is optimal. It lets us control start time and duration, while the linear schedules show that the benefit also applies to gradual decay. All comparisons use the same training budget as their references.

[1] Jordan et al. Muon: An optimizer for hidden layers in neural networks. 2024. https://kellerjordan.github.io/posts/muon/

[2] Liu et al. Muon is Scalable for LLM Training. 2025. https://arxiv.org/abs/2502.16982

We hope our response clarifies why decay timing matters and how the early window compares with constant and gradual decay. Please let us know if you have any further questions. If you feel we’ve addressed your concerns, we’d appreciate it if you would reconsider your rating.

Thank you for the detailed response and the added experiments. I support the acceptance of this paper but I will keep my score because the current decay window still feels like an interesting exploration rather than a principled algorithm to me.

Comments on the paper

1 comment

General response and additional experiments

We thank all the reviewers for their constructive feedback. We ran additional experiments to address their questions about decay scheduling, learning-rate tuning, and the role of weight norms. Losses are validation cross-entropy in nats per token; "removal" means the end of the decay window.

1. Norm reset at removal (CPFu). At the end of the first-10% window, we rescale each block matrix to the Frobenius norm of its counterpart in the other run. Matrix directions and optimizer state are preserved at the reset. Both runs continue without decay, with their original data order and learning-rate schedule.

Scale and dataNo-decay run given early-decay normsEarly-decay run given no-decay norms
162M, FineWeb-EduRecovers 96% of the gainRemoves 97%
405M, FineWeb100BRecovers 91% of the gainRemoves 71%

Percentages refer to the final early-decay advantage over no decay within each matched comparison. The norms at removal explain most of this advantage. Rescaling also changes the model's output: giving the reference run early-decay norms initially worsens loss by 0.062 nats at 162M and 0.174 at 405M. It becomes better than its unchanged copy only with further training.

2. Linear decay schedules (uG5C). We compare the early window with linear schedules reaching zero after 20% or 100% of training. The window and the 20% linear schedule start at λ=0.3; the full-training schedule starts at 0.1 for 162M and 0.04 for 405M.

ScaleNo decayBest constantFirst-10% windowLinear, first 20%Linear, all training
162M3.38693.37793.37083.36893.3668
405M3.05373.04623.02873.02573.0340

The 162M row uses reference runs separate from the paper's scale table, so its window gain over the best constant (0.0071 nats) differs from the scale-table value (0.0058). Both linear schedules beat the best sampled constant at both scales. The benefit also occurs when decay decreases gradually.

3. Learning-rate sensitivity (uG5C, xeAJ). Our original rate of 0.01 followed Dion [1]. We swept the base learning rate for no decay, constant decay, and early decay. At higher rates, we reduced λ to keep ηλ, and hence the window's total multiplicative shrinkage, unchanged.

ScaleEarly decay vs. best no decayEarly decay vs. best constant
162M−0.0090−0.0067
405M−0.0099−0.0151

Each reference is selected separately from its own tested configurations. Early decay uses 0.04 at 162M and 0.02 at 405M. With the non-matrix Adam learning rate fixed at 0.002, early decay also improves on no decay at 162M.

4. New experiments at 3.4B. After submission, we trained 3.4B models on about 69B tokens.

ScheduleFinal lossDifference from no decayMean zero-shot accuracy (%)
No decay2.4832065.30
Constant, λ=0.0012.4843+0.001165.68
Constant, λ=0.0032.4895+0.006365.69
First 10%, λ=0.22.4720−0.011266.54
First 2,800 updates, λ=0.22.4781−0.005165.96
Transferred from 162M: λ=0.15, first 44%2.4782−0.005066.85
Transferred from 162M: λ=0.15, 12,320 decay updates2.4638−0.019466.90

Neither tested constant achieves lower validation loss than no decay. Every early schedule achieves lower validation loss and higher mean zero-shot accuracy than all three references. Both transferred schedules use the coefficient selected at 162M. The 44% window gives 29,040 decay updates at 3.4B; the last row instead keeps the 12,320 updates used in the 1.4B transfer. The 2,800-update row keeps the window length of the 1.4B first-10% run. Accuracy averages the paper's five zero-shot tasks.

Items 2–4 use FineWeb100B at approximately 20 tokens per parameter. We will add the new results and full protocols to the appendix and clarify what the norm mechanism explains.

Manuscript updates. The revised paper includes the experimental protocols and results for norm resets (Appendix H), gradual decay schedules (Appendix I), learning-rate sensitivity (Appendix J), and 3.4B experiments (Appendix K). Section 5.3 summarizes the norm-reset finding, and Section 6.2 and Table 1 include the 3.4B comparisons. Blue text indicates additions in the revised PDF.

[1] Ahn et al. Dion: Distributed Orthonormalized Updates. 2025, Appendices E.1–E.2. https://arxiv.org/abs/2504.05295

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.