acceptodds
Under review as a conference paper at ICLR 2027

Scale Weight Decay and Train Better

Abstract

The discovery of scaling laws has motivated training neural networks on ever increasing quantities of data. This is typically done with a constant decoupled weight decay which causes the network weights to shrink steadily over the course of training. Taking inspiration from the Robbins–Monro conditions, we propose to scale weight decay by the fraction of the peak learning rate . We prove that this scaled weight decay preserves the asymptotic stationarity guarantees of the corresponding unregularized methods for both stochastic gradient descent and the non-Euclidean spectral optimizer Muon, thereby avoiding the additional asymptotic bias introduced by constant decoupled weight decay. This retains the stability benefits of weight decay without changing the asymptotic optimization target. Using a steady-state analysis, we explain why under standard weight decay the weight norm shrinks steadily as training proceeds, whereas under scaled weight decay it settles to a roughly constant value. When applied to the training of mixture-of-experts models, Muon with scaled weight decay (Muon-SW) consistently outpaces Muon with identical hyperparameters, reaching the same validation loss faster at our largest scale across models from million parameters trained at tokens per active parameter. If this trend continues to hold, the method promises to substantially accelerate the pre-training of frontier models while requiring only a few lines of code to implement.

Primary area:
optimization
Keywords:
Weight Decay, Muon, Pre-training, Mixture of Experts

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Fu4JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes to use a weight-decay schedule in proportion to ηt2\eta_t^2 (where ηt\eta_t is the learning-rate), instead of in proportion to ηt\eta_t, when applying the Muon optimizer to pre-training Transformer LMs. The theoretical rationale for this design choice, is based on the observation that when the learning-rate ηt\eta_t satisfies the Robbins–Monro conditions for SGD (which guarantees convergence for convex problems): ∑tηt=∞\mboxand∑tηt2<∞,\sum_t \eta_t=\infty \quad\mbox{and}\quad \sum_t \eta_t^2<\infty, a weight-decay schedule in proportion to ηt\eta_t will introduce non-zero bias to a quadratic loss, while a weight-decay schedule in proportion to ηt2\eta_t^2 does not.

The paper also proves an analogy of this observation for the Muon optimizer. Then, it presents empirical results of training small scale Transformer MoE LMs, with Muon optimizer and the proposed weight-decay schedule applied. The results are promising (approximately 30% fewer optimizer steps) compared to the baseline.

Critical strengths

I actually like the work and I believe the findings. But the truth is, some of the widely adopted hyperparameters for open-sourced Transformer LM baselines, such as the weight decay coefficient of 0.1, are just sub-optimal. It leads to very similar findings, even with the number of "approximately 30% fewer optimizer steps", get rediscovered and reported repeatedly. One previous paper (very similar, but not with the exact same settings) I'm aware of: https://arxiv.org/abs/2210.11693

So if Generative AI was involved in the idea of this work, it is a bit of a problem if such previous works are not properly cited.

Critical weaknesses

The biggest problem is with theoretical justification.

  1. In Introduction (Line 89) the paper says: "By plugging in Wt+1=WtW_{t+1} = W_t in (1) we observe that if training converges, it does so at a point where Wt=−Ut/λW_t = −U_t/\lambda", and this argument has been repeatedly used afterwards -- This is a mistake. We never assume a training convergence to be this strong. In other words, even under a convex loss function, a "convergence" would only mean that WtW_t gets closer and closer to a fixed point at t→∞t\to\infty, but exact equivalence Wt+1=WtW_{t+1} = W_t may never be achieved when ηt≠0\eta_t\neq 0.

  2. All the theory in this paper is saying that, "weight-decay schedule in proportion to ηt\eta_t will introduce non-zero bias", while "weight-decay schedule in proportion to ηt2\eta_t^2" does not, so the latter should be better. However, this argument does not logically hold. Introducing bias through weight-decay, or L2 regularization, is not necessarily a bad thing; it represents the bias-variance trade-off, and for certain tasks, such the ImageNet image classification (which the AdamW paper Loshchilov and Hutter 2019 is originally based upon), the type of "strong weight-decay" is definitely necessary because the dataset is noisy. For training LMs, indeed, "the primary benefits of weight decay is stabilizing the training process by modifying the dynamics of training and reducing the final loss", but does your theory explain why weight-decay is necessary at all? If "not introducing a bias" is the goal, why not completely get rid of weight-decay in the first place? Right, we cannot do that empirically because without weight-decay, the norm of model parameters will quickly blow up and the training will not go well. But does your theory explain why this happens?

Theoretically, we actually know that the blow-up of model parameter norms is a phenomenon specific to the training of deep neural networks -- one cannot explain it with convex quadratic models. We also know that this phenomenon is probably related to the stochastic noise of gradients, and a weight-decay schedule in proportion to ηt2\eta_t^2 may somehow be "explained" from this perspective, at least for Adam-like optimizers. Although not perfect, the paper https://arxiv.org/abs/2210.11693 provides such a theory.

  1. The Robbins–Monro conditions do not exactly match the cosine learning-rate schedule that we use in practice. Indeed, the same theory that derives Robbins–Monro conditions (yes, I like that theory) also derives that an "asymptotically optimal" learning-rate schedule should decay in proportion to 1/t1/t, which is not what a cosine schedule looks like. Recent theory regarding SGD, however, has brought much more powerful tools that can accurately predict the exact dynamics of validation loss under more general settings (see https://arxiv.org/abs/2102.04396), and in Appendix F.2 of https://arxiv.org/abs/2505.16098 the problem of SGD with learning-rate schedule has been discussed in detail. Unfortunately, a theory regarding weight-decay and weight-decay schedules remains a largely open problem, because the phenomenon cannot be fully captured by quadratic models.

  2. Empirically, as I said at the beginning, one can usually improve some widely adopted open-sourced Transformer LM baselines by tweaking weight-decay, because the default hyper-parameter is not good. My experience is that, when thoroughly tuned, a weight-decay schedule in proportion to ηt\eta_t can still perform well (maybe 5~10% behind the schedule in proportion to ηt2\eta_t^2, if that is thoroughly tuned as well; but certainly not 30% behind). It would be amazing if the authors could come up with more experimental results that defy my experience.

Questions for authors

Have you tried jointly tune the weight-decay strength and learning-rate of the baseline?

1 comment
Reviewer_Fu4JOpenReview ↗

Thanks for the response

The authors have clarified the scope of this work, and I believe we have sorted out the theoretical common grounds regarding the topic of weight decay. I would like to retain my assessment, given the clear limitation of the current results in the paper; but it was good to read the draft and the discussion.

Reviewer_5YoYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies scaling the weight-decay coefficient by the current learning rate relative to its peak, and applies this rule to Muon, providing guarantees of convergence under stated assumptions. Also providing an analysis of weight norms. Experiments on mixture of expert models show better validation loss across four widths, with around 30% fewer steps to reach the baseline loss at the largest scale

Critical strengths
  1. The method is fairly simple to implement and has close to no overhead, which makes it very practical if the gains are true
  2. The claimed improvements seem across the four widths. The weight-norm and update-alignment diagnostics are also useful to understanding.
Critical weaknesses
  1. The baseline comparison is not complete. Both methods use lambda=0.1, but scaled decay applies less shrinkage through most of the training and there is no comparisons against a tuned constant decay, zero decay, or matching the cumulative decay. Also, shrinking weights changes the relative update size and can counteract learning-rate annealing. This needs clear ablations
  2. Its not clear how theory justifies the improvement. It assumes bounded iterates, learning rates tending to zero under Robbins–Monro conditions, and exact polar factors (for muon), but the experiment uses finite cosine schedules ending at a nonzero learning rate and approximating Newton–Schulz updates, so preserving an asymptotic stationarity guarantee under these assumptions does not tell about faster training or stability in the said setup
  3. The learning-rate conventions are unclear (look questions)
  4. Explanation of the weight norms does depend on fitting the observed alignment against said learning rate, so the analysis does not establish why maintaining the norm improves validation loss. Also since the decay rule already appears in Defazio (2025), as acknowledged by authors, the additional Muon-specific explanation needs support
  5. Only one architecture family and dataset is covered, with no reported seed variation. Since, width and training duration increase together, so the trend with model scale is difficult to isolate.
Questions for authors
  1. See W3 above, the authors need to clarify which learning rate enters the update, decay and alignment fit.
  2. Do gains remain after tuning LR and decay for both methods?
  3. Can an ablation distinguish benefit of mantaining weight norms from the benefit of changing effective LR schedule?
Reviewer_GwmVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies learning-rate-dependent weight decay for SGD and Muon. It replaces the standard decoupled shrinkage factor ληt\lambda\eta_t with ληt2/ηmax⁡\lambda\eta_t^2/\eta_{\max}. The authors prove weighted stationarity guarantees for SGD and exact-polar Muon under Robbins–Monro learning-rate conditions, and construct quadratic examples where constant weight decay prevents convergence to the unregularized minimizer. A steady-state approximation is used to analyze weight norms. Experiments on mixture-of-experts language models report lower validation losses and fewer training steps than standard Muon with the same base weight-decay coefficient.

Critical strengths

Theorems 3.1 and 3.2 establish asymptotic stationarity bounds under specified smoothness, boundedness, and learning-rate assumptions. The experiments compare standard Muon and Muon-SW across four model widths under a shared training configuration.

Critical weaknesses

The central argument relies on the summability of the scaled shrinkage term under the Robbins–Monro conditions. Theorem 3.1 establishes an asymptotic weighted stationarity result for SGD, assuming bounded iterates. Theorem 3.2 gives a corresponding nuclear-norm stationarity bound for exact-polar Muon, with a nonzero noise floor at fixed batch size. Neither theorem supplies a finite-time convergence-rate improvement over a suitably tuned constant-decay optimizer. There is also a direct mismatch between the analyzed and implemented algorithms. The Muon proof assumes exact polar factors, whereas the experiments use five Newton–Schulz iterations and Nesterov momentum. Although the appendix addresses the Nesterov modification, it does not establish the same guarantee for finite-step orthogonalization. The experimental learning-rate schedule also terminates at one tenth of its peak value, rather than satisfying the infinite-horizon Robbins–Monro conditions. The convergence theorems therefore do not establish the reported training-speed advantage. The key step in the convergence proofs is that the additional shrinkage contributes a summable perturbation to the descent inequality. This argument is not specific to the quadratic dependence on the learning rate. Other decay schedules satisfying an appropriate summability condition can retain the same asymptotic stationarity property. Moreover, the paper acknowledges that Defazio (2025) introduced the same decay rule. The theoretical results provide an additional justification for that rule, but do not establish why its particular functional form should be preferred over alternative vanishing-decay schedules. The experiments hold the base coefficient λ=0.1\lambda=0.1 fixed for both methods without establishing performance against independently tuned constant-decay baselines. Thus the reported improvement does not isolate the benefit of the proposed schedule from the effect of reducing the effective regularization strength during training.

Questions for authors

Can the authors establish a finite-time result distinguishing the proposed ηt2\eta_t^2 shrinkage rule from other vanishing weight-decay schedules? The existing proofs appear to depend primarily on summability rather than the specific quadratic scaling. What mathematical property makes this choice preferable? How do the convergence guarantees extend to the five-step Newton–Schulz approximation used in the experiments? In particular, what control on the approximate polar direction is sufficient to preserve the descent and stationarity bounds? Does Muon-SW retain its reported training advantage when standard Muon is independently tuned over the weight-decay coefficient? A comparison matching the cumulative shrinkage across the training schedule would help determine whether the observed improvement comes from the schedule itself or from applying less regularization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.