- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes to use a weight-decay schedule in proportion to (where is the learning-rate), instead of in proportion to , when applying the Muon optimizer to pre-training Transformer LMs. The theoretical rationale for this design choice, is based on the observation that when the learning-rate satisfies the Robbins–Monro conditions for SGD (which guarantees convergence for convex problems): a weight-decay schedule in proportion to will introduce non-zero bias to a quadratic loss, while a weight-decay schedule in proportion to does not.
The paper also proves an analogy of this observation for the Muon optimizer. Then, it presents empirical results of training small scale Transformer MoE LMs, with Muon optimizer and the proposed weight-decay schedule applied. The results are promising (approximately 30% fewer optimizer steps) compared to the baseline.
Critical strengths
I actually like the work and I believe the findings. But the truth is, some of the widely adopted hyperparameters for open-sourced Transformer LM baselines, such as the weight decay coefficient of 0.1, are just sub-optimal. It leads to very similar findings, even with the number of "approximately 30% fewer optimizer steps", get rediscovered and reported repeatedly. One previous paper (very similar, but not with the exact same settings) I'm aware of: https://arxiv.org/abs/2210.11693
So if Generative AI was involved in the idea of this work, it is a bit of a problem if such previous works are not properly cited.
Critical weaknesses
The biggest problem is with theoretical justification.
-
In Introduction (Line 89) the paper says: "By plugging in in (1) we observe that if training converges, it does so at a point where ", and this argument has been repeatedly used afterwards -- This is a mistake. We never assume a training convergence to be this strong. In other words, even under a convex loss function, a "convergence" would only mean that gets closer and closer to a fixed point at , but exact equivalence may never be achieved when .
-
All the theory in this paper is saying that, "weight-decay schedule in proportion to will introduce non-zero bias", while "weight-decay schedule in proportion to " does not, so the latter should be better. However, this argument does not logically hold. Introducing bias through weight-decay, or L2 regularization, is not necessarily a bad thing; it represents the bias-variance trade-off, and for certain tasks, such the ImageNet image classification (which the AdamW paper Loshchilov and Hutter 2019 is originally based upon), the type of "strong weight-decay" is definitely necessary because the dataset is noisy. For training LMs, indeed, "the primary benefits of weight decay is stabilizing the training process by modifying the dynamics of training and reducing the final loss", but does your theory explain why weight-decay is necessary at all? If "not introducing a bias" is the goal, why not completely get rid of weight-decay in the first place? Right, we cannot do that empirically because without weight-decay, the norm of model parameters will quickly blow up and the training will not go well. But does your theory explain why this happens?
Theoretically, we actually know that the blow-up of model parameter norms is a phenomenon specific to the training of deep neural networks -- one cannot explain it with convex quadratic models. We also know that this phenomenon is probably related to the stochastic noise of gradients, and a weight-decay schedule in proportion to may somehow be "explained" from this perspective, at least for Adam-like optimizers. Although not perfect, the paper https://arxiv.org/abs/2210.11693 provides such a theory.
-
The Robbins–Monro conditions do not exactly match the cosine learning-rate schedule that we use in practice. Indeed, the same theory that derives Robbins–Monro conditions (yes, I like that theory) also derives that an "asymptotically optimal" learning-rate schedule should decay in proportion to , which is not what a cosine schedule looks like. Recent theory regarding SGD, however, has brought much more powerful tools that can accurately predict the exact dynamics of validation loss under more general settings (see https://arxiv.org/abs/2102.04396), and in Appendix F.2 of https://arxiv.org/abs/2505.16098 the problem of SGD with learning-rate schedule has been discussed in detail. Unfortunately, a theory regarding weight-decay and weight-decay schedules remains a largely open problem, because the phenomenon cannot be fully captured by quadratic models.
-
Empirically, as I said at the beginning, one can usually improve some widely adopted open-sourced Transformer LM baselines by tweaking weight-decay, because the default hyper-parameter is not good. My experience is that, when thoroughly tuned, a weight-decay schedule in proportion to can still perform well (maybe 5~10% behind the schedule in proportion to , if that is thoroughly tuned as well; but certainly not 30% behind). It would be amazing if the authors could come up with more experimental results that defy my experience.
Questions for authors
Have you tried jointly tune the weight-decay strength and learning-rate of the baseline?
Thanks for the response
The authors have clarified the scope of this work, and I believe we have sorted out the theoretical common grounds regarding the topic of weight decay. I would like to retain my assessment, given the clear limitation of the current results in the paper; but it was good to read the draft and the discussion.