acceptodds
Under review as a conference paper at ICLR 2027

Correction of Decoupled Weight Decay: Disentangle effective learning rate and weight norm control

Abstract

Since AdamW, decoupled weight decay has replaced regularization as the standard weight control method. More recently, steady-state analysis has suggested that decoupled weight decay should be proportional to learning rate squared to maintain stable weight norm, but such analysis can be derived from either orthogonality or independence of the weight and the gradient. We show that when applied to Muon / Scion, weight-gradient independence implies that the effective learning rate is momentum-dependent, a result that would be hard to explain with weight-gradient orthogonality alone. We show that optimal effective learning rate transfers across momentum values and enables log-time momentum scheduling for improved LLM training. Furthermore, models exhibit aggregated near-scale-invariance which explains the performance difference and altered model training dynamics due to weight decay correction. We conclude with both practical recommendations and implications of our results.

Primary area:
optimization
Keywords:
optimizer, muon, scion, ViT, NanoGPT

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QQiwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the interaction between decoupled weight decay, momentum, and effective learning rate. It derives an approximate momentum-dependent learning rate for normalized updates and proposes ScionC to separate learning-rate scheduling from weight-norm control. Experiments on ViT-S/16 and NanoGPT examine learning-rate transfer across momentum values, approximate scale invariance, and momentum scheduling, with Scion as the main performance baseline.

Critical strengths

The momentum sweeps directly test the proposed parameterization. The optimal effective learning rate varies much less than the raw learning rate across momentum settings. This provides useful evidence for simplifying hyperparameter tuning within the evaluated setups.

Critical weaknesses
  1. The theoretical contribution is limited and insufficiently distinguished from existing work. The central derivation follows from a steady-state second moment calculation under independent gradient and approximately constant momentum norm assumptions. This is a straightforward extension of existing equilibrium analyses. Section 4 acknowledges that Wen et al. (2026) obtain the same equilibrium norm as Eq. 4 for L2 normalized updates, while Liu et al. (2026) report closely related effective-learning-rate collapse. These connections deserve substantially more discussion to establish what new insight this paper contributes. The main text also spends too much space restating AdamW and Scion update rules and expanding elementary derivations. Much of this material should move to the appendix, leaving the main paper to explain the new conclusions, their assumptions, and their empirical support.

  2. The empirical evaluation is too narrow to establish broader practical value. The experiments are limited to ViT-S/16 and a single 124M parameter language model, with no evidence at larger model scales. The main performance comparison in Table 2 includes only ScionC and Scion. Particularly relevant missing comparisons are Muon Hyperball (MuonH) and Adam Hyperball (AdamH) from Wen et al. (2026), which explicitly control weight and update norms and are evaluated up to 1.2B parameters. SOAP is another relevant modern optimizer for broader performance comparisons. Despite the substantial Adam family discussion, the AdamW in Figure 1 and AdamC beta comparison in Appendix E do not replace systematic AdamW/AdamC performance comparisons under matched training and tuning budgets. Consequently, the experiments leave unclear how broadly the proposed correction helps beyond the particular Scion configurations studied.

  3. The paper appears to have been completed in a rush. Figure 11 appears to consist of wandb screenshots, with panel borders and raw metric names such as val/acc@1, l2_params, and l2_grads with small faint labels. There are also inconsistencies beyond figure styling: Figure 1's caption describes gradient-norm results, but the third panel actually plots learning rate.

Questions for authors

See critical weaknesses.

2 comments

Distinction from Existing Work

It is already stated in Related Works 3 that we derive γeff\gamma_\mathrm{eff} from the approximation of the equilibrium norm while Wen et al. (2026) does not. Furthermore, to our best knowledge ours is the only work on the scale invariance of optimizers with corrected WD. If this is "straightforward" then it is worthwhile to point out that the community has missed this critical dependency of Scion / Muon's γeff\gamma_\mathrm{eff} on momentum for the last $\approx$2 years. Other distinctions not stated for space and brevity reasons include the case with Nesterov momentum, differences between the training dynamics of input / hidden layers vs. output layer, aggregated near-scale-invariance (∥Jo∥F‾≈c∥w∥2\overline{\left\Vert \mathbf{J_o} \right\Vert_\mathrm{F}} \approx \frac{c}{\left\Vert \mathbf{w} \right\Vert_2}) as a weaker yet verifiable condition when the model lacks perfect scale invariance, etc.

We have removed repetitions and shortened derivations so the rebuttal revision has more than 1 page of new content, but it is common to state the update rules and keep the derivation in the main text at least once (e.g. Kosson et al. (2024) and Wen et al. (2026)) and in our opinion it helps notation consistency and makes it easy to follow. For more repetitive and tedious derivations with momentum variants we have already relegated them to Appx. A.

Experiment Scale and Applicability

Besides FAQ 1: We choose the optimizers to experiment with in order to answer the specific questions:

  1. How do momentum, LR, and weight norm control interact?
  2. How can we disentangle them?

This is the reason why we run both regular and Nesterov momentum while none of the concurrent hyperball counterparts does. Wen et al. (2026) actually includes a couple of 8B parameter experiments (Fig. 3MR) but the size of the biggest model is not the only measurement of scale. E.g. we perform the momentum and scale sweeps of Modded-NanoGPT with more compute and near the maximum parameter counts of their depth and width scaling (Fig. 5 and 6) so in fact we are more confident that the results hold at relevant scales. This is also apparent from comparing the validation losses reached in our experiments.

We include substantial Adam family discussion because WD correction was first purposed for AdamC and misattributed RMS-matching factors have been empirically found to allow Muon to reuse the optimal LR of AdamW. Nonetheless, we have added AdamW / AdamC comparison as Appx. D.

Caption Error and wandb Screenshot

Both are fixed.

Thank you for the response and revision. The presentation issues are now resolved, and the new AdamW/AdamC appendix and weight norm analyses add useful context. However, my main concerns remain. The revision adds no further distinction from Wen et al. (2026), whose equilibrium norm the paper describes as identical to Eq. 4, which already includes momentum. The experiment scales are still limited, with no hyperball comparisons, and the AdamW/AdamC baselines are not tuned under a budget matched to ScionC.

Reviewer_2eQcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the relationship between decoupled weight decay, effective learning rate, and weight norm control in normalized-update optimizers, particularly Muon and Scion. The authors derive a momentum-dependent effective learning rate under assumptions of temporally independent minibatch gradients and approximate steady-state behavior. Based on this analysis, they propose ScionC, an optimizer parameterization designed to disentangle effective learning rate from weight norm control. The paper also introduces an aggregated near-scale-invariance hypothesis to explain training dynamics and explores its implications for momentum scheduling. The proposed approach is evaluated using numerical simulations, ViT-S/16 on ImageNet-1k, and a 124M-parameter Modded-NanoGPT model trained on FineWeb-Edu. The authors report that the optimal effective learning rate transfers across a wide range of momentum values more consistently than the nominal learning rate. They also investigate log-time momentum scheduling and report improved validation loss relative to the tested constant-momentum configurations in their language-model experiments. The paper provides a useful theoretical perspective on normalized-update optimizers and offers a practical parameterization for studying learning-rate and weight-norm effects separately. However, the theoretical conclusions rely on simplifying assumptions whose applicability to realistic training dynamics remains incompletely established. Furthermore, the empirical evaluation is relatively narrow, and the practical advantages of ScionC are modest or inconsistent across some settings. Additional validation would strengthen the claims regarding the generality and practical significance of the proposed approach.

Critical strengths

Strength 1: A useful theoretical characterization of momentum-dependent effective learning rates. The paper derives an explicit relationship between momentum and the effective learning rate for normalized-update optimizers. The reported experiments suggest that the optimal effective learning rate transfers within approximately a factor of two across a 50-fold variation in momentum, whereas the optimal nominal learning rate varies by nearly an order of magnitude. This provides a potentially useful explanation of optimizer behavior and could help make hyperparameter selection more systematic.

Strength 2: A practical optimizer parameterization supported by experiments in different domains. The proposed ScionC formulation separates effective learning rate and weight norm control, allowing these factors to be studied more independently. The evaluation includes both image classification and language-model training, while the momentum-scheduling experiments suggest a possible practical benefit for language-model optimization. Although the reported gains are not uniformly substantial, the combination of theoretical analysis and empirical investigation gives the work practical relevance.

Critical weaknesses

Weakness 1: The theoretical predictions depend on assumptions whose applicability to realistic training remains uncertain. The analysis relies on temporally independent minibatch gradients, approximately constant gradient statistics, and steady-state behavior. In practical training, gradient correlations and training statistics can change over time, particularly under learning-rate and momentum schedules. The paper provides numerical support for some predictions, but does not sufficiently establish how sensitive the derived effective learning rate and weight norm relationships are to violations of these assumptions. This limits confidence in the generality of the theoretical conclusions.

Weakness 2: The empirical evidence does not yet establish a robust practical advantage. The experiments cover a limited range of architectures and model scales. In several ViT comparisons, the differences between ScionC and Scion are very small, and the relative advantage varies across training budgets. The main Modded-NanoGPT experiments are also reported without repeated runs, making it difficult to assess the stability of the observed validation-loss differences. More comprehensive and reproducible comparisons are needed to determine whether the proposed parameterization and momentum schedules consistently improve optimization under comparable tuning budgets.

Questions for authors

Question 1 — Clarification of theoretical validity: How sensitive are the proposed effective-learning-rate and steady-state weight-norm predictions to temporal correlations in minibatch gradients and departures from steady state? Can the authors provide quantitative evidence comparing the theoretical predictions with measured training dynamics under learning-rate and momentum schedules? The answer would help determine the scope of validity of the central theoretical claims.

Question 2 — Experimental robustness: Can the authors report repeated runs for the main Modded-NanoGPT experiments and provide variability estimates for the final validation loss? Given the relatively small differences between ScionC and Scion in some settings, such evidence would materially affect the assessment of whether the reported practical improvements are robust.

4 comments

We are glad that reviewer 2eQc learned something useful from our work. Below are our responses to the questions.

How Valid is the Analysis under Realistic Conditions?

We have compared per-layer weight norms in our experiments to the predictions as a more stringent test to our analysis (Fig. 10) and found reasonable agreement. It is also common to assume slow-changing gradient statistics in the analysis of Adam(W) optimizer (Orvieto & Gower (2025), Balles & Hennig (2018)) and find empirical agreement. We agree that there are limitations: E.g. Fig. 10 also shows some unexplained variation of the per-layer weight norms and the approximation seems to break down when α≥0.2\alpha \ge 0.2 (Fig. 2). We intend to leave further investigations to future works.

Can We Provide Variability Estimates for the Final Validation Loss of the Main Modded-NanoGPT Experiments?

In this case we actually have a good upper bound of the variability. If we consider the scale sweep of the Modded-NanoGPT experiments (Fig. 3R) N=5N=5 experiments for 5 different values of γrel\gamma_\mathrm{rel}, here is their means and sample standard deviations of the final val. loss:

12γrelo\frac{1}{2} {\gamma_\mathrm{rel}}_\mathrm{o}12γrelo\frac{1}{\sqrt{2}} {\gamma_\mathrm{rel}}_\mathrm{o}γrelo{\gamma_\mathrm{rel}}_\mathrm{o}2γrelo\sqrt{2} {\gamma_\mathrm{rel}}_\mathrm{o}2γrelo2 {\gamma_\mathrm{rel}}_\mathrm{o}
2.8126±0.00122.8126 \pm 0.00122.8020±0.00162.8020 \pm 0.00162.7964±0.00082.7964 \pm 0.00082.8001±0.00112.8001 \pm 0.00112.8154±0.00472.8154 \pm 0.0047

Other than one outlier at the highest LR and largest weight norm, the standard deviation is <0.002< 0.002. Such a small variance would leave the results of Fig. 2 (middle and bottom rows), Table 2R, Fig. 5R, and Fig. 7 qualitatively intact. Exact repetitions without changing the weight norm can only have even smaller variance.

Thank you for the additional experiments. The five-run results address my concern about the stability of the Modded-NanoGPT experiments, and the per-layer weight-norm comparison provides useful support for the theoretical analysis.

However, the effect of temporal correlations in minibatch gradients on the theoretical predictions remains unclear. The observed breakdown for α ≥ 0.2 also suggests that the approximation has a limited validity range. Clarifying these limitations and the conditions under which the theoretical predictions hold would strengthen the paper.

Approximation Breakdown When α≥0.2\alpha \ge 0.2

Our submission has mentioned (L311) but really does not have the space to expand on this. Here are our observations and speculations:

  1. The optimum is α≤0.1\alpha \le 0.1 for this ViT-S/16 setup so the breakdown is of less practical concern.
  2. One possibility is that E[⟨gt′,gt⟩]\mathbb{E}[\langle \mathbf{g_{t'}},\mathbf{g_t}\rangle] is not quite a Kronecker delta function C′2δt′t{C'}^2\delta_{t't}. I.e. the autocorrelation time is not exactly zero but ≤\le 2-5 timesteps. Such a short-term correlation would have little effect if α≪\alpha \ll autocorrelation time$^{-1}$ but results in the deviation we observe. This is also consistent with the slight decrease (factor of 12\frac{1}{\sqrt{2}}) of the optimal γeff\gamma_{\mathrm{eff}}.
  3. One interesting future direction would be to quantify this autocorrelation time and compare the corrected predictions of the weight norms and optimal γeff\gamma_{\mathrm{eff}} empirically.

Thank you for following up. It helps to know that the best ViT-S/16 settings fall below the range where the approximation breaks down. Short-term gradient autocorrelation is a plausible explanation for the discrepancy. I’ll keep my original rating.

Reviewer_UZDwOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies decoupled weight decay and points out there are two different reasons the cross-term in the l^2 norm squared of the layer weight vectors could be zero:

  1. Independence: the weights and updates are independent, so the weights perform a random walk (Kosson et al 2024 perspective)
  2. Orthogonality: normalization layers make the update orthogonal to the weights (Defazio 2025 AdamC perspective)

Under both mechanisms, the conclusion about the weight decay scaling with learning rate squared (the AdamC rule) is the same. To distinguish which mechanism is actually occurring, this paper uses a renormalized AdamW experiment that only lets the component of each update along the weights change the weight norm. If orthogonality were the primary mechanism, this should significantly change the training dynamics. In a ViT, the accuracy, weight norm and gradient norm trajectories are almost the same. This implies the random walk perspective is the relevant mechanism.

For optimizers like Muon and Scion that rescale each update to a fixed norm, momentum controls how much successive updates are correlated. For these fixed norm optimizers, longer momentum implies a larger effective step size. Optimizers like Adam don't have this dependence between momentum and effective step size. The paper proposes ScionC, which shows mild empirical gains over Scion on NanoGPT / ViT experiments.

They show that effective learning rate transfers across momentum but raw LR does not in both ViT-S/16 on ImageNet and 124M Modded-NanoGPT. Log-time momentum with fixed effective LR has the best NanoGPT loss. This hyperparameter transfer of effective learning rate also supports the random walk perspective.

They reinterpret the 0.2 RMS-matching idea that is used in practice as a correction for effective LR. They recommend parameterizing optimizers by effective LR, weight norm and momentum separately, and using constant momentum on the readout layer.

Critical strengths
  1. The overall framing of the paper is nice: there are two different mechanisms that both lead to the same weight decay scaling rule, and there are testable predictions to distinguish the mechanisms.
  2. The effective learning rate transfer result across momentum is compelling, since the momentum range is wide (50x) and the effective learning rate stays in a very narrow range (sqrt(2) factor) while raw learning rate changes substantially (10x).
  3. The connection to the RMS-matching idea gives a principled understanding of a fairly widespread empirical practice that has seemed like a hack.
  4. The observation that momentum schedule comparisons are confounded unless you also hold effective learning rate fixed is a good point.
Critical weaknesses
  1. I found the jump from the introduction section to talking about Scion confusing. I think the principle that learning rate, weight norm control, and momentum should be disentangled has some merit, but I wonder if this principle could be presented and investigated more generally.
  2. The experiments are small scale and the empirical differences are quite small.

Minor comments: The intro says "For adaptive gradient methods such as SGD with momentum" but I wouldn't consider SGD+momentum an adaptive gradient method.

Questions for authors
  1. I think this paper might be a stronger and more general contribution if it didn't focus on the Scion setting. Could you explain why you focused on Scion? Could you restate your results for a general optimizer that rescales its momentum to a fixed norm?
  2. Have you tried the effective LR transfer experiment on Muon?
3 comments

We are really glad that reviewer UZDw gained a substantially deeper understanding from our work and found our reinterpretation of the RMS-matching factors principled. We have corrected the record on SGD+momentum and answer the remaining questions below.

Can We Restate Our Results for a General Optimizer That Normalizes its Momentum?

We have reframed our results with this suggestion. In fact, Muon is merely a special case of Scion and the lmo (linear minimization oracle) in Scion / ScionC can be any normalization in principle since all norms have a dual norm. It is just that not all of them would work well as an optimizer.

Have We Tried the Effective LR Transfer Experiment on Muon?

Scion with Spectral lmo for the hidden layers is equivalent to Muon (to be exact, the adjust_lr_fn="spectral_unclamped" variant with doutdin\sqrt{\frac{d_\mathrm{out}}{d_\mathrm{in}}} LR scaling factor). In this sense we already have. The common practice for Muon is to use an auxiliary AdamW optimizer for the other parameters so the only difference for our ViT w/o bias and Modded-NanoGPT experiments is that we use Sign lmo (equivalent to SIGNUM, Bernstein et al. (2018)) for the output layer.

Dear reviewer UZDw:

We have reframed our results as general for pre-momentum optimizers and addressed questions from other reviewers and ourselves in the rebuttal revision. As we head into the last day of the discussion period we would truly appreciate if you could take another look!

Thank you for the clarifications and revision to the general optimizer with normalized momentum. I will keep my existing score.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Rebuttal Revision

We thank reviewers QQiw, 2eQc, and UZDw for their thoughtful feedbacks and have revised our submission in response. The main changes are:

  1. We noticed that the performance of ViT-S/16 stagnates with training budget stretched to 300ep and have retuned it. Based on the result we answered our own question of how the optimal LR schedule changes as we vary the number of epochs and related it to the literature (Table 2, Fig. 5M, Fig. 6, Implication 3, Recommendation 3).
  2. We have moved the RowNorm momentum sweep from the appendix to the main text for now (Fig. 2 bottom).
  3. We have compared the weight norms of individual layers to our predictions (Fig. 10).
  4. We have added the null results of training ViT-S/16 with AdamW and AdamC as Appx. D.
  5. We have expanded Related Works to include AdamP (Heo et al., 2021) and more concurrent hyperball optimizer works (Yuan et al., 2026; Yu et al., 2026).
  6. We have fixed and improved some captions and figures.

The text changes are highlighted in blue. We will respond to reviewers QQiw, 2eQc, and UZDw individually in short order.

AuthorsOpenReview ↗

FAQ 1: Experiment Scale and Applicability

A couple of questions are asked by multiple reviewers so we answer them here in a short FAQ.

Question regarding the scale of the experiments and the applicability of the results:

We train a 22M ViT-S/16 (w/ and w/o bias) and a 124M Modded-NanoGPT (weight-tied and -untied) with Scion and

  1. Both Spectral (Muon) and RowNorm (MOGA w/ row, p = 2) lmos for the hidden layers
  2. Both regular and Nesterov momentum.

While the scale is limited, it is not unusual in the area of optimization: Speed-running this exact 124M Modded-NanoGPT on 8$\times$H100 was the cradle of Muon and the Muon speed-running only trains for $\approx$5000 steps with 512$\times$1024 batch size. Each run of our 310\frac{3}{10} FineWeb-Edu-100BT experiment is 30250 steps with 1024$\times$1024 batch size so about 12$\times$ the compute. Each run of the full FineWeb-Edu-100BT experiment is about 40$\times$ the compute and such token / compute budget is necessary to demonstrate the advantage of log-time momentum scheduling. The gain of momentum scheduling with just 30B tokens is negligible. Furthermore, the idea of effective learning rate γeff\gamma_\mathrm{eff} is already supported at scales beyond in the form of misattributed RMS-matching factors as stated in Implication 1. Between our scale and that of Moonlight-2.4B-5.7T (Liu et al., 2025), DeepSeek V4 (DeepSeek- AI et al., 2026b), and V4.1 Flash (DeepSeek- AI et al., 2026a), all 3 stating that they can reuse the AdamW LR with their matching factors, it would be very unexpected if the result were somehow not widely applicable.

AuthorsOpenReview ↗

FAQ 2: Small / Inconsistent Performance Differences between Scion and ScionC

Question regarding the performance differences between Scion and ScionC:

With constant momentum α\alpha, strong scale invariance, and optimally-tuned LR schedules this is what we expected and set out to show, with two caveats regarding the ease of tuning:

  1. Knowledge is power. We suspect that the momentum of Scion / Muon has been under-tuned since it is tempting to conclude that both LR and momentum are optimal after seeing the model get worse after changing them separately (e.g. Islamov et al., 2026, Fig. 6 and 7) while we control for γeff\gamma_\mathrm{eff} when we run our MomentumAutoTuner for both Scion and ScionC. In this sense we already backported a feature of ScionC.
  2. Due to the weight norm's dependence on the LR, it is very hard to graft the optimal γrel\gamma_\mathrm{rel} to Scion / Muon with a preset LR schedule while the reverse is trivial (Fig. 5). Inverse HyperTransfer (Yuan et al., 2026) resorts to measuring weight norm and update norm at every step in order to achieve that, but in our opinion maybe we should just run hyperball or corrected WD at this point.

Finally, momentum scheduling while controlling for γeff\gamma_\mathrm{eff} is simply impossible with Scion. You need to hardcode the conversion between the raw LR and γeff\gamma_\mathrm{eff} in order to do that.

Area_Chair_m3mUOpenReview ↗

Reminder

This is a friendly reminder to the reviewers that we are halfway through the discussion period. The authors have posted responses to the reviews, so please acknowledge the rebuttal, and discuss their response and remaining questions.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.