- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the interaction between decoupled weight decay, momentum, and effective learning rate. It derives an approximate momentum-dependent learning rate for normalized updates and proposes ScionC to separate learning-rate scheduling from weight-norm control. Experiments on ViT-S/16 and NanoGPT examine learning-rate transfer across momentum values, approximate scale invariance, and momentum scheduling, with Scion as the main performance baseline.
Critical strengths
The momentum sweeps directly test the proposed parameterization. The optimal effective learning rate varies much less than the raw learning rate across momentum settings. This provides useful evidence for simplifying hyperparameter tuning within the evaluated setups.
Critical weaknesses
-
The theoretical contribution is limited and insufficiently distinguished from existing work. The central derivation follows from a steady-state second moment calculation under independent gradient and approximately constant momentum norm assumptions. This is a straightforward extension of existing equilibrium analyses. Section 4 acknowledges that Wen et al. (2026) obtain the same equilibrium norm as Eq. 4 for L2 normalized updates, while Liu et al. (2026) report closely related effective-learning-rate collapse. These connections deserve substantially more discussion to establish what new insight this paper contributes. The main text also spends too much space restating AdamW and Scion update rules and expanding elementary derivations. Much of this material should move to the appendix, leaving the main paper to explain the new conclusions, their assumptions, and their empirical support.
-
The empirical evaluation is too narrow to establish broader practical value. The experiments are limited to ViT-S/16 and a single 124M parameter language model, with no evidence at larger model scales. The main performance comparison in Table 2 includes only ScionC and Scion. Particularly relevant missing comparisons are Muon Hyperball (MuonH) and Adam Hyperball (AdamH) from Wen et al. (2026), which explicitly control weight and update norms and are evaluated up to 1.2B parameters. SOAP is another relevant modern optimizer for broader performance comparisons. Despite the substantial Adam family discussion, the AdamW in Figure 1 and AdamC beta comparison in Appendix E do not replace systematic AdamW/AdamC performance comparisons under matched training and tuning budgets. Consequently, the experiments leave unclear how broadly the proposed correction helps beyond the particular Scion configurations studied.
-
The paper appears to have been completed in a rush. Figure 11 appears to consist of wandb screenshots, with panel borders and raw metric names such as
val/acc@1,l2_params, andl2_gradswith small faint labels. There are also inconsistencies beyond figure styling: Figure 1's caption describes gradient-norm results, but the third panel actually plots learning rate.
Questions for authors
See critical weaknesses.
2 comments
Thank you for the response and revision. The presentation issues are now resolved, and the new AdamW/AdamC appendix and weight norm analyses add useful context. However, my main concerns remain. The revision adds no further distinction from Wen et al. (2026), whose equilibrium norm the paper describes as identical to Eq. 4, which already includes momentum. The experiment scales are still limited, with no hyperball comparisons, and the AdamW/AdamC baselines are not tuned under a budget matched to ScionC.
Distinction from Existing Work
It is already stated in Related Works 3 that we derive γeff from the approximation of the equilibrium norm while Wen et al. (2026) does not. Furthermore, to our best knowledge ours is the only work on the scale invariance of optimizers with corrected WD. If this is "straightforward" then it is worthwhile to point out that the community has missed this critical dependency of Scion / Muon's γeff on momentum for the last $\approx$2 years. Other distinctions not stated for space and brevity reasons include the case with Nesterov momentum, differences between the training dynamics of input / hidden layers vs. output layer, aggregated near-scale-invariance (∥Jo∥F≈∥w∥2c) as a weaker yet verifiable condition when the model lacks perfect scale invariance, etc.
We have removed repetitions and shortened derivations so the rebuttal revision has more than 1 page of new content, but it is common to state the update rules and keep the derivation in the main text at least once (e.g. Kosson et al. (2024) and Wen et al. (2026)) and in our opinion it helps notation consistency and makes it easy to follow. For more repetitive and tedious derivations with momentum variants we have already relegated them to Appx. A.
Experiment Scale and Applicability
Besides FAQ 1: We choose the optimizers to experiment with in order to answer the specific questions:
This is the reason why we run both regular and Nesterov momentum while none of the concurrent hyperball counterparts does. Wen et al. (2026) actually includes a couple of 8B parameter experiments (Fig. 3MR) but the size of the biggest model is not the only measurement of scale. E.g. we perform the momentum and scale sweeps of Modded-NanoGPT with more compute and near the maximum parameter counts of their depth and width scaling (Fig. 5 and 6) so in fact we are more confident that the results hold at relevant scales. This is also apparent from comparing the validation losses reached in our experiments.
We include substantial Adam family discussion because WD correction was first purposed for AdamC and misattributed RMS-matching factors have been empirically found to allow Muon to reuse the optimal LR of AdamW. Nonetheless, we have added AdamW / AdamC comparison as Appx. D.
Caption Error and wandb Screenshot
Both are fixed.