- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the interaction between momentum and gradient normalization in Adam style optimizers. It proposes MVN Grad, which estimates gradient variance from the residual between the gradient and its moving average, normalizes the gradient using this variance, and then applies momentum. The paper analyzes its update variance, spike response, low variance behavior, and convergence properties. Experiments on CIFAR100 and language modeling show that MVN Grad performs similarly to or slightly better than Adam, AdaBelief, and LaProp, with better robustness in some hyperparameter settings.
Critical strengths
Theoretical analysis. The paper explains why the order of normalization and momentum matters. It derives the conditional variance difference between AdaBelief and MVN Grad, considers imperfect mean tracking, and shows that normalizing gradients before momentum can reduce the effect of large spikes. It also analyzes why second moment normalization becomes close to a sign update in low variance regimes and gives a convergence guarantee for MVN Grad.
Algorithm design. MVN Grad follows from this analysis. It estimates gradient variance around a moving mean, normalizes the current gradient with this estimate, and then applies momentum. This separates variance estimation from the momentum used for the update while keeping the method close to existing Adam style optimizers.
Critical weaknesses
Novelty and presentation. MVN Grad combines variance normalization from AdaBelief with momentum after normalization from LaProp, while centered RMSProp already uses a closely related update. It is therefore unclear what is new in the optimizer itself. The writing is also not very linear. I think the paper should first state the two problems, then present the analysis, and finally introduce MVN Grad based on these results.
Experiments. The experimental coverage is limited for a general purpose optimizer. The vision experiments only use ResNet18 on CIFAR100, and the language modeling experiments are all GPT style models. The gains over AdaBelief are small in several settings, and the 350M FineWeb experiment only runs for 4,000 iterations. Centered RMSProp is also missing as a baseline despite its close connection to MVN Grad. The current results are not enough to show that MVN Grad consistently improves over existing optimizers across architectures and tasks.
Questions for authors
- The paper notes that centered RMSProp has a closely related update. Please make the difference more explicit, preferably by writing the two updates side by side, and include centered RMSProp as a baseline with the same tuning budget.
- Can the authors broaden the experiments? The current evaluation is limited to ResNet18 on CIFAR100 and GPT style language models. Recent general purpose optimizer papers such as Lion [1] and Prodigy [2] cover a much wider range of datasets, architectures, and tasks. I would like to see at least ImageNet and a ViT setting. A diffusion model would also help test the method under a different objective and gradient regime.
- The language modeling experiments are also limited in scale and training length. The 350M FineWeb experiment only runs for 4,000 iterations. Even optimizer papers focused on language model pretraining, such as Sophia [3], evaluate larger models and longer training. Can the authors provide a longer run or a larger scale experiment?
- Theorem 3.4 studies simplified limiting dynamics rather than the full MVN Grad and LaProp updates, and the lower bound uses an initialization that depends on the horizon. What does this theorem tell us about the actual optimizers used in the experiments?
- The Muon reference appears to have the wrong spelling for one author: “Franz Cecista” should be “Franz Cesista” (https://kellerjordan.github.io/posts/muon/).
My current score is 2. I would raise it to 3 if the rebuttal can clarify these concerns.
References
[1] Chen et al. Symbolic Discovery of Optimization Algorithms. NeurIPS, 2023. [2] Mishchenko and Defazio. Prodigy: An Expeditiously Adaptive Parameter-Free Learner. ICML, 2024. [3] Liu et al. Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training. ICLR, 2024.
7 comments
Answer Part 2
Question 2 and experimental weakness: Broader architectures and the empirical contribution.
The main evaluation covers ResNet-18 on CIFAR-100 at two batch sizes and GPT-style models at 30M, 124M, and 350M parameters on three language-modeling datasets. The appendix additionally includes MNIST MLP diagnostics. Across the main benchmarks, we report selected-configuration multi-seed results and performance across the specified tuning grids.
The empirical contribution should be assessed using both forms of evidence. Close performance at selected configurations establishes competitiveness; Fig. 2 evaluates whether that performance is sensitive to the hyperparameters selected. In particular:
-
At CIFAR-100 batch size 1024, MVN-Grad improves median test accuracy over AdaBelief by 1.92 percentage points (78.33% versus 76.41%), alongside a 0.29-point improvement in selected-configuration mean test accuracy (79.63% versus 79.34%). Median training accuracy is 81.65% versus 73.87%.
-
At batch size 128, the medians are comparable, while MVN-Grad has a shorter low-performance tail.
-
On OpenWebText, median losses are nearly identical for MVN-Grad and AdaBelief, while normalize-then-momentum methods exhibit narrower interquartile ranges.
Small differences between the best selected configurations therefore do not exhaust the practical evidence. Competitive performance together with reduced degradation under different hyperparameter choices is useful when each unsuccessful configuration incurs the cost of a full training run.
ImageNet, ViTs, and diffusion models would broaden the evaluation. Our claims are tied to the architectures, tasks, and grids studied. Within that scope, the paper combines mechanistic analysis, competitive tuned performance, and reduced hyperparameter sensitivity in several settings. FineWeb supplies complementary evidence of competitiveness at the larger evaluated model scale, rather than the main evidence for hyperparameter robustness.
Question 3: Longer training or larger language models.
The 4,000-update horizon was chosen to target a training-token budget of the same order as the Chinchilla prescription from Hoffmann et al., 2022 for a model of this size. Using nanoGPT's global accumulation convention, it processes
\approx 3.54\text{B tokens},$$ which is within a factor of two of the common reference of approximately 20 tokens per parameter for a nominal 350M model. We will state this budget explicitly: the choice is Chinchilla-motivated, rather than an arbitrary count of early iterations or a claim of exact compute optimality. All methods complete the prescribed schedule, use the same data and training setup and row-clipping rule, and separately select their learning rate from the reported grid before four-seed evaluation. This supports a fixed-budget comparison at 350M. The submission also includes a substantially longer OpenWebText experiment: 120,000 updates for GPT-2 124M, corresponding to approximately 59B processed tokens. Thus, the evidence is not limited to 4,000-update language-model runs. Longer training at 124M is complementary to longer training at 350M or a larger model, and we do not infer the latter's outcome from the former. The FineWeb conclusion remains that MVN-Grad retains competitive performance at the larger evaluated model scale and budget. Longer training or larger models would extend the evaluation, but the existing 4,000-update horizon is already motivated by model size and a token budget of the same order as the Chinchilla prescription. The iteration count alone therefore does not justify characterizing this experiment as an insignificant training horizon. Together with the completed learning-rate schedule, matched per-run training budgets, and method-specific learning-rate selection, this supports a meaningful comparison at the evaluated scale. The 120,000-update OpenWebText experiment supplies complementary evidence over a substantially longer training horizon.Answer Part 3
Question 4: What Theorem 3.4 implies for the implemented optimizers.
Theorem 3.4 analyzes exact surrogate recursions motivated by an idealized local regime. Its purpose is to explain a consequence of normalizer choice. When the second-moment estimate tracks the squared signal and the signal dominates the noise and stabilizer, the normalized input becomes approximately coordinate-wise sign. When the variance normalizer remains approximately fixed, the input instead preserves coordinate-wise gradient magnitudes.
The theorem formalizes properties of these surrogate dynamics. The sign construction uses a horizon-dependent initialization, with initial gap , and a specified learning-rate schedule. As explicitly stated after the theorem, it is not a standard fixed-initial-gap oracle-complexity separation or a comparison over optimally tuned learning rates.
The theoretical insight is that second-moment normalization can discard gradient-magnitude information and produce sign-like dynamics, whereas a stabilized variance normalizer preserves that information. Theorem 3.4 illustrates an optimization consequence of this distinction under its stated assumptions; it does not assert that the resulting worst-case bounds determine the empirical convergence speed of the implemented optimizers. Transferring this comparison to the full EMA algorithms would require controlling finite-time normalizer tracking errors. We will make that interpretation clearer and avoid wording that suggests a general dimension-dependent advantage for the full optimizers. Theorem 3.5 separately establishes the general nonconvex guarantee for the uncorrected MVN-Grad recursion.
Question 5. Thank you for identifying the author-name error.
Presentation weakness: Order of motivation, analysis, and algorithm. We appreciate this suggestion and will improve the presentation to make the connection between the two design questions, the algorithmic construction, and the theoretical analysis more explicit. We will also clarify the assumptions and main conclusions of each result to improve readability.
References
- Hoffmann, J., et al. (2022). Training compute-optimal large language models.
Answer Part 1
We thank the reviewer for recognizing the theoretical analysis and algorithmic motivation. We organize the response around the five questions, indicating where each also addresses the novelty or experimental weaknesses, and then address the separate presentation concern.
Question 1 and novelty weakness: Centered RMSProp comparison and baseline.
Centered RMSProp with momentum already combines centered normalization and momentum applied afterward. We acknowledge this relationship in the introduction and agree that an explicit comparison would improve the presentation.
Using elementwise operations and omitting bias corrections, regularization, and variance floors, a modern centered RMSProp formulation is
a_t &= \alpha a_{t-1}+(1-\alpha)g_t,\\\\ v_t &= \alpha v_{t-1}+(1-\alpha)g_t^2,\\\\ q_t &= v_t-a_t^2,\\\\ b_t &= \mu b_{t-1}+\frac{g_t}{\sqrt{q_t}+\varepsilon}. \end{aligned}$$ The corresponding MVN-Grad equations are $$\begin{aligned} m_t &= \beta_1 m_{t-1}+(1-\beta_1)g_t,\\\\ s_t &= \beta_2 s_{t-1}+(1-\beta_2)(g_t-m_t)^2,\\\\ u_t &= \beta_1 u_{t-1} +(1-\beta_1)\frac{g_t}{\sqrt{s_t}+\varepsilon}. \end{aligned}$$ The structural distinction concerns how the mean and variance statistics evolve. Centered RMSProp uses a common decay for its first and second moments. Its centered statistic therefore satisfies the exact recursion $$q_t=\alpha q_{t-1}+\alpha(1-\alpha)(g_t-a_{t-1})^2.$$ In contrast, MVN-Grad uses $$s_t=\beta_2s_{t-1}+(1-\beta_2)\beta_1^2(g_t-m_{t-1})^2,$$ where the mean-tracking decay $\beta_1$ can differ from the residual-averaging decay $\beta_2$. Thus, the two methods impose different couplings between mean tracking, normalization, and update momentum. A constant learning-rate rescaling does not generally reconcile these different statistical recursions. There is an instructive special case: with $\beta_1=\beta_2=\alpha=\beta\in(0,1)$, zero initialization, and no variance floor, $$s_t=\beta(v_t-m_t^2).$$ Without stabilizers or bias corrections, and with matching update momentum, this yields equivalence up to a constant learning-rate scaling. This identity identifies where the formulations coincide; it does not establish their equivalence for distinct mean and variance decays. Algorithm 1 also specifies bias corrections and an optional variance floor. The equations above use the modern outside-root stabilizer convention; Graves's original formulation places the stabilizer inside the square root. Our contribution combines this residual-based construction with an explicit analysis of two design choices: what information the normalizer retains, and how its placement relative to momentum affects the update. The conditional variance identity, spike-response analysis, and limiting normalization comparison characterize these mechanisms under their respective assumptions. The experiments then evaluate selected-configuration performance and sensitivity to hyperparameter choices. These analytical and empirical contributions remain substantive even where particular parameter choices recover an existing update. We therefore acknowledge centered RMSProp as a precursor to the broad combination of centered normalization and post-normalization momentum. Our experiments compare MVN-Grad with Adam, AdaBelief, and LaProp; comparative performance against centered RMSProp remains unmeasured. We will include the explicit relationship above and revise any wording suggesting that this broad combination was previously unoccupied.