acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Optimization via Momentum on Variance-Normalized Gradients

Abstract

We introduce MVN-Grad (Momentum on Variance-Normalized Gradients), an Adam-style optimizer that improves stability and performance by combining two complementary ideas: variance-based normalization and momentum applied after normalization. MVN-Grad scales each coordinate by an exponential moving average of gradient uncertainty and applies momentum to the resulting normalized gradients, removing the cross-time coupling between stale momentum and a stochastic normalizer present in standard Adam-type updates. We prove that, when the mean estimate tracks the conditional mean exactly and the noise is symmetric, this decoupling yields no larger one-step conditional update variance than momentum-then-normalize variance methods, and that, as in LaProp, an isolated spike is normalized before it enters the momentum buffer. In low-variance regimes, we further show that variance normalization avoids sign-type collapse of second-moment scaling in idealized limiting dynamics. Beyond these comparisons, we prove a general nonconvex convergence guarantee for MVN-Grad under bounded-gradient stochastic assumptions. On CIFAR-100 and GPT-style language modeling with models up to 350M parameters, MVN-Grad matches or improves on Adam, AdaBelief, and LaProp, with improved robustness in several evaluated hyperparameter sweeps at the cost of one additional optimizer-state tensor.

Primary area:
optimization
Keywords:
adaptive optimization, stochastic optimization, adaptive gradient methods, momentum, variance normalization, Adam, deep learning optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_63dhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the interaction between momentum and gradient normalization in Adam style optimizers. It proposes MVN Grad, which estimates gradient variance from the residual between the gradient and its moving average, normalizes the gradient using this variance, and then applies momentum. The paper analyzes its update variance, spike response, low variance behavior, and convergence properties. Experiments on CIFAR100 and language modeling show that MVN Grad performs similarly to or slightly better than Adam, AdaBelief, and LaProp, with better robustness in some hyperparameter settings.

Critical strengths

Theoretical analysis. The paper explains why the order of normalization and momentum matters. It derives the conditional variance difference between AdaBelief and MVN Grad, considers imperfect mean tracking, and shows that normalizing gradients before momentum can reduce the effect of large spikes. It also analyzes why second moment normalization becomes close to a sign update in low variance regimes and gives a convergence guarantee for MVN Grad.

Algorithm design. MVN Grad follows from this analysis. It estimates gradient variance around a moving mean, normalizes the current gradient with this estimate, and then applies momentum. This separates variance estimation from the momentum used for the update while keeping the method close to existing Adam style optimizers.

Critical weaknesses

Novelty and presentation. MVN Grad combines variance normalization from AdaBelief with momentum after normalization from LaProp, while centered RMSProp already uses a closely related update. It is therefore unclear what is new in the optimizer itself. The writing is also not very linear. I think the paper should first state the two problems, then present the analysis, and finally introduce MVN Grad based on these results.

Experiments. The experimental coverage is limited for a general purpose optimizer. The vision experiments only use ResNet18 on CIFAR100, and the language modeling experiments are all GPT style models. The gains over AdaBelief are small in several settings, and the 350M FineWeb experiment only runs for 4,000 iterations. Centered RMSProp is also missing as a baseline despite its close connection to MVN Grad. The current results are not enough to show that MVN Grad consistently improves over existing optimizers across architectures and tasks.

Questions for authors
  1. The paper notes that centered RMSProp has a closely related update. Please make the difference more explicit, preferably by writing the two updates side by side, and include centered RMSProp as a baseline with the same tuning budget.
  2. Can the authors broaden the experiments? The current evaluation is limited to ResNet18 on CIFAR100 and GPT style language models. Recent general purpose optimizer papers such as Lion [1] and Prodigy [2] cover a much wider range of datasets, architectures, and tasks. I would like to see at least ImageNet and a ViT setting. A diffusion model would also help test the method under a different objective and gradient regime.
  3. The language modeling experiments are also limited in scale and training length. The 350M FineWeb experiment only runs for 4,000 iterations. Even optimizer papers focused on language model pretraining, such as Sophia [3], evaluate larger models and longer training. Can the authors provide a longer run or a larger scale experiment?
  4. Theorem 3.4 studies simplified limiting dynamics rather than the full MVN Grad and LaProp updates, and the lower bound uses an initialization that depends on the horizon. What does this theorem tell us about the actual optimizers used in the experiments?
  5. The Muon reference appears to have the wrong spelling for one author: “Franz Cecista” should be “Franz Cesista” (https://kellerjordan.github.io/posts/muon/).

My current score is 2. I would raise it to 3 if the rebuttal can clarify these concerns.

References

[1] Chen et al. Symbolic Discovery of Optimization Algorithms. NeurIPS, 2023. [2] Mishchenko and Defazio. Prodigy: An Expeditiously Adaptive Parameter-Free Learner. ICML, 2024. [3] Liu et al. Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training. ICLR, 2024.

7 comments
AuthorsOpenReview ↗

Answer Part 1

We thank the reviewer for recognizing the theoretical analysis and algorithmic motivation. We organize the response around the five questions, indicating where each also addresses the novelty or experimental weaknesses, and then address the separate presentation concern.

Question 1 and novelty weakness: Centered RMSProp comparison and baseline.

Centered RMSProp with momentum already combines centered normalization and momentum applied afterward. We acknowledge this relationship in the introduction and agree that an explicit comparison would improve the presentation.

Using elementwise operations and omitting bias corrections, regularization, and variance floors, a modern centered RMSProp formulation is

a_t &= \alpha a_{t-1}+(1-\alpha)g_t,\\\\ v_t &= \alpha v_{t-1}+(1-\alpha)g_t^2,\\\\ q_t &= v_t-a_t^2,\\\\ b_t &= \mu b_{t-1}+\frac{g_t}{\sqrt{q_t}+\varepsilon}. \end{aligned}$$ The corresponding MVN-Grad equations are $$\begin{aligned} m_t &= \beta_1 m_{t-1}+(1-\beta_1)g_t,\\\\ s_t &= \beta_2 s_{t-1}+(1-\beta_2)(g_t-m_t)^2,\\\\ u_t &= \beta_1 u_{t-1} +(1-\beta_1)\frac{g_t}{\sqrt{s_t}+\varepsilon}. \end{aligned}$$ The structural distinction concerns how the mean and variance statistics evolve. Centered RMSProp uses a common decay for its first and second moments. Its centered statistic therefore satisfies the exact recursion $$q_t=\alpha q_{t-1}+\alpha(1-\alpha)(g_t-a_{t-1})^2.$$ In contrast, MVN-Grad uses $$s_t=\beta_2s_{t-1}+(1-\beta_2)\beta_1^2(g_t-m_{t-1})^2,$$ where the mean-tracking decay $\beta_1$ can differ from the residual-averaging decay $\beta_2$. Thus, the two methods impose different couplings between mean tracking, normalization, and update momentum. A constant learning-rate rescaling does not generally reconcile these different statistical recursions. There is an instructive special case: with $\beta_1=\beta_2=\alpha=\beta\in(0,1)$, zero initialization, and no variance floor, $$s_t=\beta(v_t-m_t^2).$$ Without stabilizers or bias corrections, and with matching update momentum, this yields equivalence up to a constant learning-rate scaling. This identity identifies where the formulations coincide; it does not establish their equivalence for distinct mean and variance decays. Algorithm 1 also specifies bias corrections and an optional variance floor. The equations above use the modern outside-root stabilizer convention; Graves's original formulation places the stabilizer inside the square root. Our contribution combines this residual-based construction with an explicit analysis of two design choices: what information the normalizer retains, and how its placement relative to momentum affects the update. The conditional variance identity, spike-response analysis, and limiting normalization comparison characterize these mechanisms under their respective assumptions. The experiments then evaluate selected-configuration performance and sensitivity to hyperparameter choices. These analytical and empirical contributions remain substantive even where particular parameter choices recover an existing update. We therefore acknowledge centered RMSProp as a precursor to the broad combination of centered normalization and post-normalization momentum. Our experiments compare MVN-Grad with Adam, AdaBelief, and LaProp; comparative performance against centered RMSProp remains unmeasured. We will include the explicit relationship above and revise any wording suggesting that this broad combination was previously unoccupied.
AuthorsOpenReview ↗

Answer Part 2

Question 2 and experimental weakness: Broader architectures and the empirical contribution.

The main evaluation covers ResNet-18 on CIFAR-100 at two batch sizes and GPT-style models at 30M, 124M, and 350M parameters on three language-modeling datasets. The appendix additionally includes MNIST MLP diagnostics. Across the main benchmarks, we report selected-configuration multi-seed results and performance across the specified tuning grids.

The empirical contribution should be assessed using both forms of evidence. Close performance at selected configurations establishes competitiveness; Fig. 2 evaluates whether that performance is sensitive to the hyperparameters selected. In particular:

  • At CIFAR-100 batch size 1024, MVN-Grad improves median test accuracy over AdaBelief by 1.92 percentage points (78.33% versus 76.41%), alongside a 0.29-point improvement in selected-configuration mean test accuracy (79.63% versus 79.34%). Median training accuracy is 81.65% versus 73.87%.

  • At batch size 128, the medians are comparable, while MVN-Grad has a shorter low-performance tail.

  • On OpenWebText, median losses are nearly identical for MVN-Grad and AdaBelief, while normalize-then-momentum methods exhibit narrower interquartile ranges.

Small differences between the best selected configurations therefore do not exhaust the practical evidence. Competitive performance together with reduced degradation under different hyperparameter choices is useful when each unsuccessful configuration incurs the cost of a full training run.

ImageNet, ViTs, and diffusion models would broaden the evaluation. Our claims are tied to the architectures, tasks, and grids studied. Within that scope, the paper combines mechanistic analysis, competitive tuned performance, and reduced hyperparameter sensitivity in several settings. FineWeb supplies complementary evidence of competitiveness at the larger evaluated model scale, rather than the main evidence for hyperparameter robustness.

Question 3: Longer training or larger language models.

The 4,000-update horizon was chosen to target a training-token budget of the same order as the Chinchilla prescription from Hoffmann et al., 2022 for a model of this size. Using nanoGPT's global accumulation convention, it processes

\approx 3.54\text{B tokens},$$ which is within a factor of two of the common reference of approximately 20 tokens per parameter for a nominal 350M model. We will state this budget explicitly: the choice is Chinchilla-motivated, rather than an arbitrary count of early iterations or a claim of exact compute optimality. All methods complete the prescribed schedule, use the same data and training setup and row-clipping rule, and separately select their learning rate from the reported grid before four-seed evaluation. This supports a fixed-budget comparison at 350M. The submission also includes a substantially longer OpenWebText experiment: 120,000 updates for GPT-2 124M, corresponding to approximately 59B processed tokens. Thus, the evidence is not limited to 4,000-update language-model runs. Longer training at 124M is complementary to longer training at 350M or a larger model, and we do not infer the latter's outcome from the former. The FineWeb conclusion remains that MVN-Grad retains competitive performance at the larger evaluated model scale and budget. Longer training or larger models would extend the evaluation, but the existing 4,000-update horizon is already motivated by model size and a token budget of the same order as the Chinchilla prescription. The iteration count alone therefore does not justify characterizing this experiment as an insignificant training horizon. Together with the completed learning-rate schedule, matched per-run training budgets, and method-specific learning-rate selection, this supports a meaningful comparison at the evaluated scale. The 120,000-update OpenWebText experiment supplies complementary evidence over a substantially longer training horizon.
AuthorsOpenReview ↗

Answer Part 3

Question 4: What Theorem 3.4 implies for the implemented optimizers.

Theorem 3.4 analyzes exact surrogate recursions motivated by an idealized local regime. Its purpose is to explain a consequence of normalizer choice. When the second-moment estimate tracks the squared signal and the signal dominates the noise and stabilizer, the normalized input becomes approximately coordinate-wise sign. When the variance normalizer remains approximately fixed, the input instead preserves coordinate-wise gradient magnitudes.

The theorem formalizes properties of these surrogate dynamics. The sign construction uses a horizon-dependent initialization, with initial gap Θ(c2dT/L)\Theta(c^2dT/L), and a specified learning-rate schedule. As explicitly stated after the theorem, it is not a standard fixed-initial-gap oracle-complexity separation or a comparison over optimally tuned learning rates.

The theoretical insight is that second-moment normalization can discard gradient-magnitude information and produce sign-like dynamics, whereas a stabilized variance normalizer preserves that information. Theorem 3.4 illustrates an optimization consequence of this distinction under its stated assumptions; it does not assert that the resulting worst-case bounds determine the empirical convergence speed of the implemented optimizers. Transferring this comparison to the full EMA algorithms would require controlling finite-time normalizer tracking errors. We will make that interpretation clearer and avoid wording that suggests a general dimension-dependent advantage for the full optimizers. Theorem 3.5 separately establishes the general nonconvex guarantee for the uncorrected MVN-Grad recursion.

Question 5. Thank you for identifying the author-name error.

Presentation weakness: Order of motivation, analysis, and algorithm. We appreciate this suggestion and will improve the presentation to make the connection between the two design questions, the algorithmic construction, and the theoretical analysis more explicit. We will also clarify the assumptions and main conclusions of each result to improve readability.

References

  • Hoffmann, J., et al. (2022). Training compute-optimal large language models.
Reviewer_Fq97OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors first address that there are two design choices for Adam-like optimizer design: (1) the choice of the denominator (second-moment vs. variance), and the choice of the ordering (momentum-after-normalization vs. normalization-after-momentum). Previous works, i.e., Adam, AdaBelief, and LaProp consist three out of four available configurations in this perspective and the presented MVN-Grad fills the remaining one, applying Adam-like gradient preconditioning and then applying an additional EMA smoothing. Theoretical justification of this design choice focuses on variance reduction and spike response (forgetting of instantaneous signal in the denominator). Experiments on CIFAR-100 (ResNet) and various language modeling task (few hundreds of millions of parameters) are conducted to justify the design, showing consistently improved performance of MVN-Grad with respect to three other design choices: Adam, AdaBelief, and LaProp.

Critical strengths
  • The problem setting and the introductory remark on the missing combination in the existing optimizer families is persuasive.
  • The paper invests a great proportion of the manuscript on justifying the design choices through diverse mathematical claims. Many of which is relative and supportive of the design.
  • The experimental results show consistent performance boost.
Critical weaknesses
  • First of all, the design is more complicated than how it was sold in the introductory section, and seems not natural enough. Concretely speaking, since we are calculating s_t with additional EMA which is generally different from the variance estimating clock, there is a bias in the estimation.
  • I find that there are actually three EMAs in Algorithm 1: lines 4, 5, and 8, with lines 4 and 8 coupled with the same hyperparameter beta_1. In the lines 128-132, the authors note that the reuse of the same hyperparameter is to match with Adam’s interface. However, since we are introducing a new algorithm that may replace Adam’s supremacy, we do not really have to match the same interface. I wonder what happens if we can change the hyperparameters.
  • I generally like the experimental sections except FineWeb (350M). This experiment only uses 4k iterations. This is quite unnatural to compare optimizers. It is known (Fascinating Pretraining Optimizers and Where to Find Them by Wen et al (2025)) that different optimization algorithms may start in different speed in the first tens of thousands of iterations but they eventually leads to similar performance in the long run. So I believe 4k iteration results are not credible in this sense. The idea itself in the introductory section is novel and worth exploring. But the current version of the manuscript looks unfinished to fully deliver the advantage of this idea.
  • This is not critical in assessment but the theorem items and how they are presented and discusses in the main manuscript can be more compact. For example, there are many in-line equations that are mixed up with elaborative definitions, but we can make it concise by explaining what each theorem item wants to deliver. I understand the space constraint but we need some linebreaks for enhanced readability.
Questions for authors
  • How should we choose beta_1 and beta_2? Even though the algorithm wants to follow the Adam’s interface, it is generally a different algorithm. Different combinations of these hyperparmeters might be more beneficial. Hyperparameter sweep test comparing with Adam will make the “interface-matching” philosophy more believable.
4 comments
AuthorsOpenReview ↗

Answer Part 1

We thank the reviewer for recognizing the motivation, analysis, and empirical results. Our main empirical objective is to evaluate both selected-configuration performance and robustness across hyperparameter choices. We address the four weaknesses in their order, followed by the explicit question on parameter selection and comparison with Adam.

Weakness 1: Variance-estimation bias and the roles of the three EMAs.

The three EMAs have distinct roles: mtm_t estimates the gradient mean, sts_t tracks squared residuals, and utu_t smooths normalized gradients. Maintaining separate mean and update-momentum states allows centering to operate on raw gradients while momentum operates on normalized gradients.

The quantity sts_t is a variance proxy, rather than a generally unbiased estimate of the instantaneous conditional variance. Because mtm_t includes the current gradient, gt−mt=β1(gt−mt−1).g_t-m_t=\beta_1(g_t-m_{t-1}).

For an i.i.d. gradient sequence with fixed mean and variance σ2\sigma^2, the stationary limit satisfies lim⁡t→∞E[(gt−mt)2]=2β121+β1σ2.\lim_{t\to\infty}\mathbb E[(g_t-m_t)^2] = \frac{2\beta_1^2}{1+\beta_1}\sigma^2.

Thus, even in this stationary setting, the residual statistic has a multiplicative calibration factor. In nonstationary settings it also reflects mean-tracking error. The correction 1−β2t1-\beta_2^t removes the zero-initialization attenuation of the residual EMA; it does not make the residual statistic unbiased for the gradient variance.

This distinction does not invalidate the analysis: Theorem 3.1 and the general convergence guarantee use the actual residual recursion, rather than assuming an unbiased variance estimator. Section 3.3 separately introduces explicitly idealized stabilized-normalizer assumptions. We will clarify this terminology throughout.

Weakness 2: Why share the mean and update-momentum decay?

Mean tracking and update momentum serve different purposes and need not have identical optimal timescales. They can be decoupled as

\qquad u_t=\mu u_{t-1}+(1-\mu)z_t.$$ with $\beta_2$ controlling the residual statistic. In exploratory experiments outside the reported evaluation, we have observed performance improvements from decoupling $\rho$ and $\mu$. Algorithm 1 nevertheless ties $\rho=\mu=\beta_1$ to retain the same number of decay hyperparameters as the baselines. Independently sweeping an additional decay would enlarge MVN-Grad's search space and, if added to the existing grid, increase its tuning budget. We therefore use the tied parameterization to keep the comparison controlled and the tuning protocol directly comparable across methods. This is a deliberate experimental-design choice, rather than a claim that identical timescales are intrinsically optimal. The reported competitiveness and robustness are obtained with this simpler parameterization.
AuthorsOpenReview ↗

Answer Part 2

Weakness 3: Credibility of the 4,000-update FineWeb comparison.

We respectfully disagree that the number of iterations alone makes this comparison non-credible. Training duration should be considered together with the global token batch, model size, and learning-rate schedule.

The 4,000-update horizon was chosen to target a training-token budget of the same order as the Chinchilla prescription from Hoffmann et al., 2022 for this model size. Under nanoGPT's accumulation convention, micro-batch size 12, configured accumulation 72, and context length 1024 give 884,736 tokens per update. The run therefore processes approximately 3.54B tokens for a nominal 350M-parameter model, within a factor of two of the commonly used reference of approximately 20 tokens per parameter. Thus, the horizon was motivated by model size and token budget. This is a Chinchilla-motivated choice of scale, rather than a claim that the precise allocation is compute-optimal. We will state the token budget explicitly.

The reported checkpoint is the end of the prescribed training schedule, with 400 warmup updates and a final-to-peak learning-rate ratio of 0.1. It is not an intermediate checkpoint selected from a longer unfinished schedule. Wen et al. (2025) emphasize final-checkpoint evaluation, careful optimizer-specific tuning, and dependence on model and data scale. Their findings do not establish a universal minimum number of training iterations or imply that all optimizers necessarily become equivalent after sufficiently many updates. Our restricted momentum search remains a limitation, but it is distinct from the concern about evaluating intermediate checkpoints.

Furthermore, the paper includes GPT-2 124M training on OpenWebText for 120,000 updates, with an effective batch of 480 sequences and context length 1024, corresponding to approximately 59B processed tokens. Thus, the evaluation is not restricted to early training. This provides complementary evidence at 124M without determining the outcome of longer training at 350M.

The FineWeb conclusion is appropriately limited: MVN-Grad achieves 3.2344±0.00893.2344\pm0.0089 validation loss, compared with 3.2360±0.00653.2360\pm0.0065 for AdamW and 3.2350±0.00773.2350\pm0.0077 for AdaBelief. These results establish competitiveness at the evaluated scale and budget. The primary evidence for improved hyperparameter robustness comes from the sweep distributions, rather than interpreting these small FineWeb differences as a decisive advantage.

Weakness 4: Readability of the theorem statements.

We appreciate the suggestion to streamline the theorem discussion. We will separate assumptions and key equations from the surrounding prose and state the purpose of each result before its formal statement.

Question:

How we choose the parameters. We select optimizer configurations using the task-specific search spaces and selection protocols reported in Appendices C.5--C.6, tuning the learning rate together with the decay parameters where the protocol permits and evaluating the selected configurations over multiple seeds. In MVN-Grad, β1\beta_1 controls the mean-tracking and update-momentum timescales, while β2\beta_2 controls the residual-statistic timescale. Their values should be selected jointly with the learning rate; sharing Adam's interface does not establish that its selected values are optimal for MVN-Grad.

Existing comparison with Adam. Matching Adam's interface does not mean transferring Adam's selected hyperparameter values. Appendix C.5 reports the CIFAR-100 sweeps and method-specific configurations, while Appendix C.6 reports the language-modeling grids. For example, at CIFAR-100 batch size 1024, the selected pairs are (0.8,0.999)(0.8,0.999) for Adam and (0.7,0.99)(0.7,0.99) for MVN-Grad. WikiText-103 includes a shared focused grid over learning rate, β1\beta_1, and β2\beta_2, and OpenWebText evaluates the same nine momentum pairs for each optimizer at the common learning-rate regime.

The requested comparison of hyperparameter sensitivity with Adam is therefore already part of the submission. Figure 2 reports performance across these sweeps. For instance, at CIFAR-100 batch size 1024, median test accuracy is 78.33% for MVN-Grad versus approximately 74% for Adam. These distributions are central to the empirical claim that the method performs reliably across different evaluated configurations.

FineWeb uses a more restricted protocol: all methods use (β1,β2)=(0.9,0.95)(\beta_1,\beta_2)=(0.9,0.95), which preliminary experiments identified as a stable common setting across all four optimizers. Each optimizer separately selects its learning rate from the same grid. This experiment evaluates competitiveness at the larger model scale under a controlled protocol; it is not an exhaustive optimizer-specific momentum search.

References

  • Wen, K., et al. (2025). Fantastic pretraining optimizers and where to find them.
  • Hoffmann, J., et al. (2022). Training compute-optimal large language models.

I have read through the rebuttal. Thank you for the response. I think most of my concerns are well-addressed. I will raise my score accordingly, and recommend acceptance.

AuthorsOpenReview ↗

Revised manuscript uploaded — changes addressing your comments

Thank you for your constructive feedback and for reconsidering the submission in light of our responses. We have uploaded the revised manuscript, with revisions highlighted in blue. In particular:

  • Variance estimation: Section 2 explains the roles of the three EMAs, identifies the residual statistic as a variance proxy, and gives its stationary calibration factor.
  • Decay parameters: We explain that sharing the mean-tracking and update-momentum decay is a parameter-count choice, rather than a claim that their optimal timescales must coincide.
  • Training budget: Section 4.2 and Appendix C.6 explicitly report the approximately 3.54B-token FineWeb budget and its Chinchilla-motivated scale, together with the completed learning-rate schedule. We also state the approximately 59B processed tokens in the complementary OpenWebText experiment.
  • Readability: We improve the presentation of the theorem statements by displaying key conclusions and clarifying the purpose of the results.

We appreciate your suggestions, which helped us make the algorithmic choices and experimental protocol clearer.

Reviewer_hHesOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MVN-Grad, an Adam-style adaptive optimizer that combines two existing optimization techniques: variance-based gradient normalization and momentum applied after normalization. The proposed method aims to address two limitations of Adam: (1) instability caused by the interaction between historical momentum and the current adaptive normalizer, and (2) the loss of gradient-magnitude information when second-moment normalization produces sign-like updates. The paper provides several theoretical results:

  1. Update stability: Under exact mean tracking and symmetric gradient noise, MVN-Grad has no larger one-step conditional update variance than AdaBelief.
  2. Spike robustness: MVN-Grad prevents isolated gradient spikes from introducing unbounded contributions into subsequent updates under the studied conditions.
  3. High-signal optimization: Under idealized low-variance dynamics, variance normalization preserves gradient magnitudes, whereas second-moment normalization can reduce to sign-based optimization.
  4. Nonconvex convergence: The paper establishes an O(1/T)O(1/\sqrt{T}) convergence rate under bounded stochastic gradients with an appropriately chosen second-moment decay parameter.
Critical strengths
  1. The paper is clear and well-written. It develops a principled combination of two optimization mechanisms with clear motivation.
  2. The paper contains relatively comprehensive empirical evaluations.
Critical weaknesses
  1. The theoretical examinations do not establish strong evidence for the effectiveness of the algorithm

a. A smaller update variance does not mean a more stable optimization, or a better optimization. The optimization stability arguments (e.g. https://arxiv.org/html/2307.15196v2) should also examine trajectory-wise instability, and the generalization benefit is also well known in the empirical literatures of SGD vs GD.

b. Spike response is not a stand-out advantage over Adam as well. The paper acknowledges in Remark 3.1 that Adam's update is uniformly bounded when β2>β12\beta_2>\beta_1^2 as well, and insufficient evidence is provided to show that a definite edge is present under the Adam parameterization used commonly in practice. Furthermore, as the denominator in MVN-grad is a moving average of (gt−mt)2(g_t-m_t)^2, in an optimization with near-constant gradient and zero gradient noise, the optimizer actually creates great instability as the scale of updates is only controlled by ϵ\epsilon and can potentially be very large.

c. Comparison of MVN-Grad vs LaProp: the factor dd appears in the LaProp lower bound can be removed by applying a smaller learning rate of factor 1/d1/\sqrt{d}, so the conclusion is invalid. Only comparison of the two algorithm under optimal hyperparameters make sense in establishing an advantage.

Therefore the theoretical part fails to build an argument of why MVN-Grad is better than previous methods. The generalization benefits are also not shown.

  1. The idea in MVN-Grad design lacks novelty. It is a straight-forward combination of tricks used in AdaBelief and LaProp.

  2. The practical application of MVN-Grad requires 33% more memory than AdaBelief (extensive for large models), yet the benefits observed in the experiment is marginal (generally within 0.1%).

Questions for authors
  1. Is MVN-Grad better than LaProp with optimally tuned hyperparameters (potentially different)?
  2. Is MVN-Grad better than Adam in cases with gradient bounds?
4 comments
AuthorsOpenReview ↗

Answer Part 1

We thank the reviewer for recognizing the clarity, motivation, and empirical evaluation. The paper combines mechanism-level analysis with competitive tuned performance and reduced sensitivity to hyperparameter choices across several evaluated settings. The selected-configuration tables establish competitiveness, while Fig. 2 evaluates robustness across configurations. Small differences between the best selected configurations therefore do not capture the full empirical contribution. We address the reviewer's weaknesses and questions individually below.

Weakness 1(a): Conditional update variance, trajectory stability, and generalization.

Theorem 3.1 establishes an exact ordering-dependent difference at a common history and with the same variance proxy. Under its stated symmetry assumption, writing Yt=(st+ε)−1Y_t=(\sqrt{s_t}+\varepsilon)^{-1} gives

-\operatorname{Var}(\Delta_t^{\mathrm{MV}}\mid\mathcal F_{t-1}) = (2\beta_1-\beta_1^2)m_{t-1}^2 \operatorname{Var}(Y_t\mid\mathcal F_{t-1}) \ge 0.$$ This identifies the extra conditional variability introduced when the current random denominator rescales carried momentum. Proposition 3.2 considers imperfect mean tracking, and the frozen-checkpoint diagnostics in Fig. 1 and Appendix Fig. 4 evaluate the corresponding local quantity during training. The generalization behavior of SGD versus GD concerns a different question from the one addressed by Theorem 3.1. We compare two orderings using the same stochastic gradient and variance proxy at a common history. The identity isolates the additional conditional variability introduced by multiplying historical momentum by the current random normalizer; it does not assume that eliminating gradient noise improves generalization. Consequently, the possibility that stochasticity benefits generalization does not contradict this result. Theorem 3.1 makes a precise one-step statement, rather than a claim about trajectory-wise stability or generalization. Section 3 explicitly identifies this scope. A trajectory-level analysis would address an additional question; it is not required to establish the ordering-dependent mechanism proved here. We will clarify that references to "stability" in this subsection concern conditional update variability. The empirical claims are evaluated separately: complete training runs measure performance across hyperparameter configurations, while test accuracy and held-out loss measure predictive performance. Thus, our robustness argument rests on the sweep distributions, not on an inference from smaller conditional variance to better generalization. **Weakness 1(b): Spike response under common Adam settings and small variance denominators.** The spike analysis isolates the effect of normalizing an outlier before storing it in momentum. Its bounded-response property is shared with LaProp, as expected for a result concerning the ordering axis. Remark 3.1 explicitly identifies the regime $\beta_2>\beta_1^2$ in which Adam's updates are also uniformly bounded, including common settings such as $(0.9,0.95)$. The result characterizes the carry-over mechanism and its parameter dependence; it does not assert that Adam is unbounded at its usual defaults. The nearly constant, noise-free gradient example concerns the separate normalizer axis. With $g_t\equiv g\ne0$ and $\varepsilon_s=0$, we have $s_t\to0$, so the normalized input approaches $g/\varepsilon$. This explains why the effective step size and denominator regularization matter; it does not by itself establish instability of the optimization trajectory. Algorithm 1 permits a variance floor satisfying $\frac{s_t}{1-\beta_2^t} \ge \frac{\varepsilon_s}{1-\beta_2},$ which controls this amplification when enabled. Our OpenWebText and FineWeb settings use $\varepsilon_s=0$, so their performance is not attributed to a positive floor. The practical question is how these trade-offs affect complete training runs under different parameter choices. This is precisely why the sweep evidence is central to the paper.
AuthorsOpenReview ↗

Answer Part 2

Weakness 1(c) and Question 1: LaProp comparison and learning-rate tuning.

Theoretical point. The proposed 1/d1/\sqrt d learning-rate rescaling does not invalidate Theorem 3.4. On the same constructed instance, it reduces progress toward the minimizer by a factor of d\sqrt d and leaves a larger, dimension-dependent gradient residual. The apparent cancellation of d\sqrt d in the theorem's displayed expression occurs only if the initialization is also changed.

Indeed, following Lemma B.4, define

\qquad b_T := \frac{\eta_{T-1}}{2}(1-\beta_1^T).$$ On $F(x)=\frac{L}{2}\|x\|^2$, the construction starts at $x_0=(S_T+b_T)\mathbf{1}$ and reaches $x_T=b_T\mathbf{1}$. With the same objective and initialization, replacing $\eta_t$ by $\eta_t/\sqrt d$ instead yields $$\widetilde{x}_T= \left[b_T+(1-d^{-1/2})S_T\right]\mathbf{1}.$$ Both trajectories remain in the positive orthant through time $T$, so these expressions follow exactly from the sign recursion. In particular, $$\min_{0\leq t\leq T} \| \nabla F( \widetilde{x}_t) \|_2 = \underbrace{ L \sqrt{d} \, b_T}_{ \displaystyle \min_{0 \leq t \leq T} \| \nabla F(x_t) \|_2 } + L ( \sqrt{d} -1) S_T.$$ For $d>1$, the rescaled schedule therefore produces a strictly larger residual, with an additional dimension-dependent term. It slows progress on this instance rather than removing the difficulty exhibited by the construction. Substituting $\eta_t/\sqrt d$ directly into the theorem's original residual formula implicitly replaces $x_0$ by $x_0/\sqrt d$, reducing the initial objective gap by a factor of $d$. The resulting cancellation therefore compares different initializations; it does not demonstrate improved convergence on the original instance. The theoretical insight concerns the information retained by normalization: in the stated high-signal limiting regime, second-moment normalization produces sign-like updates that discard coordinate-wise gradient magnitudes. Shrinking a scalar learning rate does not recover this information, whereas the stabilized variance normalizer preserves it. Theorem 3.4 makes this distinction concrete through a finite-horizon construction. As explicitly stated in the submission, it does not claim a complexity separation under optimal tuning at a fixed initial objective gap. **Question 1: Is MVN-Grad better than LaProp when their hyperparameters are tuned separately?** Within the reported search spaces, MVN-Grad achieves better selected-configuration results on CIFAR-100 and OpenWebText, while FineWeb results are closely matched. For example, CIFAR-100 test accuracy is 79.94\% versus 77.72\% at batch size 128 and 79.63\% versus 78.40\% at batch size 1024; OpenWebText validation loss is $2.973\pm0.007$ versus $2.990\pm0.010$. Configurations are selected separately for each optimizer under the dataset-specific protocol. This is empirical evidence under the declared tuning searches, not a claim that a finite search identifies globally optimal hyperparameters. The full sweep distributions provide the additional robustness evidence discussed under Weakness 3. **Weakness 2: Novelty beyond combining AdaBelief and LaProp.** Describing MVN-Grad as "LaProp + AdaBelief" captures the simplicity of its implementation, but does not account for the analysis or empirical findings. The paper separates two design choices governing different mechanisms: the normalizer determines what gradient information is retained, while the ordering determines how historical momentum interacts with the current normalizer. The comparison with AdaBelief varies ordering while retaining the residual-based normalizer; the comparison with LaProp varies the normalizer while retaining normalize-then-momentum ordering. Together with Adam, these methods organize the evaluation around the two design axes under common training recipes. The formal comparisons isolate their effects under stated assumptions, and the sweeps evaluate their practical behavior as optimizer parameters vary. Centered RMSProp is acknowledged in the introduction. Accordingly, the contribution is not the invention of centered normalization, post-normalization momentum, or their broad combination, but the residual-based construction, the explicit comparative analysis, and its empirical evaluation. Algebraic simplicity does not remove the value of understanding these interactions.
AuthorsOpenReview ↗

Answer Part 3

Weakness 3:

Empirical benefit. On CIFAR-100 at batch size 1024, MVN-Grad improves median test accuracy across the evaluated sweep from 76.41% to 78.33%, a gain of 1.92 percentage points. Its selected-configuration mean test accuracy also increases from 79.34% to 79.63%, a difference of 0.29 percentage points. Thus, the characterization of the benefits as generally within 0.1% overlooks both the selected-configuration result in this setting and, more importantly, the improvement across hyperparameter choices.

Comparison protocol. Within each benchmark, the training recipe and per-run training budget are matched across optimizers. For CIFAR-100, we use shared learning-rate and momentum search grids, with variance-floor tuning for the variance-normalized methods where specified in Appendix C.5. On OpenWebText, every optimizer is evaluated on the same nine (β1,β2)(\beta_1,\beta_2) pairs at the common learning rate η=10−4\eta=10^{-4}; both variance-normalized methods use εs=0\varepsilon_s=0. Figure 2 therefore reports sensitivity to the evaluated hyperparameters under controlled training conditions, rather than comparing arbitrary training recipes.

The remaining sweep results complement the AdaBelief comparison:

  • CIFAR-100, batch size 1024. The improvement also appears in median training accuracy: 81.65% for MVN-Grad versus 73.87% for AdaBelief. Adam and LaProp attain approximately 74% median test accuracy and 65% median training accuracy.
  • CIFAR-100, batch size 128. MVN-Grad and AdaBelief have comparable median test accuracies, 77.72% and 77.94%, respectively, while MVN-Grad has a shorter low-performance tail. The benefit here is reduced degradation under less favorable configurations.
  • OpenWebText. MVN-GradW and AdaBeliefW have nearly identical median validation losses, 2.9714 and 2.9715, versus approximately 2.9816--2.9818 for AdamW and LaPropW. The normalize-then-momentum methods exhibit narrower interquartile ranges than their momentum-then-normalize counterparts, supporting reduced sensitivity to the evaluated momentum parameters at the common learning rate.

The medians describe performance under a typical evaluated configuration; the interquartile ranges and performance tails characterize sensitivity to tuning. These results capture a practical benefit that selected-configuration means alone cannot establish. Additionally, Table 1 reports selected-configuration improvements over Adam and LaProp of 2.12 and 2.22 percentage points on CIFAR-100 at batch size 128.

This distinction becomes particularly relevant when each hyperparameter trial requires substantial training compute. An optimizer that remains competitive across different parameter choices can reduce the risk of selecting a poor configuration even when the best achieved performance is similar. Our evidence supports this robustness over the reported grids; we do not claim to have measured end-to-end hyperparameter-search savings. The FineWeb experiment serves the complementary purpose of establishing competitiveness at 350M parameters, while Fig. 2 provides the main evidence concerning hyperparameter sensitivity.

Memory and runtime cost. MVN-Grad stores one additional parameter-sized tensor: 50% more optimizer-state storage, or 25% more in the equal-precision subtotal counting parameters, gradients, and optimizer states. Actual total-memory overhead also depends on activations, precision, and sharding. The measured OpenWebText runtime is 46.8k seconds versus 46.2k for AdamW, approximately 1.3% overhead. When the additional state is affordable, retaining competitive performance while reducing sensitivity to tuning choices can therefore constitute a useful trade-off.

Question 2:

Bounded gradients alone do not imply a uniform advantage over Adam. Theorem 3.5 establishes a general nonconvex guarantee for the uncorrected MVN-Grad recursion. With β2=1−1/T\beta_2=1-1/T and the stated learning-rate choice, it gives an O(T−1/2)O(T^{-1/2}) bound on the average expected squared gradient norm. It is a convergence guarantee, not a worst-case dominance result over Adam.

The comparative theoretical contribution lies in the specific mechanisms identified in Theorems 3.1, 3.3, and 3.4 under their respective assumptions. Empirically, the selected-configuration and sweep results provide evidence about performance and hyperparameter sensitivity under the reported training protocols. Worst-case convergence guarantees characterize behavior over the problem class covered by their assumptions; they do not by themselves determine the relative performance of optimizers on a particular training task. Consequently, methods with the same worst-case rate can exhibit substantially different practical convergence behavior and hyperparameter sensitivity. Our mechanism-level analysis and empirical evaluation address these distinctions.

AuthorsOpenReview ↗

Revised manuscript uploaded — changes addressing your comments

Thank you again for your detailed feedback. We have uploaded the revised manuscript, with revisions highlighted in blue. The changes relevant to your comments include:

  • Theoretical interpretation: We clarify that the ordering comparison concerns one-step conditional update variance and distinguish the finite-horizon result for the idealized normalization dynamics from a general complexity comparison between optimally tuned optimizers.
  • Empirical contribution: Section 4.1 now more explicitly distinguishes selected-configuration performance from robustness across the evaluated hyperparameter sweeps, including the 1.92-percentage-point improvement in median test accuracy over AdaBelief at batch size 1024.
  • Memory and runtime: We clarify the additional optimizer-state storage, its relationship to total training memory, and the measured runtime overhead.
  • Algorithmic relationship: Appendix A.1 explicitly compares MVN-Grad with centered RMSProp, specifying the conditions for their special-case equivalence and the corresponding learning-rate scaling.

These revisions make the scope of the theoretical results and the empirical robustness evidence more explicit. Thank you for helping us improve the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.