acceptodds
Under review as a conference paper at ICLR 2027

Momentum Acceleration of Normalized Steepest Descent at the Edge of Stability

Abstract

Optimizers based on normalized steepest descent with momentum are widely used to train large language models, yet existing analyses attribute the benefit of momentum to stochastic noise reduction and favor no momentum in deterministic training. We show that full-batch training nevertheless benefits from momentum and explain why. Once training reaches the Edge of Stability, normalization pins the effective learning rate at twice the inverse curvature transverse to the gradient, so progress is limited by the loss geometry rather than by the learning rate. Momentum attenuates the alternating gradient component before normalization, which raises the threshold at which oscillations start and multiplies the progress after it by . A switched flow built from this mechanism predicts the onset of oscillations, the speed afterward, and, integrated from initialization, the training loss in full-batch experiments. We prove the mechanism on high-dimensional quadratic valleys, including a strict speedup from the same initialization without assuming periodic behavior, and show that on general smooth losses normalized descent follows the flow locally once it oscillates.

Primary area:
optimization
Keywords:
momentum, normalized steepest descent, edge of stability

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bsbKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work investigates why momentum helps normalized optimizers in the deterministic regime (evidenced empirically in Figure 1), while existing theory on noise reduction or acceleration on quadratics does not predict improvement in that setting. This work shows that in a two-dimensional valley loss landscape, the use of normalization creates two regimes of training: first a stability window succeeded by oscillations. The use of momentum both delays the latter and levels up the overall learning rate in both regimes, which leads to improvement. Based on this, the authors introduce the switch flow, which transitions between these two regimes using gradient and sharpness estimates in neural network training. They empirically show that this flow indeed transitions between these two regimes with the predicted increase of the effective learning rate when using momentum. Finally, the authors rigorously prove that NormGD-M follows the switch flow on general losses in the oscillating regime and that, on a quadratic loss, NormGD-M involves these two regimes for almost every initialization and that momentum delays the oscillating regime and increases the effective learning rate in that regime.

Critical strengths

A main strength of that work is the clarity and the flow of the paper. The motivation is very well explained in the introduction: momentum benefits are not currently well explained by either deterministic studies or through the lens of acceleration on quadratics when using normalization, which motivates a clear question on understanding 'why does momentum help normalized optimizers in deterministic training '". Additionally, the structure of the paper is very clear, with a heuristic/theoretical derivation of the switch flow in Section 2, an empirical study of the predictions of that framework in practice on deterministic neural network training, and finally theoretical rigorous results on high-dimensional quadratics further supporting the study.

A second main strength is the novelty of the switch flow, which helps explain why momentum helps the normalized optimizer by evidencing two regimes of training whose transition and levels are dictated by the momentum parameter β\beta.

Critical weaknesses

A weakness of that work is the limited empirical validation of that framework. The experiments are done on an MLP with only 20k parameters and on synthetic data, which is at odds. It would have been very interesting to push that empirical investigation a bit further to test the limits of that framework, especially in more realistic settings with real data.

Additionally, the novelty of the multiplicative factor 1+β1−β\frac{1+\beta}{1-\beta} on the effective learning rate for momentum at the edge of stability is unclear, as it already appeared in [1] in a similar context.

[1] Cohen, Jeremy M., et al. "Adaptive gradient methods at the edge of stability." arXiv preprint arXiv:2207.14484 (2022).

Questions for authors
  1. In Section 2.3, the authors explain that the previous parameter bb from the loss-valley model in equation 2 can be recovered by the gradient b=∥∇L(X)∥2b = \|\nabla L (X)\|_2. However, the gradient ∥∇L(X)∥2\|\nabla L (X)\|_2, when applied to the model in Equation 2, has two components: one from the linear part bb and one from the quadratic gradient, which is not optimal: λu\lambda u. Can the authors explain whether that would introduce a discrepancy between the previous theoretical discussion and the switch flow in Equation 7?

  2. A similar factor Kβ=1+β1−βK_\beta = \frac{1+\beta}{1-\beta} was evidenced in [1] in the context of Adam at the edge of stability. Could the authors comment on the link between their study and that work and the novelty of this factor KβK_{\beta} regarding momentum methods at the edge of stability?

4 comments
AuthorsOpenReview ↗

Response to Reviewer bsbK (Part I)

We thank the reviewer for the constructive suggestions on extending experiments, presentation gaps and highlighting novelty against prior literatures. Below we respond to your questions and weaknesses.

W1. Realistic setting experiments.

We thank the reviewer for this suggestion. We added two settings with real data, a fully connected network on CIFAR-10 and a two-layer Transformer on C4, and tested the predictions of Section 3 on them (Table R1; see Appendices B.7 and B.8 in the revision for details).

Table R1. Predictions of Section 3 on three settings. Each entry holds for every β\beta; the loss error is defined as in Figure 5 (Appendix B.1).

Relative error ofPredictionMLP (Section 3), 20k parametersCIFAR-10, 657k parametersTransformer on C4, 4.5M parameters
η/∥mk∥\eta/\Vert m_k\Vert after the switch2Kβ/λmax⁡2K_\beta/\lambda_{\max}≤ 0.1%≤ 0.2%≤ 0.2%
Speed after the switchEq. (6)≤ 1.5%≤ 5%≤ 12%
LossEq. (7)≤ 2.3%≤ 1.0%≤ 5.1%

Table R2 reports this error over a range of learning rates: in most cases, the switched flow predicts the loss to within a few percent.

Table R2. Relative error of the loss predicted by the switched flow of Eq. (7), over a range of learning rates η\eta.

Settingη\etaβ=0\beta=0β=0.4\beta=0.4β=0.64\beta=0.64β=0.784\beta=0.784β=0.8704\beta=0.8704
CIFAR-100.0010.0010.2%0.1%0.1%0.4%1.0%
0.0020.0020.7%0.3%0.3%0.4%0.9%
0.0040.0042.8%1.3%1.0%1.2%1.6%
0.0080.0085.5%3.5%2.6%2.5%2.9%
Transformer0.0006250.0006250.6%0.8%1.0%1.0%5.0%
0.001250.001251.6%2.7%3.6%4.6%5.1%
0.00250.00254.6%8.0%12%15%18%

We also include the 14.6M-parameter language model of Figure 1 (Appendix B.4) as qualitative evidence at a larger scale.

Q1. Gradient on the switch flow.

We thank the reviewer for pointing out the gap between the discrete iterates and the flow, especially where the gradient is taken. We answer the question in two parts:

Center of the discrete iterations. At an individual iterate at Eq. (2), the gradient is (λuk,b)(\lambda u_k,b), whose norm exceeds bb during the oscillation. On the period-two orbit, the buffer is mk=(±μ,b)m_k=(\pm\mu,b), where μ\mu is its sharp component: μ\mu flips sign at every update, so it cancels from the two-step average, while ∥mk∥\Vert m_k\Vert stays fixed. The center xˉk=(xk+xk+1)/2\bar x_k=(x_k+x_{k+1})/2 therefore moves by

xˉk+1−xˉk=−η∥mk∥ (0,b),η∥mk∥=2Kβλ,\bar x_{k+1}-\bar x_k=-\frac{\eta}{\Vert m_k\Vert}\,(0,b),\qquad\frac{\eta}{\Vert m_k\Vert}=\frac{2K_\beta}{\lambda},

where the second identity is the period-two condition, which holds exactly on the period-two orbit: the transverse multiplier 1−ηλ/(Kβ∥mk∥)1-\eta\lambda/(K_\beta\Vert m_k\Vert) must equal −1-1. The norm λ2uk2+b2\sqrt{\lambda^2u_k^2+b^2} thus enters only the normalizer, and the progress is carried by bb alone (see Appendix B.6 in the revision for an illustration).

From center to switch-flow. We introduce the continuous flow in order to capture the movement of the oscillation center in the training process. In our formulation in Eq. (7), the gradient is evaluated on the flow point itself. This is valid for two reasons:

  • Eq. (7) is a rescaled gradient flow which decreases the loss monotonically and does not oscillate.
  • Along the flow, the transverse component decays (u˙∝−λu\dot u\propto-\lambda u in the valley). Therefore the flow remains close to the valley floor where the gradient is nearly orthogonal to the sharp direction and captures the role of (0,b)(0,b) in the 2D-quadratic model.

We verify the second point empirically on our teacher–student task: along the gradient-flow path, the cosine between the gradient and the top Hessian eigenvector remains negligible, below 3×10−43\times10^{-4} in absolute value (see Appendix B.6 in the revision). Since the ∥∇L∥\Vert\nabla L\Vert in Eq. (7) is evaluated at the center, where it equals bb, Eq. (7) gives the progress speed min⁡{η,2Kβb/λ}\min\lbrace\eta,2K_\beta b/\lambda\rbrace, which matches the displacement per update in Eq. (6). Hence there is no discrepancy; we now state explicitly in Section 2.3 of the revision that the gradient is evaluated at the center.

AuthorsOpenReview ↗

Response to Reviewer bsbK (Part II)

W2 & Q2. The factor KβK_\beta.

The factor KβK_\beta is known. We agree that the same stability threshold with momentum has already been studied in the literature for different settings. Throughout, we write heavy-ball momentum in the EMA form, mk+1=βmk+(1−β)∇L(xk)m_{k+1}=\beta m_k+(1-\beta)\nabla L(x_k), as in our paper. For example:

  • Polyak (1964); Goh (2017): on a quadratic, heavy-ball momentum is stable if and only if ηλ<2Kβ\eta\lambda<2K_\beta.
  • Cohen et al. (2021): gradient descent with heavy-ball momentum diverges on a quadratic once the sharpness exceeds 2Kβ/η2K_\beta/\eta (their Appendix B), and in neural network training the sharpness rises to this threshold and plateaus there (see Figure 7 and Appendix N of their paper).
  • Cohen et al. (2022) [1]: for full-batch Adam, the preconditioned sharpness equilibrates near 2Kβ/η2K_\beta/\eta, derived with the preconditioner held fixed.
  • Lu et al. (2026): on river-valley losses, gradient descent with heavy-ball momentum tolerates learning rates up to about 2Kβ/λmax⁡2K_\beta/\lambda_{\max}, that is, (1+β)/(1−β)(1+\beta)/(1-\beta) times the threshold of plain gradient descent. At a fixed learning rate, its speed along the valley barely depends on β\beta, so the acceleration comes primarily from the larger admissible learning rate.

We remark that the appearance of the same (1+β)/(1−β)(1+\beta)/(1-\beta) in these stability thresholds over sharpness and learning rates is rooted in the same linearization: about straight motion along the valley floor, the transverse dynamics of NormGD-M follow the heavy-ball recurrence with effective learning rate η/b\eta/b (Appendix E). And we do not claim novelty over identifying this specific stability threshold.

Acceleration of training progress. Instead, our finding that momentum can accelerate the optimization by a factor of (1+β)/(1−β)(1+\beta)/(1-\beta) for the same learning rate is novel. Specifically, we show that when both have entered the oscillation phase, NormGD-M moves KβK_\beta times faster than NormGD at the same learning rate (Eq. (6); Theorem 4.2; Figure 4). Indeed, this is a joint effect of normalization and momentum. Without normalization, gradient descent with momentum moves ηb\eta b per update for every β\beta, and momentum only raises the threshold (Lu et al., 2026). To our knowledge, this acceleration at a fixed learning rate has not been characterized before. Moreover, based on this characterization of the oscillation phase, we are able to derive a unified switch-flow framework that describes the whole training dynamics (Eq. (7); Figure 5).

I thank the authors for their detailed answer to my questions.

The additional experiments strengthen the theoretical results in particular using transformers and real data experiments. I appreciate the discussion on where the gradient is taken between the discrete updates and the flow.

I also appreciate the discussion on the novelty of the factor KβK_\beta in particular the emphasis on the acceleration of training progress.

I will keep my score and recommend acceptance, as the paper's results are strong and clear. Reasons for not a higher score are the novelty and significance of the results.

Thank you for taking the time to read our response. We greatly appreciate your help to improve the quality of our work, especially on your questions about the the presentation gap and the suggestions to improve experiments.

Reviewer_X5zLOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why momentum can accelerate normalized steepest descent methods even in deterministic full-batch training, where its benefits cannot be explained solely by stochastic gradient noise reduction. The authors propose a geometric explanation based on the Edge of Stability (EoS) phenomenon, in which normalized optimization dynamics develop oscillations along sharp directions of the loss landscape. The central contribution is a switched-flow model that explains two complementary effects of momentum. First, momentum attenuates the alternating component of the gradient before normalization, extending the stable optimization phase and postponing the onset of oscillations. Second, once oscillations are established, momentum increases the progress of the trajectory center by a factor related to (K_\beta=(1+\beta)/(1-\beta)). The paper provides theoretical analyses for high-dimensional open quadratic valleys and establishes local flow-tracking results for general smooth losses under specific geometric assumptions. The authors validate their predictions through experiments on a synthetic teacher–student multilayer perceptron and a decoder-only Transformer trained on a fixed subset of C4. Additional experiments investigate Signum and Muon. The results support the proposed explanations of delayed oscillation, accelerated progress after the transition, and training-loss prediction. Overall, the paper offers a potentially useful theoretical perspective on momentum in normalized optimization. Its main strengths are the explicit mechanism connecting gradient oscillations to optimization speed and the combination of mathematical analysis with empirical validation. However, the theoretical guarantees for general smooth losses remain conditional, particularly regarding the transition from initialization to an established oscillatory regime. The empirical evaluation also leaves room to investigate the robustness of the findings across independent initializations and a broader range of training settings.

Critical strengths
  1. A mechanistic explanation of momentum acceleration in deterministic normalized optimization. The paper identifies a specific interaction between momentum, alternating gradient components, and normalization to explain why momentum can improve optimization even without stochastic gradient noise. The proposed mechanism leads to explicit predictions about the stability threshold and the speed of the trajectory center after oscillations emerge. This provides more insight than an empirical observation that momentum improves training loss and offers a useful framework for understanding the dynamics of normalized optimizers.
  2. A combination of theoretical analysis and empirical validation. The authors connect their proposed mechanism to mathematical results on high-dimensional open quadratic losses and local flow-tracking results for general smooth losses. They further examine the predicted dynamics through controlled neural-network experiments and experiments involving Signum and Muon. This combination of analysis and empirical investigation strengthens the paper by testing several distinct implications of the proposed framework rather than relying on a single performance comparison.
Critical weaknesses
  1. Incomplete theoretical characterization of the transition into the oscillatory regime. The theoretical guarantees for general smooth losses primarily describe the dynamics once oscillations have already been established and under specific assumptions about the loss geometry. They do not fully characterize how the optimizer enters the relevant oscillatory regime from a standard initialization with a zero momentum buffer. This distinction is important because the proposed explanation is intended to account for the training dynamics leading up to and following the transition. A clearer characterization of the sufficient conditions, together with a more explicit separation between established results and unresolved questions, would strengthen the theoretical contribution.
  2. Limited evidence regarding empirical robustness and generality. The empirical evaluation provides useful evidence under controlled full-batch training conditions, but the principal experiments use a synthetic teacher–student MLP and a Transformer trained on a fixed subset of C4. The reported setup does not fully establish the sensitivity of the observed effects to independent initializations or broader model and data configurations. In particular, it remains unclear how consistently the predicted transition threshold and post-transition acceleration hold when the loss geometry and training conditions vary. Additional robustness analyses would help establish the practical scope of the proposed mechanism.
Questions for authors
  1. Theoretical scope and transition into oscillations. The general smooth-loss flow-tracking result assumes that the optimizer is already in an established oscillatory regime. Can the authors clarify the sufficient conditions under which the dynamics enter this regime from a zero momentum buffer? If a complete characterization is currently unavailable, could the authors more explicitly distinguish the proven local flow-tracking and acceleration results from the unresolved transition dynamics?
  2. Robustness across initializations and training settings. The main experiments provide evidence consistent with the proposed mechanism under controlled full-batch training conditions. Can the authors report results across multiple independent initializations, particularly for the predicted switching time, post-switch progress, and loss-prediction error? Such an analysis would help determine whether the observed behavior is robust rather than dependent on the selected initialization and experimental configuration.
3 comments
AuthorsOpenReview ↗

Response to Reviewer X5zL

We thank the reviewer for pointing out the lack of justification for oscillation establishment and robustness test for experiments. We address the specific questions below.

W1 & Q1. Zero-buffer initialization and transition.

Currently, our analysis does not cover sufficient conditions for NormGD-M to enter the period-two oscillation from a zero buffer. This difficulty already appears on quadratic losses, where the curvature is fixed. We separate the established results from what remains open in our discussion below.

Acceleration on quadratic losses. On quadratic losses with sufficiently large curvature, NormGD provably settles into the period-two oscillation regime for almost every initialization (Theorem 4.2). From a zero buffer, NormGD-M is strictly faster than NormGD whenever NormGD oscillates, without any assumption on periodicity (Theorem 4.3). The exact KβK_\beta factor is established near the period-two oscillation (Theorem 4.2).

For oscillation establishment, currently we are only able to prove that a constructed period-two oscillation orbit is locally attractive: NormGD-M converges to it when both the position and the buffer start close to it (Theorem E.4). We illustrate the reasons below.

Difficulty in period-two oscillation and transition phase. The transition from near-valley initialization to stable period-two oscillation is hard to analyze. Moreover, NormGD-M need not oscillate with period two, even on quadratic losses. We demonstrate this hardness through a hyperparameter search over 2D open quadratic losses (see Appendix E in the revision for details). Specifically, we fix the sharpness λ\lambda, the gradient bb along the valley and the initialization near the valley floor, search over the learning rate η\eta and momentum decay rate β\beta, and start the iteration from a zero buffer. Our experiment shows that the oscillation period can be very sensitive to the hyperparameters, especially in the practical regime where 1−β1-\beta is small: as η\eta varies, the iterates end in oscillations that repeat every 2, 4, 6 or 12 updates, in an irregular pattern. The period also changes with the initialization at the same η\eta and β\beta, and at some parameters two of these oscillations are both stable. Hence, for such η\eta and β\beta, no condition on η\eta, β\beta and the sharpness alone can guarantee the period-two oscillation.

Empirical evidence in neural networks. Empirically, the period-two oscillation is widely observed. On the teacher–student MLP, in the runs of Section 3, the iterates oscillate in a period-two pattern and the effective learning rate η/∥mk∥\eta/\Vert m_k\Vert approximates 2Kβ/λmax⁡2K_\beta/\lambda_{\max} closely (Figure 3(c)), as predicted by our analysis. We added this separation to Section 4 in the revision, and leave analysis for the transitioning phase for future work.

W2 & Q2. Robustness of experiments.

We thank the reviewer for this suggestion. We repeated the experiments of Section 3 from four more initializations. Table R1 reports the relative error of each prediction, as mean ± standard deviation over the five initializations (see Appendix B.5 in the revision for details).

Table R1. Teacher–student MLP at η=0.008\eta=0.008. Relative error of the switch-flow prediction.

β\betaSwitch time (flow vs. run)Speed-up over NormGD (vs. KβK_\beta)Loss (flow vs. run, as in Figure 5)
012.1 ± 0.8%–0.57 ± 0.04%
0.45.2 ± 0.4%0.32 ± 0.05%0.68 ± 0.08%
0.643.5 ± 0.2%0.46 ± 0.10%0.71 ± 0.07%
0.7842.0 ± 0.1%0.78 ± 0.19%0.81 ± 0.03%
0.87041.8 ± 0.1%1.57 ± 0.30%1.18 ± 0.46%
0.92222.2 ± 0.5%3.44 ± 0.48%2.89 ± 0.83%

The error stays small at the other learning rates of the grid (see Appendix B.5 in the revision). On two further settings with real data, a Transformer trained on C4 and a fully connected network on CIFAR-10, the predictions also hold at various learning rates (see Appendices B.7 and B.8 in the revision).

Thank you for clarifying the scope of the theoretical results and for reporting results across five initializations. I will keep my original rating.

Thank you for following up and for encouraging us to clarify the scope of our theoretical results and test the predictions across multiple initializations, which greatly helped to improve the completeness of our work!

Reviewer_8dNFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why momentum improves deterministic, full-batch normalized optimization. Its proposed mechanism is that momentum attenuates the alternating gradient component along a sharp direction before normalization, preserving more of the update for progress along the valley.

The authors analyze open quadratic valleys, and establish both local exact dynamics and a weaker zero-buffer acceleration guarantee without assuming periodic behavior. They also prove local flow tracking and conditional path-traversal comparisons for general losses under restrictive geometric assumptions. Full-batch teacher–student experiments quantitatively test a switched-flow model; Signum, Muon, and small-transformer experiments provide additional qualitative evidence.

Critical strengths

A coherent explanation of deterministic momentum benefits. The paper explains why averaging an alternating sharp component can improve a normalized update even when there is no minibatch sampling noise. The same mechanism predicts both delayed instability and faster progress during persistent oscillation.

Critical weaknesses
  1. The general-loss theorem’s assumptions are not connected to the neural-network experiments. Assumption F.1 requires strictly positive curvature in every direction normal to the reference gradient-flow path. The exact local cycle analysis also requires a simple top eigenvalue and a spectral gap. Section 4.1 explicitly tests the conclusion experimentally rather than checking these sufficient conditions.

  2. The exact acceleration law has narrower support than the broader optimizer interpretation. The precise KβK_\beta factor is established near a specified position–buffer period-two orbit. Zero-buffer results are weaker, and the general-loss comparison starts from a prepared position with slowly varying geometry and sufficient path length to amortize startup costs. The exact factor is therefore not a training-wide finite-budget guarantee.

Questions for authors
  1. Clarification of the theory–experiment connection. Is the softmax common-logit-shift freedom explicitly removed, or is the theory intended to apply on an invariant subspace? After that reduction, what evidence or reasoning supports the remaining normal-positivity and spectral assumptions? Clarifying the intended relationship is more important here than checking every conservative sufficient constant.

  2. Clarification of claim scope and non-Euclidean evidence. Can the authors explicitly separate the exact local KβK_\beta law, the weaker zero-buffer comparisons, and empirical flow prediction from initialization? For Signum and Muon, which aspects of the proposed mechanism are directly supported, given the different threshold shifts and flip behavior? A new general non-Euclidean theorem is not required, but the limits of the explanation should be clear.

2 comments
AuthorsOpenReview ↗

Response to reviewer 8dNF (Part I)

We thank the reviewer for the careful review and helpful suggestions on organizing results from different levels. We address the specific questions below.

W1 & Q1. Theory-experiments connections.

We acknowledge that current manuscript does not highlight the connections between theory and experiments, especially the conditions which practical neural nets might fail to satisfy. We clarify these issues as follows:

Logit shift and invariant subspace. The softmax logit shift spans a linear subspace SS on which the loss LL is exactly invariant. Hence ∇L⊥S\nabla L \perp S throughout the training process, and the momentum buffer and NormGD-M updates stay in S⊥S^\perp. Therefore, NormGD-M on LL coincides step by step with NormGD-M on the reduced loss L~(y)=L(x0+Py)\tilde L(y)=L(x_0+Py), where the columns of PP form an orthonormal basis of S⊥S^\perp. We can therefore apply our theory to L~\tilde L.

Spectral assumptions on real neural nets. For curvature assumptions, we verified empirically that the top eigenvalue within our training horizon remains simple (median λ2/λ1\lambda_2/\lambda_1 at most 0.970.97 for β≤0.8704\beta\le0.8704 and 0.9950.995 for β=0.9222\beta=0.9222; see Appendix B.6 in the revision). We acknowledge that negative curvature directions, together with many near-zero curvature directions, exist in the real network training process including the oscillation regime. However, their magnitude is small compared to top eigenvalues. Along our teacher–student MLP training trajectories, the most negative Hessian eigenvalue is dominated by the top eigenvalue by (∣λmin⁡∣/λ1|\lambda_{\min}|/\lambda_1 between 10−510^{-5} and 2×10−32\times10^{-3}). These directions are essentially negligible because the gradient almost vanishes in these directions and the corresponding loss curvature is nearly zero. Extending the analysis to a normal space that includes negative curvature direction remains challenging and is left for future exploration.

W2 & Q2. Conditional result and global transition.

We agree that exact KβK_\beta factor remains a conditional result given the establishment of period-two oscillations. We thank the reviewer for the suggestion on clarifying the structure of the results. Below, we separate three levels of results in the manuscript as follows:

(i) Exact conditional law. Near the period-two orbit of position and buffer, NormGD-M moves KβK_\beta times farther per update than NormGD (Theorem 4.2; Propositions F.5 and F.6 for general losses).

(ii) Zero-buffer comparison. From a zero-buffer initialization, we only prove that NormGD-M is strictly faster than NormGD, by a factor that is smaller than KβK_\beta (Theorem 4.3; Corollary F.12(b)). This comparison holds from every initialization on quadratic losses, and on general losses from every start near the valley floor, where NormGD oscillates and the geometry changes slowly (Theorem F.10(b)); in both cases it assumes no periodic behavior. We restrict the general-loss statement to initializations near the valley, since a fully global statement would depend on the landscape far from gradient flow trajectories. We remark that an exact acceleration with factor KβK_\beta depends on period-two oscillation.

(iii) Flow prediction from initialization. This is an empirical result showing that our proposed switch flow can describe the actual optimization dynamics (Figure 5; see Appendices B.5, B.7, and B.8 in the revision). We remark that global theoretical guarantees for flow prediction are difficult due to the following reasons:

  1. The dynamics around the establishment of oscillation, or switch, are hard to characterize.
  2. Similar to the hardness above, the oscillation period is sensitive to the input conditions and currently we don't have a sufficient condition over the input conditions for period-two oscillation (see Appendix E in the revision).

Still, we believe that our formulation of the switch flow provides insights for understanding the normalized optimizers with momentum. Also, our extrapolation of this conditional law to a switched flow over the whole run follows the common approach in the literature. For example:

  • The central flows in Cohen et al. (2025) are derived by informal reasoning without a proof, and are supported only by how well their predictions match experiments.
  • Damian et al. (2023) prove that gradient descent follows their constrained trajectory only locally and up to a finite time, under assumptions that they check numerically, and state that their analysis is not sufficient to prove global convergence.

Our results are at a comparable level: the tracking is proved on general losses once the oscillation is established (Theorem 4.1), the acceleration is proved exactly on quadratic losses (Theorems 4.2 and 4.3), and the prediction from initialization is tested empirically, as in these works. It remains a challenging open problem how to make such flow-based descriptions rigorous characterization of the global dynamics.

AuthorsOpenReview ↗

Response to reviewer 8dNF (Part II)

Q2. Extension to Signum and Muon.

We summarize our experimental results for Signum and Muon. In short, qualitative accelerations and oscillation deferrals generalize to Signum and Muon (Appendix B.3, Figure 6). However, the quantitative acceleration factor and flow tracking do not fully generalize from the Euclidean to the non-Euclidean setting.

Result 1: Momentum also accelerates non-Euclidean steepest descent. Both optimizers first make stable progress and then start to oscillate, and a larger β\beta postpones the switch (Figure 6(b,f)). At the same learning rate, momentum also improves the loss within the same number of updates (Figure 6(a,d,e,h)).

Result 2: non-Euclidean steepest descent also exhibits switch and oscillation behavior. The flip ratio of both Signum and Muon rises, though more gradually compared to NormGD-M. We remark that this can be due to the non-smooth nature of ℓ∞\ell^\infty and spectral norm. A simple example is the SignGD over an isotropic quadratic loss, with each coordinate initialized at different scale. Then each coordinate starts oscillating at different time, potentially causing this gradually increasing flip ratio phenomenon.

Progression speed after the switch. The quantitative acceleration factor KβK_\beta does not generalize to Signum and Muon. Our argument on NormGD-M flat-direction speed relies on projections and orthogonal decompositions, which are consequences of ℓ2\ell^2 geometry. In general, non-Euclidean descent in EoS might have complicated behavior. For example, the Signum dynamic might stop when all coordinates are oscillating in a period-two pattern, and the loss stops descending. Understanding the progress mode in the edge of stability for non-Euclidean descents requires further exploration.

We added this comparison to Appendix B.3 in the revision and stated explicitly that the exact speed law and the flow prediction are claimed only for NormGD-M. A theory for non-Euclidean normalization is left for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.