acceptodds
Under review as a conference paper at ICLR 2027

Geometric-Steering Muon for Online Large Language Model Fine-Tuning

Abstract

Fine-tuning is a fundamental approach for adapting pretrained large language models (LLMs) to downstream tasks and evolving application requirements. Although Muon has recently attracted increasing attention as a matrix-structured optimizer for LLM training and fine-tuning, existing studies have largely focused on conventional offline settings, leaving its behavior under online adaptation comparatively underexplored. In online LLM fine-tuning, evolving objectives can render accumulated momentum increasingly inconsistent with the current optimization signal, yet Muon's Newton–Schulz (NS) transformation primarily reshapes the spectral structure inherited from momentum without explicitly adapting its update orientation to the current objective. We therefore propose Geometric-Steering Muon (GS-Muon), which introduces Geometric-Steering Newton–Schulz (GS-NS) to explicitly couple Muon's spectral transformation with the current optimization signal. GS-NS decouples orientation adaptation from spectral orthogonalization: a short NS prefix establishes a near-orthogonal geometry, a tangent-inspired steering step redirects the momentum-inherited orientation toward the current optimization signal, and a final NS correction restores the orthogonalization structure. Theoretically, we establish geometric stability properties of GS-NS and derive variation-dependent nonconvex regret guarantees for GS-Muon under both first-order and zeroth-order signal access, explicitly characterizing the effects of momentum tracking and zeroth-order signal estimation. Finally, we establish a unified streaming evaluation protocol across two model scales and eleven NLP datasets, covering single-dataset streams, sequential within-family task shifts, and heterogeneous cross-task streams.

Primary area:
optimization
Keywords:
Online learning, Muon, Newton–Schulz

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QVVKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to improve Muon optimizer in the online learning setting. The authors notice that the momentum of vanilla Muon may conflict with the current gradient and therefore cannot adaptively adjust its behavior for both history smoothing and task switching. Then, they propose to inject the current gradient signal into Muon's momentum by geometrically steering, which can be viewed as finding a minor direction close to the current gradient in the tagent space of the momentum matrix in the Stiefel manifold to preserve the orthogonality. Finally, they propose a 3NS-steering-1NS framework, with solid theoretical results. Experimental results show that the proposed optimizer improve the final performance compared with Muon.

Critical strengths
  1. The paper identifies a meaningful limitation of Muon in non-stationary online learning.

  2. GS-Muon outperforms standard Muon across various tasks, achieving better average performance both in zeroth-order optimization and task-switching scenarios.

  3. The paper provides geometric stability properties and nonconvex convergence guarantees for both FO and ZO settings.

Critical weaknesses
  1. The paper does not compare geometric steering with simpler alternatives, such as gradient mixing or momentum adjustment. Consequently, the necessity of the proposed geometric formulation remains unclear. The authors are encouraged to provide some theoretical or empirical evidences to support the benefits of the proposed geometric steering method.

  2. The noticed limitation of Muon can also be addressed by introduing an adaptive memory coefficient in the computation of Muon's momentum, which adaptively adjusts how much we smooth the history and how much we switch the task-related direction. Can geometrical steering empirically surpass the counterpart?

Questions for authors

See above.

9 comments
AuthorsOpenReview ↗

Response to Reviewer QVVK-Part 1

Thank you for the constructive comments. We address each weakness and question separately below; all supplementary tables are collected in the shared Official Comment.

Response to W1

We thank the reviewer for this important suggestion. We agree that demonstrating improvements over standard Muon alone is insufficient to establish the necessity of geometric steering. To address this concern, we provide complementary theoretical and empirical evidence showing that GS-Muon offers a distinct advantage over simpler momentum-based alternatives.

W1.1. Geometric Motivation and Theoretical Advantages

1. Unlike direct gradient mixing, geometric steering adapts the update orientation without sacrificing the historical information accumulated by momentum.

Standard Muon accumulates historical gradients through

Mt=βMt−1+gt.M_t=\beta M_{t-1}+g_t.

A straightforward alternative is to directly mix the accumulated momentum with the current gradient:

Mˉt=(1−λ)Mt+λgt=gt+(1−λ)βMt−1.\bar M_t=(1-\lambda)M_t+\lambda g_t=g_t+(1-\lambda)\beta M_{t-1}.

As λ\lambda increases, the historical contribution to the current NS input decreases. In particular, λ=1\lambda=1 completely removes the historical component, producing a gradient-only NS update. Although increasing λ\lambda can improve responsiveness to the current optimization signal, it simultaneously weakens the historical information used by the update. This creates a trade-off between current-gradient responsiveness and momentum-based temporal smoothing, which is particularly relevant in online learning under stochastic feedback and distribution shifts.

In contrast, GS-Muon retains the original accumulated momentum and first constructs the momentum-derived NS prefix Xt,3X_{t,3}. It then introduces a tangent-inspired steering correction:

Tt=Ht−Xt,3sym⁡(Xt,3⊤Ht),T_t=H_t-X_{t,3}\operatorname{sym}(X_{t,3}^{\top}H_t),

where HtH_t is the normalized current optimization signal.

Let St=sym⁡(Xt,3⊤Ht)S_t=\operatorname{sym}(X_{t,3}^{\top}H_t) and Rt=I−Xt,3⊤Xt,3R_t=I-X_{t,3}^{\top}X_{t,3}. As established in Lemma B.3, the steering direction satisfies

⟨Ht,Tt⟩F=∥Ht∥F2−∥St∥F2≥0,\langle H_t,T_t\rangle_F=\lVert H_t\rVert_F^2-\lVert S_t\rVert_F^2\geq0,

and

∥Xt,3⊤Tt+Tt⊤Xt,3∥F≤2∥Rt∥op∥St∥F.\lVert X_{t,3}^{\top}T_t+T_t^{\top}X_{t,3}\rVert_F\leq2\lVert R_t\rVert_{\mathrm{op}}\lVert S_t\rVert_F.

The first inequality guarantees nonnegative alignment between the steering direction and the current optimization signal. The second shows that its FO effect on the Gram matrix is controlled by the orthogonality residual of the NS prefix.

2. A new theoretical comparison provides sufficient conditions under which the complete GS-Muon update achieves better gradient alignment than standard Muon.

We introduce a new corollary in the Corollary C.1 of revised manuscript, as detailed in our response to Reviewer Lz3j (W1.2). The analysis directly compares the complete GS-Muon update with standard five-step Muon under identical momentum and current-gradient inputs. By explicitly accounting for the positive steering contribution and the residual terms introduced by the final NS transformation, we establish a sufficient condition M>0\mathcal M>0 such that

⟨H,DGS⟩F>⟨H,DMuon⟩F.\langle H,D^{\mathrm{GS}}\rangle_F>\langle H,D^{\mathrm{Muon}}\rangle_F.

The analysis identifies a nondegenerate near-orthogonal regime in which the sufficient condition is satisfied. This result provides direct theoretical evidence that geometric steering can improve the alignment of the complete final update with the current optimization signal, rather than merely modifying an intermediate NS representation. We measure the actual alignment gain ΔM=⟨H,DGS−DMuon⟩F\Delta_{\mathcal M}=\langle H,D^{\mathrm{GS}}-D^{\mathrm{Muon}}\rangle_F in *Table G of the shared Official Comment.

W1.2. Empirical Comparisons with Gradient Mixing and Momentum Adjustment

Systematic gradient-mixing experiments demonstrate that simply increasing the current-gradient contribution cannot reproduce the performance gains of GS-Muon. To directly evaluate whether the benefits of GS-Muon can be explained by simpler historical-gradient reweighting, we conduct additional FO experiments on Qwen2.5-1.5B across all eleven single-dataset online streams. Specifically, we evaluate direct gradient–momentum mixing with

λ∈{0,0.1,0.2,…,1.0},\lambda\in\lbrace 0,0.1,0.2,\ldots,1.0\rbrace,

To isolate the effect of the optimization mechanism, all methods use identical learning rates, momentum coefficients, online data streams. Only the specified mixing or steering mechanism is changed. As shown in Table A of the shared Official Comment, GS-Muon achieves superior overall performance compared with all evaluated gradient-mixing configurations. This observation supports the proposed motivation: the advantage of GS-Muon does not arise solely from assigning greater weight to the current gradient, but from explicitly incorporating the current optimization signal into the geometry of the momentum-derived update.

AuthorsOpenReview ↗

Response to Reviewer QVVK-Part 2

Response to W2

We thank the reviewer for this insightful suggestion. We agree that adaptively adjusting the momentum coefficient is a meaningful alternative for balancing historical information and responsiveness to the current optimization signal. To directly evaluate this possibility, we introduce an adaptive-momentum Muon baseline and compare it with GS-Muon. Importantly, although both approaches aim to improve adaptation under changing objectives, they operate through fundamentally different mechanisms: adaptive momentum modifies the accumulation of historical gradients, whereas geometric steering explicitly corrects the orientation of the momentum-derived update.

1. We introduce a gradient-alignment-based adaptive momentum rule that dynamically adjusts historical memory without requiring additional gradient computations or objective evaluations. Specifically, we measure the cosine similarity between the current optimization signal and the previously accumulated momentum:

γt=⟨gt,Mt−1AM⟩F∥gt∥F∥Mt−1AM∥F.\gamma_t=\frac{\langle g_t,M_{t-1}^{\mathrm{AM}}\rangle_F}{\lVert g_t\rVert_F\lVert M_{t-1}^{\mathrm{AM}}\rVert_F}.

Based on this alignment, we adapt the momentum coefficient as

βt=βmax⁡1+γt2,MtAM=βtMt−1AM+gt,\beta_t=\beta_{\max}\frac{1+\gamma_t}{2},\qquad M_t^{\mathrm{AM}}=\beta_tM_{t-1}^{\mathrm{AM}}+g_t,

where βmax⁡\beta_{\max} controls the maximum historical contribution. The resulting momentum is subsequently processed using the standard five-step Newton–Schulz transformation. This design adapts the historical weighting according to the relationship between the current and previous optimization signals. When the two directions are well aligned, βt\beta_t approaches βmax⁡\beta_{\max}, preserving momentum-based historical smoothing. Conversely, when their directions conflict, βt\beta_t decreases, reducing the influence of outdated historical information and allowing the update to respond more strongly to the current objective. For numerical stability, the cosine similarity is clipped to [−1,1][-1,1]. When either input norm is zero, we set βt=βmax⁡\beta_t=\beta_{\max}. The similarity is computed over the concatenated trainable parameter blocks. The adaptive coefficient is computed directly from the existing optimization signal, without introducing additional training queries or requiring task-boundary information.

2. We directly compare adaptive-momentum Muon with GS-Muon to determine whether dynamically adjusting historical weighting can reproduce the benefits of geometric steering. We evaluate the adaptive-momentum baseline on all eleven Qwen2.5-1.5B single-dataset online streams. For adaptive momentum, we consider

βmax⁡∈{0.8,0.9,0.95},\beta_{\max}\in\lbrace 0.8,0.9,0.95\rbrace,

with the learning rate selected under a comparable hyperparameter-tuning budget. All methods share the same model checkpoints, trainable parameters, online data streams, and evaluation protocol. The baseline employs standard five-step Muon after updating its adaptive momentum, ensuring that the comparison isolates the effect of momentum adaptation. The results are reported in Table A of the shared Official Comment.

AuthorsOpenReview ↗

With Appreciation for Your Feedback and Consideration

Dear Reviewer QVVK,

We sincerely appreciate your thoughtful comments and suggestions. In response, we have carefully addressed the points you raised and provided further explanations.

With the discussion period drawing to a close, we would be grateful if you could review our responses when convenient and share any remaining questions or concerns. We would be pleased to offer additional clarification if needed.

Thank you once again for your time and valuable feedback.

Reviewer_XmYSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GS-Muon for online LLM fine-tuning. It inserts a gradient-guided steering step into Muon’s Newton–Schulz transformation to address momentum lag under changing objectives. The paper provides geometric properties and nonconvex stationarity bounds under first-order and zeroth-order access, together with experiments across two model scales, eleven datasets, and task-shift streams.

Critical strengths

1.The method addresses a clear problem: historical momentum can become misaligned with the current objective in nonstationary environments. 2.The evaluation covers diverse tasks and reports promising gains. On Qwen2.5-1.5B, average first-order performance improves from 70.7 to 72.9, and task-shift performance improves from 63.4 to 67.0.

Critical weaknesses

1.The specific benefit of geometric steering is not isolated. Comparisons with reduced or reset momentum and direct gradient–momentum mixing are missing. AdamW is also absent from the first-order comparisons. The current ablations do not exclude simpler explanations for the gains. 2.The paper does not report multiple seeds or uncertainty estimates. Although Appendix A.1 specifies the GS-Muon hyperparameter search grids and states that comparable tuning budgets are used, the selection metric, tuning-data splits, and exact baseline search configurations remain unclear. Precise dataset splits, the exact trainable parameter set, and measured runtime and memory usage also need clarification. 3. The bounds allow α = 0 and become looser as α increases. Moreover, the residual contraction in Lemma B.4 of Appendix B.2 requires a local condition that the algorithm does not automatically guarantee. Its empirical applicability should be demonstrated.

Questions for authors

1.Under equal tuning budgets, how does GS-Muon compare with reduced/reset momentum and direct gradient mixing? 2.Can the authors provide repeated-run results, precise data and tuning splits, the hyperparameter-selection criterion, and the missing first-order results for Llama-3-8B? 3.Can the authors measure orientation mismatch, gradient alignment, and orthogonality residuals around task transitions to validate the proposed mechanism?

7 comments
AuthorsOpenReview ↗

Response to Reviewer XmYS-Part 1

Thank you for the constructive comments. We address each weakness and question separately below; all supplementary tables are collected in the shared Official Comment.

Response to W1&Q1

We thank the reviewer for this important comment. To better isolate the contribution of geometric steering, we provide additional theoretical and empirical comparisons with simpler momentum-based alternatives and extend our first-order evaluation to AdamW. In particular, our analysis highlights that GS-Muon preserves momentum-based historical smoothing while introducing a geometrically constrained correction toward the current gradient, whereas direct gradient mixing improves responsiveness by reducing historical contributions.

W1.1: Comparisons with Reduced/Reset Momentum and Direct Gradient Mixing

1) Geometric steering preserves historical momentum while explicitly adapting the update orientation, unlike direct mixing, which progressively suppresses historical information. As detailed in our response to Reviewer Lz3j (W1.1), standard Muon accumulates historical gradients through

Mt=βMt−1+gt.M_t=\beta M_{t-1}+g_t.

Direct gradient–momentum mixing instead constructs

Mˉt=(1−λ)Mt+λgt=gt+(1−λ)βMt−1.\bar M_t=(1-\lambda)M_t+\lambda g_t=g_t+(1-\lambda)\beta M_{t-1}.

Therefore, increasing λ\lambda reduces the contribution of historical momentum to the current NS input. In particular, λ=1\lambda=1 yields Mˉt=gt\bar M_t=g_t, corresponding to a gradient-only NS transformation that removes historical information from the current update.

Although this endpoint retains Muon's matrix-structured NS transformation, it no longer benefits from momentum-based temporal smoothing. This distinction is important in online optimization, where historical information can help suppress stochastic-gradient fluctuations while the objective evolves.

In contrast, GS-Muon retains the original accumulated momentum and constructs a tangent-inspired correction within the NS transformation:

T=H−Xsym⁡(X⊤H),T=H-X\operatorname{sym}(X^\top H),

where XX is the momentum-derived NS prefix and HH is the normalized current gradient.

As established in Lemma B.3, this construction provides nonnegative alignment with the current optimization signal while controlling first-order perturbations to the near-orthogonal structure. Furthermore, the additional corollary presented in our response to Reviewer Lz3j (W1.2) establishes sufficient conditions under which the complete GS-Muon update achieves a strictly larger current-gradient inner product than standard five-step Muon. Thus, the benefit of geometric steering is not simply an increased emphasis on the current gradient. Instead, GS-Muon separates current-gradient orientation adaptation from historical momentum accumulation, preserving temporal smoothing while introducing a geometrically controlled correction.

2) Additional mixing experiments demonstrate that simple historical-gradient reweighting cannot fully reproduce the benefits of GS-Muon.

To directly evaluate the proposed simpler alternative, we conduct a systematic first-order gradient-mixing ablation on Qwen2.5-1.5B across all eleven single-dataset online streams. Specifically, we consider

Mˉt=(1−λ)Mt+λgt,λ∈{0,0.1,0.2,…,1.0},\bar M_t=(1-\lambda)M_t+\lambda g_t,\qquad \lambda\in \lbrace0, 0.1,0.2,\ldots,1.0\rbrace,

For a controlled comparison, we maintain identical learning rates, momentum coefficients, trainable parameters, and online data streams across the evaluated configurations. As shown in Table A of the shared Official Comment, GS-Muon achieves superior overall performance compared with all evaluated mixing configurations. Notably, this advantage persists across the full range of mixing coefficients, including the gradient-only endpoint λ=1\lambda=1. These results demonstrate that directly increasing the current-gradient contribution is insufficient to reproduce the performance improvements obtained through geometric steering. We additionally include an adaptive-momentum baseline in the same comparison, as described in our response to Reviewer QVVK (W2). Unlike input-only mixing, adaptive momentum modifies the recursively accumulated momentum state, providing a complementary assessment of dynamic historical weighting.

W1.2: Additional First-Order Comparisons with AdamW

We agree that comparisons with AdamW are important for establishing the competitiveness of GS-Muon beyond the Muon optimizer family. To address this concern, we extend our first-order evaluation to include AdamW on both Qwen2.5-1.5B and Llama-3-8B. To ensure a fair comparison, all methods use the same initial model checkpoints, online streams, and training objectives. The corresponding results are provided in Table B of the shared Official Comment.

AuthorsOpenReview ↗

Response to Reviewer XmYS-Part 2

Response to W2&Q2

We thank the reviewer for these constructive suggestions. To strengthen the empirical evaluation, we provide additional five-seed experiments, clarify the online evaluation and hyperparameter-selection protocols, and expand the reporting of implementation details and computational costs. We also extend the first-order experiments to Llama-3-8B, enabling a more comprehensive comparison across model scales.

W2.1: Multiple Seeds and Statistical Reliability

Five paired runs demonstrate that GS-Muon maintains its performance advantage across random seeds in Table C of the shared Official Comment, providing additional evidence that the observed improvements are reproducible rather than artifacts of a particular experimental run. To evaluate run-to-run variability, we conduct five independent runs with paired random seeds S={1,2,3}\mathcal S=\lbrace 1,2,3 \rbrace on all four Qwen2.5-1.5B single-dataset online streams under both FO and ZO access. For each seed, all compared methods use the same initial model checkpoint and online data order, ensuring matched evaluation conditions. The selected hyperparameter configurations are fixed before repeated-run evaluation. For each method and task, we report the mean performance and sample standard deviation:

Pˉ=1n∑s=1nPs,sP=1n−1∑s=1n(Ps−Pˉ)2,\bar P=\frac1n\sum_{s=1}^{n}P_s,\qquad s_P=\sqrt{\frac1{n-1}\sum_{s=1}^{n}(P_s-\bar P)^2},

where n=3n=3 and PsP_s denotes the online performance obtained with seed ss.

The results in Table C of the shared Official Comment GS-Muon achieves the highest average performance among the evaluated baselines under both FO and ZO access.

W2.2. Experimental Reproducibility, Hyperparameter Selection, and Computational Costs

We provide these details in the supplementary tables of our shared Official Comment and clarify the experimental protocol below.

1. Online evaluation, data splits, and hyperparameter-selection criteria.

All optimizers follow the same causal online evaluation protocol, and hyperparameters are selected using a consistent task-specific performance criterion. For the single-dataset experiments in Tables 1 and 2, each task is evaluated using a stream of 1,000 sequential sample positions. All compared optimizers use matched sampled streams and the same prediction-before-update protocol. At each online step, the model first predicts using its current parameters, after which the feedback is revealed and the prediction is evaluated. The model then performs one parameter update using the newly observed example:

Wt →prediction evaluation →feedback Wt+1.W_t\ \xrightarrow{\mathrm{prediction}}\ \mathrm{evaluation}\ \xrightarrow{\mathrm{feedback}}\ W_{t+1}.

Previously processed stream positions are not replayed during the run.

Hyperparameters are selected according to the cumulative online performance measured using the dataset-specific evaluation metric. Classification and multiple-choice tasks use their specified classification or choice metrics, while generation tasks use answer-matching metrics appropriate to the corresponding benchmark. The exact metric definitions, answer normalization, and scoring procedures are summarized in Table D of the shared Official Comment.

2. Hyperparameter configurations, implementation details, and computational efficiency.

We provide a unified experimental configuration and resource report to make the comparisons reproducible and to clarify the computational overhead of GS-Muon relative to the baselines. To provide a complete account of the experimental settings, Table E of the shared Official Comment additionally summarizes the configurations of the compared optimizers.

For data stream construction, we use a fixed random seed of 0 to randomly sample up to 1,000 examples from each dataset. If a dataset contains fewer than 1,000 available examples, all available examples are included without replacement. The sampled examples and their order are determined and fixed before training begins, ensuring that all optimization methods are evaluated on identical, reproducible online data streams.

To address the reviewer's concern regarding computational cost, we additionally report total wall-clock runtime and peak GPU memory consumption across the eleven datasets for both model scales in the Table F of shared Official Comment. Runtime is measured in minutes, and peak GPU memory is reported in GiB.

Q2: Additional first-order experiments on Llama-3-8B.

We agree that first-order comparisons on Llama-3-8B are important for evaluating whether the improvements of GS-Muon extend beyond Qwen2.5-1.5B. We therefore supplement the original experiments with additional FO comparisons at the larger model scale. We extend the FO evaluation to Llama-3-8B, comparing AdamW, standard five-step FT-Muon, and GS-Muon across the same eleven NLP benchmarks. The expanded FO results for both Qwen2.5-1.5B and Llama-3-8B are presented in Table B of the shared Official Comment.

AuthorsOpenReview ↗

Response to Reviewer XmYS-Part 3

Response to W3

We thank the reviewer for these insightful comments. We clarify that the dependence on the steering strength α\alpha arises from conservative worst-case inequalities rather than a limitation of the geometric steering mechanism itself.

W3.1: The Regret Bounds Allow Zero Steering and Become Looser as α\alpha Increases

The dependence on α\alpha reflects a conservative worst-case analysis, whereas our new corollary directly characterizes the potential alignment advantage of geometric steering. The existing theoretical analysis establishes variation-dependent nonconvex stationarity guarantees. Specifically, Lemma B.5 provides the following lower bound on the alignment between the current gradient and the GS-Muon update:

⟨gt,DtGS⟩F≥(1−4α)∥gt∥F−2k∥gt−(1−β)Mt∥F.\langle g_t,D_t^{\mathrm{GS}}\rangle_F\geq(1-4\alpha)\|g_t\|_F-2\sqrt{k}\|g_t-(1-\beta)M_t\|_F.

This inequality separates the effects of steering-induced perturbations and historical momentum tracking.

Building on this result, Theorems 5.4 and 5.6 contain the factor

11−4α,0≤α<14.\frac{1}{1-4\alpha},\qquad 0\leq\alpha<\frac14.

We agree that the resulting worst-case bound becomes less favorable as α\alpha increases. However, this dependence originates from bounding the possible adverse effects of steering without explicitly exploiting its positive contribution to current-gradient alignment.

We have introduced a new corollary (Corollary C.1 in the revised manuscript). Unlike the original worst-case analysis, the new corollary directly compares the complete updates of GS-Muon and standard five-step Muon under identical momentum and current-gradient inputs. By decomposing the final-update difference into a positive steering contribution and NS residual correction terms, we establish an explicit sufficient condition M>0\mathcal M>0 under which

⟨H,DGS⟩F>⟨H,DMuon⟩F.\langle H,D^{\mathrm{GS}}\rangle_F>\langle H,D^{\mathrm{Muon}}\rangle_F.

Therefore, the new corollary provides a direct theoretical characterization of when geometric steering improves the final update's alignment with the current optimization signal, complementing the conservative stationarity guarantees of Theorems 5.4 and 5.6. We measure the actual alignment gain ΔM=⟨H,DGS−DMuon⟩F\Delta_\mathcal M=\langle H,D^{\mathrm{GS}}-D^{\mathrm{Muon}}\rangle_F in Table G of the shared Official Comment

W3.2: The Final NS Correction Require an Unverified Local Contraction Condition

1. We establish a residual non-expansion guarantee that does not require the strict local contraction condition in Lemma B.4. We agree that the strict contraction result in Lemma B.4 depends on the local condition ρ<1\rho<1, which is not automatically guaranteed for every steered matrix. However, the exact NS residual identity remains valid without this assumption, allowing us to establish a more general residual non-expansion property.

Let

X=Xt,3,E=γtTt,3,Z=X+E,X=X_{t,3},\qquad E=\gamma_tT_{t,3},\qquad Z=X+E,

and define the residual before the final NS correction as

R~=I−Z⊤Z.\widetilde R=I-Z^\top Z.

The final NS update is

Y=12Z(3I−Z⊤Z).Y=\frac12Z(3I-Z^\top Z).

As established in Lemma B.4, the resulting orthogonality residual satisfies the exact identity

I−Y⊤Y=34R~2+14R~3.I-Y^\top Y=\frac34\widetilde R^2+\frac14\widetilde R^3.

Importantly, this identity holds regardless of whether ρ=∥R~∥op<1\rho=\|\widetilde R\|_{\mathrm{op}}<1.

We now show that the final NS correction is residual-nonexpansive under the steering-strength range used in our theoretical analysis. First, the normalized NS initialization and the classical NS iteration guarantee

∥Xt,3∥op≤1.\|X_{t,3}\|_{\mathrm{op}}\leq1.

Moreover, Lemma B.5 establishes

∥Xt,3−Xt,2∥F≤98.\|X_{t,3}-X_{t,2}\|_F\leq\frac98.

Under the relative steering rule in Eq. (12),

∥E∥op≤∥E∥F≤9α8.\|E\|_{\mathrm{op}}\leq\|E\|_F\leq\frac{9\alpha}{8}.

Consequently, for 0≤α≤1/40\leq\alpha\leq1/4,

∥Z∥op≤1+9α8≤4132<2.\|Z\|_{\mathrm{op}}\leq1+\frac{9\alpha}{8}\leq\frac{41}{32}<\sqrt2.

Since Z⊤ZZ^\top Z is positive semidefinite, every eigenvalue rir_i of R~=I−Z⊤Z\widetilde R=I-Z^\top Z satisfies −1<ri≤1-1<r_i\leq1. The exact NS residual identity transforms each residual eigenvalue according to

ψ(r)=3r2+r34.\psi(r)=\frac{3r^2+r^3}{4}.

For r∈[−1,1]r\in[-1,1], we have

∣ψ(r)∣=∣r∣2(3+r)4≤∣r∣.|\psi(r)|=\frac{|r|^2(3+r)}4\leq|r|.

Therefore,

\|I-Y^\top Y\|_F^2=\sum_i|\psi(r_i)|^2\leq\sum_i|r_i|^2=\|I-Z^\top Z\|_F^2. \end{aligned}$$ Hence, $$\|I-Y^\top Y\|_F\leq\|I-Z^\top Z\|_F.$$ **This establishes that the final NS correction cannot increase the Frobenius-norm orthogonality residual relative to the steered input when $0\leq\alpha\leq1/4$, even if the strict local condition $\rho<1$ is not satisfied.** Theorems 5.4 and 5.6 instead use the uniform norm and alignment bounds established in Lemma B.5, which follow from controlling the steering displacement. Thus, failure of the strict local contraction condition does not invalidate the global stationarity guarantees under their remaining assumptions.
Reviewer_Lz3jOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes GS-Muon for online LLM fine-tuning. It adds a geometric steering step to Muon's Newton–Schulz iterations, helping momentum-based updates follow the current gradient when tasks change. The authors provide geometric analysis and nonconvex regret bounds, and evaluate the method on 11 NLP datasets using Qwen2.5-1.5B and Llama-3-8B. Results show improvements over Muon and several zeroth-order optimizers.

Critical strengths
  1. Steering the update toward the current gradient while preserving Muon's matrix structure makes sense, and the proposed method is relatively easy to incorporate into existing optimizers.

  2. GS-Muon improves average performance under both first-order and zeroth-order settings. The task-shift experiments in Table 3 are particularly relevant, and Table 4 provides useful evidence that the steering component matters.

Critical weaknesses
  1. I think the benefit of geometric steering is not fully isolated. The main idea is to correct stale momentum using the current gradient. But couldn't a simpler approach, such as blending the current gradient with momentum before standard NS, achieve something similar? Table 4 compares against removing steering, but not against these simpler alternatives. The regret bounds also do not establish an advantage over standard Muon. I'd like to see clearer evidence that the geometric construction itself is necessary.

  2. The results look promising, but the first-order comparison is only reported for Qwen2.5-1.5B, and there is no AdamW baseline. I also couldn't find confidence intervals or run-to-run variance for the main results. I think for an optimizer paper, these comparisons matter. Without them, it's difficult to judge how robust the improvements are and whether GS-Muon is consistently better than simpler, well-tuned alternatives.

Questions for authors

Please refer to the Weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer Lz3j-Part 1

Response to W1

W1.1: Why Geometric Steering Outperforms Gradient–Momentum Mixing

We thank the reviewer for this important question. The key advantage of GS-Muon is that it improves responsiveness to the current gradient without sacrificing the historical smoothing provided by momentum. In contrast, direct gradient–momentum mixing progressively reduces the contribution of historical information as the current gradient's weight increases. GS-Muon decouples these two objectives through a geometrically constrained orientation correction, and our additional mixing ablations support the advantage of this design over directly reweighting historical gradients.

1. Excessive Gradient Mixing Weakens the Historical Smoothing Mechanism of Muon.

Momentum accumulation is an integral component of Muon, allowing the optimizer to exploit historical gradients and suppress stochastic fluctuations. Increasing the current-gradient mixing coefficient can alleviate momentum lag, but it simultaneously reduces temporal smoothing.

Standard Muon accumulates gradients as

=\sum_{j=0}^{t-1}\beta^jg_{t-j}.$$ Direct gradient–momentum mixing computes $$\bar M_t=(1-\lambda)M_t+\lambda g_t =g_t+(1-\lambda)\sum_{j=1}^{t-1}\beta^jg_{t-j}.$$ Thus, larger $\lambda$ values progressively reduce the historical contribution to the current NS input. At $\lambda=1$, the update becomes $$D_t^{\mathrm{Mix}} =\operatorname{NS}_5 \left(\frac{g_t}{\|g_t\|_F}\right).$$ This remains a current matrix-structured NS update, but no longer uses historical momentum for the current transformation. To illustrate the associated loss of temporal smoothing, consider locally stationary stochastic gradients $$g_t=\mu+\xi_t,\qquad \mathbb E[\xi_t]=0,\qquad \mathbb E\|\xi_t\|_F^2=\sigma^2,$$ with independent noise across iterations. After normalizing the mixing input by its temporal weight sum, the steady-state estimation variance is $$V(\lambda)= \sigma^2 \frac{ 1+\frac{(1-\lambda)^2\beta^2}{1-\beta^2} }{ \left(1+\frac{(1-\lambda)\beta}{1-\beta}\right)^2}.$$ In particular, $$V(0)=\sigma^2\frac{1-\beta}{1+\beta}, \qquad V(1)=\sigma^2.$$ For $0<\beta<1$, $V(\lambda)$ is increasing on $\lambda\in[0,1]$. Thus, stronger mixing systematically weakens the variance-reduction benefit of historical momentum. This observation is consistent with the motivation for temporal smoothing in nonstationary online optimization. [1] and [2] study time-smoothed optimization using historical objectives, while [3] characterize the challenges associated with temporally varying objectives. Therefore, the fundamental limitation of direct mixing is that increasing responsiveness requires progressively suppressing the same historical information that supports temporal stability. **2. Geometric Steering Preserves Momentum While Explicitly Correcting the Update Orientation.** GS-Muon addresses this trade-off by retaining the original momentum accumulation and applying a separate, geometrically constrained correction to the momentum-derived NS reference. This allows historical smoothing and current-gradient orientation adaptation to be handled through distinct mechanisms. Mixing applies standard NS to a reweighted gradient: $$D_t^{\mathrm{Mix}} =\operatorname{NS}_5 \left( \frac{(1-\lambda)M_t+\lambda g_t} {\|(1-\lambda)M_t+\lambda g_t\|_F} \right).$$ In contrast, GS-Muon first computes $$X=\operatorname{NS}_3 \left(\frac{M_t}{\|M_t\|_F}\right),$$ and then constructs $$T=H-X\operatorname{sym}(X^\top H), \quad H=\frac{g_t}{\|g_t\|_F}.$$ Defining $S=\operatorname{sym}(X^\top H), \ R=I-X^\top X$, Lemma B.3 establishes $$\langle H,T\rangle_F= \|H\|_F^2-\|S\|_F^2 \geq0, \quad X^\top T+T^\top X=RS+SR.$$ For $Z=X+\gamma T$, it follows that $$Z^\top Z-X^\top X =\gamma(RS+SR)+\gamma^2T^\top T,$$ and hence $$\lVert Z^\top Z-X^\top X\rVert_F \leq 2\gamma\lVert R\rVert_{\mathrm{op}}\lVert S\rVert_F +\gamma^2\lVert T\rVert_F^2.$$ At an orthogonal reference $X^\top X=I$, the first-order term vanishes: $$Z^\top Z-I=\gamma^2T^\top T.$$ Moreover, the tangent direction maximizes alignment with the current gradient among first-order orthogonality-preserving perturbations of equal norm. Specifically, when $T\neq0$: $$\begin{aligned} \max_V\quad &\langle H,V\rangle_F \qquad \mathrm{s.t.} X^\top V+V^\top X=0,\quad \|V\|_F\leq\delta \end{aligned}$$ has the solution $$V^\star=\delta\frac{T}{\|T\|_F},$$ GS-Muon additionally controls the displacement through $$\lVert \gamma T\rVert_F=\alpha\lVert X_{t,3}-X_{t,2}\rVert_F.$$ when $T\neq0$, with zero displacement otherwise, before applying the final NS correction. These results reveal the principal advantage of GS-Muon: rather than sacrificing historical smoothing to increase the current-gradient contribution, it preserves the momentum-derived reference and introduces an explicitly aligned, geometrically controlled orientation correction.
AuthorsOpenReview ↗

Response to Reviewer Lz3j-Part 2

3. Additional Mixing Ablations Demonstrate the Advantage of GS-Muon.

To directly assess whether gradient–momentum mixing can reproduce the benefits of geometric steering, we conducted additional FO comparisons by systematically varying the mixing coefficient. Specifically, we consider

\lambda\in\lbrace 0,0.1,0.2,\ldots,1.0\rbrace.$$ The experiments are conducted on Qwen2.5-1.5B under FO gradient access across eleven single-dataset online streams, with results reported in **Table A of the Shared Official Comment**. To ensure a fair comparison, we use identical learning rates, momentum coefficients, and other optimization settings across all methods. The results demonstrate that GS-Muon consistently achieves superior overall performance compared with all evaluated gradient–momentum mixing configurations. In particular, even when the mixing coefficient is systematically varied from small values to the gradient-only limit, direct mixing fails to match the performance of GS-Muon. This indicates that simply increasing the contribution of the current gradient is insufficient to reproduce the benefits of geometric steering. Overall, these results provide empirical evidence for the key advantage of GS-Muon: it preserves momentum-based historical smoothing while explicitly adapting the update orientation through a geometry-aware correction. #### W1.2: Do the regret bounds show an advantage over standard Muon? We thank the reviewer for requesting a direct theoretical comparison with standard Muon. In the revised manuscript, we add a corollary and a detailed proof establishing a sufficient condition under which the complete GS-Muon update has a strictly larger inner product with the current gradient than standard five-step Muon. The comparison uses identical momentum and gradient inputs and the actual steering step size in GS-Muon. **1. The new corollary establishes a sufficient condition for improved final-update alignment.** The added Corollary C.1 in revised manuscript compares two updates sharing the first three NS iterations. Standard Muon performs two additional NS steps, whereas GS-Muon applies the steering correction followed by one final NS step. This same-input construction isolates the geometric transformation rather than comparing different optimizer trajectories. The proof proceeds in three main stages. First, we expand the two complete transformations and derive an exact final-update difference. Second, Lemma B.3 identifies the nonnegative current-gradient contribution of steering; bounding the remaining NS correction terms gives a computable lower bound $\mathcal M$ on the actual alignment gain. Third, using the algorithm's relative steering step size and the NS residual recurrence, we show that the condition can hold in a nondegenerate near-orthogonal regime; an explicit normalized NS trajectory further verifies that the condition is nonvacuous. In particular, the corollary establishes $$\mathcal M>0\quad\Longrightarrow\quad \langle H,D^{\mathrm{GS}}\rangle_F>\langle H,D^{\mathrm{Muon}}\rangle_F.$$ The accompanying Remark C.2 specifies an empirical check (**Table G in Shared Official Comment**). On the three first-order task-shift streams, at sampled updates we measure the actual alignment gain $\Delta_\mathcal M=\langle H,D^{\mathrm{GS}}-D^{\mathrm{Muon}}\rangle_F$. **2. The existing regret bounds establish stationarity and complement the new comparison.** Theorems 5.4 and 5.6 establish variation-dependent nonconvex stationarity guarantees under first-order and zeroth-order gradient access. The factor $1/(1-4\alpha)$ results from conservative worst-case control of steering-induced perturbations and does not capture the positive alignment mechanism isolated by our new corollary. Consequently, we do not claim that the existing stationarity bounds prove unconditional regret dominance over standard Muon. In summary, the new corollary provides a rigorous, directly testable sufficient condition for a final-update alignment advantage, while the existing regret bounds establish complementary optimization guarantees. Together with the proposed same-input diagnostic, this strengthens the theoretical evidence for geometric steering. ### References [1] Hazan, E., Singh, K., and Zhang, C. (2017). [Efficient Regret Minimization in Non-Convex Games](https://proceedings.mlr.press/v70/hazan17a.html). ICML, 1433-1441. [2] Hallak, N., Mertikopoulos, P., and Cevher, V. (2021). [Regret Minimization in Stochastic Non-Convex Learning via a Proximal-Gradient Approach](https://proceedings.mlr.press/v139/hallak21a.html). ICML, 4008-4017. [3] Besbes, O., Gur, Y., and Zeevi, A. (2015). [Non-Stationary Stochastic Optimization](https://doi.org/10.1287/opre.2015.1408). Operations Research, 63(5), 1227-1244.
AuthorsOpenReview ↗

Response to Reviewer Lz3j-Part 3

Response to W2

W2.1: Missing AdamW and Llama-3-8B FO comparisons

We extend the FO comparison to AdamW and Llama-3-8B using a common online protocol. We will compare AdamW, five-step FT-Muon, and GS-Muon on Qwen2.5-1.5B and Llama-3-8B across all eleven tasks. Within each model, the checkpoint, trainable parameters, data stream, prediction-before-update procedure, and metric will be matched. See Table B in the shared Official Comment. The proposed AdamW grid is η∈{5×10−6,10−5,2×10−5}\eta \in \lbrace 5\times10^{-6}, 10^{-5}, 2\times10^{-5} \rbrace, β1∈{0.8,0.9,0.95}\beta_1 \in \lbrace 0.8, 0.9, 0.95 \rbrace, with β2=0.999\beta_2=0.999.

W2.2: Confidence Intervals

Five paired runs confirm the consistent performance advantage and robustness of GS-Muon across different random seeds. To assess run-to-run variability, we conduct five independent runs using paired random seeds S={1,2,3}\mathcal S=\lbrace1,2,3 \rbrace across all four Qwen2.5-1.5B single-dataset online streams under both FO and ZO access. For each seed, all methods share the same initial checkpoint and data order to ensure a fair comparison. We report the mean performance and sample standard deviation for each method and task:

Pˉ=1n∑s=1nPs,sP=1n−1∑s=1n(Ps−Pˉ)2.\bar P=\frac1n\sum_{s=1}^{n}P_s,\qquad s_P=\sqrt{\frac1{n-1}\sum_{s=1}^{n}(P_s-\bar P)^2}.

As shown in Table C of the shared Official Comment, GS-Muon achieves the highest performance among all evaluated baselines under both FO and ZO access.

AuthorsOpenReview ↗

With Appreciation for Your Feedback and Consideration

Dear Reviewer Lz3j,

We sincerely appreciate your thoughtful comments and suggestions. In response, we have carefully addressed the points you raised and provided further explanations.

With the discussion period drawing to a close, we would be grateful if you could review our responses when convenient and share any remaining questions or concerns. We would be pleased to offer additional clarification if needed.

Thank you once again for your time and valuable feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Supplementary Experiments and Tables

We provide additional experimental results, implementation details, and geometric diagnostics in the anonymous supplementary repository:

  • Table A. Comparisons with Gradient Mixing and Adaptive Momentum. View Table A
  • Table B. Additional First-Order Comparisons Across Model Scales. View Table B
  • Table C. Five-Seed Evaluation and Statistical Reliability. View Table C
  • Table D. Task-Specific Evaluation Metrics. View Table D
  • Table E. Implementation Details and Hyperparameter Configurations. View Table E
  • Table F. Runtime and Peak GPU Memory Across Model Scales. View Table F
  • Table G. Geometric Diagnostics and Validation of the Steering Mechanism. View Table G

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.