- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to improve Muon optimizer in the online learning setting. The authors notice that the momentum of vanilla Muon may conflict with the current gradient and therefore cannot adaptively adjust its behavior for both history smoothing and task switching. Then, they propose to inject the current gradient signal into Muon's momentum by geometrically steering, which can be viewed as finding a minor direction close to the current gradient in the tagent space of the momentum matrix in the Stiefel manifold to preserve the orthogonality. Finally, they propose a 3NS-steering-1NS framework, with solid theoretical results. Experimental results show that the proposed optimizer improve the final performance compared with Muon.
Critical strengths
-
The paper identifies a meaningful limitation of Muon in non-stationary online learning.
-
GS-Muon outperforms standard Muon across various tasks, achieving better average performance both in zeroth-order optimization and task-switching scenarios.
-
The paper provides geometric stability properties and nonconvex convergence guarantees for both FO and ZO settings.
Critical weaknesses
-
The paper does not compare geometric steering with simpler alternatives, such as gradient mixing or momentum adjustment. Consequently, the necessity of the proposed geometric formulation remains unclear. The authors are encouraged to provide some theoretical or empirical evidences to support the benefits of the proposed geometric steering method.
-
The noticed limitation of Muon can also be addressed by introduing an adaptive memory coefficient in the computation of Muon's momentum, which adaptively adjusts how much we smooth the history and how much we switch the task-related direction. Can geometrical steering empirically surpass the counterpart?
Questions for authors
See above.
9 comments
Response to Reviewer QVVK-Part 2
Response to W2
We thank the reviewer for this insightful suggestion. We agree that adaptively adjusting the momentum coefficient is a meaningful alternative for balancing historical information and responsiveness to the current optimization signal. To directly evaluate this possibility, we introduce an adaptive-momentum Muon baseline and compare it with GS-Muon. Importantly, although both approaches aim to improve adaptation under changing objectives, they operate through fundamentally different mechanisms: adaptive momentum modifies the accumulation of historical gradients, whereas geometric steering explicitly corrects the orientation of the momentum-derived update.
1. We introduce a gradient-alignment-based adaptive momentum rule that dynamically adjusts historical memory without requiring additional gradient computations or objective evaluations. Specifically, we measure the cosine similarity between the current optimization signal and the previously accumulated momentum:
Based on this alignment, we adapt the momentum coefficient as
where controls the maximum historical contribution. The resulting momentum is subsequently processed using the standard five-step Newton–Schulz transformation. This design adapts the historical weighting according to the relationship between the current and previous optimization signals. When the two directions are well aligned, approaches , preserving momentum-based historical smoothing. Conversely, when their directions conflict, decreases, reducing the influence of outdated historical information and allowing the update to respond more strongly to the current objective. For numerical stability, the cosine similarity is clipped to . When either input norm is zero, we set . The similarity is computed over the concatenated trainable parameter blocks. The adaptive coefficient is computed directly from the existing optimization signal, without introducing additional training queries or requiring task-boundary information.
2. We directly compare adaptive-momentum Muon with GS-Muon to determine whether dynamically adjusting historical weighting can reproduce the benefits of geometric steering. We evaluate the adaptive-momentum baseline on all eleven Qwen2.5-1.5B single-dataset online streams. For adaptive momentum, we consider
with the learning rate selected under a comparable hyperparameter-tuning budget. All methods share the same model checkpoints, trainable parameters, online data streams, and evaluation protocol. The baseline employs standard five-step Muon after updating its adaptive momentum, ensuring that the comparison isolates the effect of momentum adaptation. The results are reported in Table A of the shared Official Comment.
With Appreciation for Your Feedback and Consideration
Dear Reviewer QVVK,
We sincerely appreciate your thoughtful comments and suggestions. In response, we have carefully addressed the points you raised and provided further explanations.
With the discussion period drawing to a close, we would be grateful if you could review our responses when convenient and share any remaining questions or concerns. We would be pleased to offer additional clarification if needed.
Thank you once again for your time and valuable feedback.
Response to Reviewer QVVK-Part 1
Thank you for the constructive comments. We address each weakness and question separately below; all supplementary tables are collected in the shared Official Comment.
Response to W1
We thank the reviewer for this important suggestion. We agree that demonstrating improvements over standard Muon alone is insufficient to establish the necessity of geometric steering. To address this concern, we provide complementary theoretical and empirical evidence showing that GS-Muon offers a distinct advantage over simpler momentum-based alternatives.
W1.1. Geometric Motivation and Theoretical Advantages
1. Unlike direct gradient mixing, geometric steering adapts the update orientation without sacrificing the historical information accumulated by momentum.
Standard Muon accumulates historical gradients through
Mt=βMt−1+gt.
A straightforward alternative is to directly mix the accumulated momentum with the current gradient:
Mˉt=(1−λ)Mt+λgt=gt+(1−λ)βMt−1.
As λ increases, the historical contribution to the current NS input decreases. In particular, λ=1 completely removes the historical component, producing a gradient-only NS update. Although increasing λ can improve responsiveness to the current optimization signal, it simultaneously weakens the historical information used by the update. This creates a trade-off between current-gradient responsiveness and momentum-based temporal smoothing, which is particularly relevant in online learning under stochastic feedback and distribution shifts.
In contrast, GS-Muon retains the original accumulated momentum and first constructs the momentum-derived NS prefix Xt,3. It then introduces a tangent-inspired steering correction:
Tt=Ht−Xt,3sym(Xt,3⊤Ht),
where Ht is the normalized current optimization signal.
Let St=sym(Xt,3⊤Ht) and Rt=I−Xt,3⊤Xt,3. As established in Lemma B.3, the steering direction satisfies
⟨Ht,Tt⟩F=∥Ht∥F2−∥St∥F2≥0,
and
∥Xt,3⊤Tt+Tt⊤Xt,3∥F≤2∥Rt∥op∥St∥F.
The first inequality guarantees nonnegative alignment between the steering direction and the current optimization signal. The second shows that its FO effect on the Gram matrix is controlled by the orthogonality residual of the NS prefix.
2. A new theoretical comparison provides sufficient conditions under which the complete GS-Muon update achieves better gradient alignment than standard Muon.
We introduce a new corollary in the Corollary C.1 of revised manuscript, as detailed in our response to Reviewer Lz3j (W1.2). The analysis directly compares the complete GS-Muon update with standard five-step Muon under identical momentum and current-gradient inputs. By explicitly accounting for the positive steering contribution and the residual terms introduced by the final NS transformation, we establish a sufficient condition M>0 such that
⟨H,DGS⟩F>⟨H,DMuon⟩F.
The analysis identifies a nondegenerate near-orthogonal regime in which the sufficient condition is satisfied. This result provides direct theoretical evidence that geometric steering can improve the alignment of the complete final update with the current optimization signal, rather than merely modifying an intermediate NS representation. We measure the actual alignment gain ΔM=⟨H,DGS−DMuon⟩F in *Table G of the shared Official Comment.
W1.2. Empirical Comparisons with Gradient Mixing and Momentum Adjustment
Systematic gradient-mixing experiments demonstrate that simply increasing the current-gradient contribution cannot reproduce the performance gains of GS-Muon. To directly evaluate whether the benefits of GS-Muon can be explained by simpler historical-gradient reweighting, we conduct additional FO experiments on Qwen2.5-1.5B across all eleven single-dataset online streams. Specifically, we evaluate direct gradient–momentum mixing with
λ∈{0,0.1,0.2,…,1.0},
To isolate the effect of the optimization mechanism, all methods use identical learning rates, momentum coefficients, online data streams. Only the specified mixing or steering mechanism is changed. As shown in Table A of the shared Official Comment, GS-Muon achieves superior overall performance compared with all evaluated gradient-mixing configurations. This observation supports the proposed motivation: the advantage of GS-Muon does not arise solely from assigning greater weight to the current gradient, but from explicitly incorporating the current optimization signal into the geometry of the momentum-derived update.