- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper replaces Muon’s linear optimization problem over a spectral-norm ball with a curvature-aware quadratic problem. It shows that the optimum can change both singular amplitudes and directions, then develops an inversion-free implementation using calibrated Kronecker-factored curvature and warm-started Frank–Wolfe updates. GPT pretraining experiments report improved validation loss and some matched-loss wall-clock gains.
Critical strengths
I like the framing of this work, the fact that many existing curvature aware methods reduce to Muon under the isotropic setting, whereas QSD does not. The efficiency improvement techniques are interesting. It is neat that QSD can be decomposed to matmul operations. I appreciate the ablations and sensitivity analyses.
Critical weaknesses
The evaluation is limited to small models 350M parameters or fewer, with modest gains. QSD introduces many hyperparameters. The theoretical contribution is relatively limited: Propositions 4.1–4.2 follow from standard scalar minimization and trust-region KKT arguments, while Appendix B specializes standard Frank–Wolfe guarantees. “Dominance over Muon” (Prop E.2) holds on the quadratic surrogate only, since the Muon update sits on the first line-search segment. Nothing connects surrogate decrease to loss decrease, there is no outer-loop convergence result, and the warm-start tracking error (the target moves every step as momentum and factors change) is unanalyzed. Also, important baselines such as Mousse, FISMO, standard AdamW, and recent Muon spectral shaping methods are missing. It is also unclear if or how the baselines that are presented (NorMuon and Newton-Muon) are tuned.
Questions for authors
Please refer to the weaknesses above. I am in particular concerned about empirical validation, regarding the breadth of methods and the scale of evaluation.
7 comments
Dominance over Muon holds on the quadratic surrogate only, Nothing connects surrogate decrease to loss decrease: We appreciate the reviewer for highlighting the distinction between surrogate improvement and actual loss reduction. We agree that Proposition E.2 establishes dominance only for the quadratic surrogate under zero initialization and exact line search, and does not imply unconditional dominance in the actual training loss. The proposition is intended to provide a heuristic explanation on QSD's potential advantage over Muon at the surrogate-objective level, rather than a strict guarantee of superior training performance of QSD.
To investigate the relationship between the quadratic approximation and actual loss reduction, we conducted an additional one-step analysis of the update at identical checkpoints, comparing first-order gain , second-order curvature cost , and actual loss reduction over 36 checkpoint–batch pairs (9 checkpoints 4 batches). Here, denotes the actual parameter update, including the learning-rate scaling. We use the independently tuned learning rates of 0.01 for Muon and 0.02 for QSD. To control for differences in applied update magnitude, we rescale each Muon update to match the Frobenius norm of the corresponding QSD update while preserving its direction.
| Method | Update norm | First-order gain ↑ | Curvature cost ↓ | Actual reduction ↑ | |
|---|---|---|---|---|---|
| Muon | Original | 1.78 | 43.80 ± 4.26 | 22.22 ± 2.88 | 21.44 ± 1.52 |
| QSD | Original | 2.44 | 36.96 ± 3.48 | 7.82 ± 1.12 | 29.07 ± 2.42 |
| Muon | Matched Fro. (to QSD) | 2.44 | 57.10 ± 5.43 | 37.34 ± 4.59 | 19.49 ± 1.13 |
As shown in the above table, QSD achieves a larger average actual loss reduction than Muon (29.07 vs. 21.44), with a substantially lower curvature cost (7.82 vs. 22.22). Moreover, QSD maintains a higher average loss reduction under matched Frobenius norms, suggesting that the improvement is not merely due to update rescaling.
Accordingly, we have clarified the scope of Proposition E.2 and include these empirical findings in the revised version.
There is no outer-loop convergence result: We thank the reviewer for pointing out this limitation. To address this concern, we have added an outer-loop convergence analysis for stochastic QSD.
Specifically, under standard smoothness and bounded-variance assumptions, together with uniformly bounded positive-semidefinite curvature estimates, we establish the following expected stationarity bound:
\leq O(T^{-1/2})+ O\left(\sigma\sqrt{\frac{1-\beta}{1+\beta}}\right),$$ where $\sigma$ denotes the stochastic gradient noise level and $\beta$ is the momentum coefficient. The proof of this bound could be found in the Appendix G in the revised manuscript. Importantly, the analysis accommodates time-varying curvature estimates, arbitrary feasible warm starts, and a fixed number of Frank–Wolfe iterations. In the deterministic setting, the bound further yields an $O(T^{-1/2})$ first-order stationarity guarantee. While this analysis considers an idealized single-block QSD setting rather than the complete LLM training implementation, it establishes a theoretical connection between the inner solver and outer-loop optimization. We have add the convergence analysis in the revised version.The warm-start tracking error (the target moves every step as momentum and factors change) is unanalyzed: We thank the reviewer for highlighting the need to analyze warm-start tracking under evolving optimization objectives. To address this concern, we conducted additional experiments over 20 sampled training steps and three representative layers, examining both the effect of the Frank-Wolfe iteration budget and the tracking behavior during actual training.
We measure the normalized objective error relative to a numerical reference obtained by running 200 NS-FW iterations from zero initialization on each frozen quadratic subproblem. Specifically, for training step and layer , we define
where is the quadratic objective at the current training step, denotes the solution after Frank-Wolfe iterations, and is the NS-200 reference. The error is normalized separately for each checkpoint-layer pair before averaging. A smaller value indicates a better approximation to the reference objective value.
First, we examine the effect of the Frank-Wolfe iteration budget on frozen quadratic objectives, starting from the saved warm-start initialization.
| Method | K=1 | K=2 | K=3 | K=4 | K=5 | K=6 | K=7 | K=8 | K=9 | K=10 |
|---|---|---|---|---|---|---|---|---|---|---|
| Warm-start | 15.2862 | 9.2965 | 7.2547 | 5.9201 | 5.1318 | 4.5180 | 4.0342 | 3.6718 | 3.3787 | 3.1335 |
As shown in the above table, the average normalized objective error decreases from 15.29% at to 7.25% at , and further to 3.13% at . These results show that additional Frank-Wolfe iterations progressively improve the quadratic objective.
We further examine warm-start tracking during actual training, where momentum and curvature estimates evolve over time. Here, denotes the saved warm-start initialization inherited from the previous training step, while denotes the actual output of the online solver.
| Training step | 2 | 3 | 160 | 161 | 162 | 192 | 193 | 194 | 2272 | 2273 | 2274 | 2304 | 2305 | 2306 | 4192 | 4193 | 4194 | 4224 | 4225 | 4226 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Warm () | 70.4827 | 40.4531 | 45.6993 | 44.4927 | 42.7178 | 44.0450 | 51.2341 | 34.4228 | 30.7363 | 26.6496 | 26.2394 | 31.3098 | 29.5178 | 28.1232 | 24.6983 | 23.9965 | 24.5139 | 22.4591 | 23.1947 | 21.4141 |
| Warm () | 0.0990 | 1.0100 | 10.5900 | 7.8357 | 10.1029 | 8.5558 | 13.6193 | 13.1409 | 10.2285 | 9.9689 | 10.0135 | 9.9538 | 9.7414 | 9.5574 | 4.3101 | 4.3937 | 4.2906 | 4.3415 | 1.2603 | 2.0806 |
| Updates | -- | -- | -- | F | -- | -- | F+C | -- | -- | F | -- | -- | F+C | -- | -- | F | -- | -- | F+C | -- |
Note: F = factor refresh; C = first use of updated GGN calibration.
As shown in the above table, the average normalized objective error decreases from 34.32% at warm-start initialization to 7.25% after three Frank-Wolfe iterations, corresponding to a 78.8% reduction in the mean reference-relative objective gap. Although the residual error varies across training steps, including those involving curvature-factor refreshes and calibration updates, these results indicate that a small, fixed number of Frank-Wolfe iterations can substantially improve the solution under evolving objectives.
While these experiments do not establish a formal tracking-error guarantee, they provide empirical evidence of the practical tracking behavior of the warm-started solver under time-varying momentum and curvature estimates.
We thank the reviewer for recognizing the distinctive formulation of QSD, its computational efficiency, and the comprehensive experimental analyses. We also appreciate the constructive feedback and address the specific concerns below.
W1: The evaluation is limited to small models 350M parameters or fewer, with modest gains.
We thank the reviewer for highlighting the limitations in model scale and the magnitude of the improvements. To address the concern about model scale, we extended our evaluation to GPT-770M, more than twice the size of the largest model in our original experiments. QSD achieves a validation loss of 2.7892, compared with 2.8013 for Muon, yielding an absolute improvement of 0.0121.
Furthermore, the benefits of QSD extend beyond final validation loss. On GPT-124M, QSD achieves 15% token speedup and 8.4% wall-clock speedup at matched validation loss, demonstrating practical training efficiency improvements despite its computational overhead.
While experiments at substantially larger scales would further strengthen the evaluation, such experiments remain challenging given our current computational resources. We leave a more comprehensive large-scale evaluation to future work.
W2: QSD introduces many hyperparameters. The theoretical contribution is relatively limited: Propositions 4.1–4.2 follow from standard scalar minimization and trust-region KKT arguments, while Appendix B specializes standard Frank–Wolfe guarantees. “Dominance over Muon” (Prop E.2) holds on the quadratic surrogate only, since the Muon update sits on the first line-search segment. Nothing connects surrogate decrease to loss decrease, there is no outer-loop convergence result, and the warm-start tracking error (the target moves every step as momentum and factors change) is unanalyzed.
QSD introduces many hyperparameters: We acknowledge that QSD introduces additional hyperparameters for curvature estimation and the Frank–Wolfe solver. However, as shown in Appendix G.2 (Now H.2), QSD consistently outperforms Muon across a wide range of hyperparameter settings, indicating that its performance is not quite sensitive to these choices. Furthermore, most QSD-specific hyperparameters can be shared across different model sizes, thus reducing the need for extensive model-specific tuning.
The theoretical contribution is relatively limited: We thank the reviewer for pointing out this limitation. While our analysis builds on established optimization techniques, its main contribution is to characterize the structure of curvature-aware updates under spectral-norm constraints. Specifically, Propositions 4.1–4.2 show how curvature can induce nonuniform singular values and simultaneously alter singular directions, providing theoretical insights on how QSD differs from Muon. These structural properties are further supported by the empirical observations in Figure 1. In addition, we establish convergence guarantees for the proposed Frank–Wolfe solver. Our contribution thus lies in the formulation, structural analysis, and efficient solution of the curvature-aware quadratic optimization problem, rather than in developing new general-purpose optimization theory.