acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Matrix Sign: Quadratic Spectral Descent

12 views

Abstract

Muon emerges as a strong competitor of the AdamW for LLM pretraining, because the matrix-wise update it employs can potentially incur smaller second-order penalty than the once dominating AdamW, which performs coordinate-wise update. However, the spectral flattening procedure in Muon is quite debatable since it discards the spectral amplitude information totally. To seek for better spectral allocation (and the associated spectral subspace), we propose to solve the quadratic model of loss function under the spectral norm constraint directly (i.e., in a genuinely Newtonian way) and thus obtaining the Quadratic Spectral Descent (QSD) algorithm. In contrast, many existing curvature-aware methods either exploit the second-order information in an implicit way by changing the weight update geometry (such as Mousse, FISMO) or rely on strong assumptions (such as the weight displacement isotropy assumption in Newton-Muon). QSD's potential advantage over these methods is best illustrated in the isotropic curvature scenario, where Mousse, FISMO and Newton-Muon all reduce to Muon while the spectral allocation in QSD is still non-flat (since the spectral allocation in QSD depends on the gradient to curvature ratio). Meanwhile, to control the complexity of QSD, we employ inversion-free K-FAC and online Frank-Wolfe update which is essentially a matrix sign operator. Overall, the complexity increase can be rather mild. Experiments on GPT pre-training show that QSD consistently improves validation loss over Muon and recent Muon variants, while achieving up to an wall-clock speedup at matched validation loss.

Primary area:
optimization
Keywords:
Matrix optimization, Muon, Second-order optimization, K-FAC, Frank–Wolfe, Large language model training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_549nOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper replaces Muon’s linear optimization problem over a spectral-norm ball with a curvature-aware quadratic problem. It shows that the optimum can change both singular amplitudes and directions, then develops an inversion-free implementation using calibrated Kronecker-factored curvature and warm-started Frank–Wolfe updates. GPT pretraining experiments report improved validation loss and some matched-loss wall-clock gains.

Critical strengths

I like the framing of this work, the fact that many existing curvature aware methods reduce to Muon under the isotropic setting, whereas QSD does not. The efficiency improvement techniques are interesting. It is neat that QSD can be decomposed to matmul operations. I appreciate the ablations and sensitivity analyses.

Critical weaknesses

The evaluation is limited to small models 350M parameters or fewer, with modest gains. QSD introduces many hyperparameters. The theoretical contribution is relatively limited: Propositions 4.1–4.2 follow from standard scalar minimization and trust-region KKT arguments, while Appendix B specializes standard Frank–Wolfe guarantees. “Dominance over Muon” (Prop E.2) holds on the quadratic surrogate only, since the Muon update sits on the first line-search segment. Nothing connects surrogate decrease to loss decrease, there is no outer-loop convergence result, and the warm-start tracking error (the target moves every step as momentum and factors change) is unanalyzed. Also, important baselines such as Mousse, FISMO, standard AdamW, and recent Muon spectral shaping methods are missing. It is also unclear if or how the baselines that are presented (NorMuon and Newton-Muon) are tuned.

Questions for authors

Please refer to the weaknesses above. I am in particular concerned about empirical validation, regarding the breadth of methods and the scale of evaluation.

7 comments

We thank the reviewer for recognizing the distinctive formulation of QSD, its computational efficiency, and the comprehensive experimental analyses. We also appreciate the constructive feedback and address the specific concerns below.

W1: The evaluation is limited to small models 350M parameters or fewer, with modest gains.

We thank the reviewer for highlighting the limitations in model scale and the magnitude of the improvements. To address the concern about model scale, we extended our evaluation to GPT-770M, more than twice the size of the largest model in our original experiments. QSD achieves a validation loss of 2.7892, compared with 2.8013 for Muon, yielding an absolute improvement of 0.0121.

MethodVal lossTime (s)Time Overhead (%)
Muon2.8013163728.30.0
QSD2.7892 (-0.0121)174698.1+6.70

Furthermore, the benefits of QSD extend beyond final validation loss. On GPT-124M, QSD achieves 15% token speedup and 8.4% wall-clock speedup at matched validation loss, demonstrating practical training efficiency improvements despite its computational overhead.

While experiments at substantially larger scales would further strengthen the evaluation, such experiments remain challenging given our current computational resources. We leave a more comprehensive large-scale evaluation to future work.

W2: QSD introduces many hyperparameters. The theoretical contribution is relatively limited: Propositions 4.1–4.2 follow from standard scalar minimization and trust-region KKT arguments, while Appendix B specializes standard Frank–Wolfe guarantees. “Dominance over Muon” (Prop E.2) holds on the quadratic surrogate only, since the Muon update sits on the first line-search segment. Nothing connects surrogate decrease to loss decrease, there is no outer-loop convergence result, and the warm-start tracking error (the target moves every step as momentum and factors change) is unanalyzed.

QSD introduces many hyperparameters: We acknowledge that QSD introduces additional hyperparameters for curvature estimation and the Frank–Wolfe solver. However, as shown in Appendix G.2 (Now H.2), QSD consistently outperforms Muon across a wide range of hyperparameter settings, indicating that its performance is not quite sensitive to these choices. Furthermore, most QSD-specific hyperparameters can be shared across different model sizes, thus reducing the need for extensive model-specific tuning.

The theoretical contribution is relatively limited: We thank the reviewer for pointing out this limitation. While our analysis builds on established optimization techniques, its main contribution is to characterize the structure of curvature-aware updates under spectral-norm constraints. Specifically, Propositions 4.1–4.2 show how curvature can induce nonuniform singular values and simultaneously alter singular directions, providing theoretical insights on how QSD differs from Muon. These structural properties are further supported by the empirical observations in Figure 1. In addition, we establish convergence guarantees for the proposed Frank–Wolfe solver. Our contribution thus lies in the formulation, structural analysis, and efficient solution of the curvature-aware quadratic optimization problem, rather than in developing new general-purpose optimization theory.

Dominance over Muon holds on the quadratic surrogate only, Nothing connects surrogate decrease to loss decrease: We appreciate the reviewer for highlighting the distinction between surrogate improvement and actual loss reduction. We agree that Proposition E.2 establishes dominance only for the quadratic surrogate under zero initialization and exact line search, and does not imply unconditional dominance in the actual training loss. The proposition is intended to provide a heuristic explanation on QSD's potential advantage over Muon at the surrogate-objective level, rather than a strict guarantee of superior training performance of QSD.

To investigate the relationship between the quadratic approximation and actual loss reduction, we conducted an additional one-step analysis of the update UU at identical checkpoints, comparing first-order gain ⟨g,U⟩\langle g, U\rangle, second-order curvature cost 12⟨U,HU⟩\tfrac{1}{2}\langle U, HU\rangle, and actual loss reduction L(W)−L(W−U)L(W)-L(W-U) over 36 checkpoint–batch pairs (9 checkpoints ×\times 4 batches). Here, U=ηDU=\eta D denotes the actual parameter update, including the learning-rate scaling. We use the independently tuned learning rates of 0.01 for Muon and 0.02 for QSD. To control for differences in applied update magnitude, we rescale each Muon update to match the Frobenius norm of the corresponding QSD update while preserving its direction.

MethodUpdate norm∥U∥F\|U\|_FFirst-order gain ↑Curvature cost ↓Actual reduction ↑
MuonOriginal1.7843.80 ± 4.2622.22 ± 2.8821.44 ± 1.52
QSDOriginal2.4436.96 ± 3.487.82 ± 1.1229.07 ± 2.42
MuonMatched Fro. (to QSD)2.4457.10 ± 5.4337.34 ± 4.5919.49 ± 1.13

As shown in the above table, QSD achieves a larger average actual loss reduction than Muon (29.07 vs. 21.44), with a substantially lower curvature cost (7.82 vs. 22.22). Moreover, QSD maintains a higher average loss reduction under matched Frobenius norms, suggesting that the improvement is not merely due to update rescaling.

Accordingly, we have clarified the scope of Proposition E.2 and include these empirical findings in the revised version.

There is no outer-loop convergence result: We thank the reviewer for pointing out this limitation. To address this concern, we have added an outer-loop convergence analysis for stochastic QSD.

Specifically, under standard smoothness and bounded-variance assumptions, together with uniformly bounded positive-semidefinite curvature estimates, we establish the following expected stationarity bound:

\leq O(T^{-1/2})+ O\left(\sigma\sqrt{\frac{1-\beta}{1+\beta}}\right),$$ where $\sigma$ denotes the stochastic gradient noise level and $\beta$ is the momentum coefficient. The proof of this bound could be found in the Appendix G in the revised manuscript. Importantly, the analysis accommodates time-varying curvature estimates, arbitrary feasible warm starts, and a fixed number of Frank–Wolfe iterations. In the deterministic setting, the bound further yields an $O(T^{-1/2})$ first-order stationarity guarantee. While this analysis considers an idealized single-block QSD setting rather than the complete LLM training implementation, it establishes a theoretical connection between the inner solver and outer-loop optimization. We have add the convergence analysis in the revised version.

The warm-start tracking error (the target moves every step as momentum and factors change) is unanalyzed: We thank the reviewer for highlighting the need to analyze warm-start tracking under evolving optimization objectives. To address this concern, we conducted additional experiments over 20 sampled training steps and three representative layers, examining both the effect of the Frank-Wolfe iteration budget and the tracking behavior during actual training.

We measure the normalized objective error relative to a numerical reference obtained by running 200 NS-FW iterations from zero initialization on each frozen quadratic subproblem. Specifically, for training step tt and layer ℓ\ell, we define

et,ℓ(K)=qt,ℓ(Dt,ℓ(K))−qt,ℓ(Dt,ℓref)qt,ℓ(0)−qt,ℓ(Dt,ℓref)+ϵ×100%,e_{t,\ell}^{(K)}=\frac{q_{t,\ell}(D_{t,\ell}^{(K)})-q_{t,\ell}(D_{t,\ell}^{\mathrm{ref}})}{q_{t,\ell}(0)-q_{t,\ell}(D_{t,\ell}^{\mathrm{ref}})+\epsilon}\times 100\%,

where qt,ℓq_{t,\ell} is the quadratic objective at the current training step, Dt,ℓ(K)D_{t,\ell}^{(K)} denotes the solution after KK Frank-Wolfe iterations, and Dt,ℓrefD_{t,\ell}^{\mathrm{ref}} is the NS-200 reference. The error is normalized separately for each checkpoint-layer pair before averaging. A smaller value indicates a better approximation to the reference objective value.

First, we examine the effect of the Frank-Wolfe iteration budget on frozen quadratic objectives, starting from the saved warm-start initialization.

MethodK=1K=2K=3K=4K=5K=6K=7K=8K=9K=10
Warm-start15.28629.29657.25475.92015.13184.51804.03423.67183.37873.1335

As shown in the above table, the average normalized objective error decreases from 15.29% at K=1K=1 to 7.25% at K=3K=3, and further to 3.13% at K=10K=10. These results show that additional Frank-Wolfe iterations progressively improve the quadratic objective.

We further examine warm-start tracking during actual training, where momentum and curvature estimates evolve over time. Here, K=0K=0 denotes the saved warm-start initialization inherited from the previous training step, while K=3K=3 denotes the actual output of the online solver.

Training step23160161162192193194227222732274230423052306419241934194422442254226
Warm (K=0K=0)70.482740.453145.699344.492742.717844.045051.234134.422830.736326.649626.239431.309829.517828.123224.698323.996524.513922.459123.194721.4141
Warm (K=3K=3)0.09901.010010.59007.835710.10298.555813.619313.140910.22859.968910.01359.95389.74149.55744.31014.39374.29064.34151.26032.0806
Updates------F----F+C----F----F+C----F----F+C--

Note: F = factor refresh; C = first use of updated GGN calibration.

As shown in the above table, the average normalized objective error decreases from 34.32% at warm-start initialization to 7.25% after three Frank-Wolfe iterations, corresponding to a 78.8% reduction in the mean reference-relative objective gap. Although the residual error varies across training steps, including those involving curvature-factor refreshes and calibration updates, these results indicate that a small, fixed number of Frank-Wolfe iterations can substantially improve the solution under evolving objectives.

While these experiments do not establish a formal tracking-error guarantee, they provide empirical evidence of the practical tracking behavior of the warm-started solver under time-varying momentum and curvature estimates.

Reviewer_hn52OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Quadratic Spectral Descent (QSD), a curvature-aware optimizer that extends Muon by directly optimizing a quadratic loss model using a spectral-norm constraint. Compared with Muon, which produces a flat singular-value spectrum, QSD uses curvature information to adapt the singular values and directions of the update. It combines K-FAC curvature estimation with an online Frank–Wolfe solver to keep computational overhead manageable. Experiments show that QSD achieves lower validation loss than Muon.

Critical strengths
  1. The paper extends Muon by directly optimizing a quadratic loss model under the spectral-norm constraint. It also develops a practical solver using K-FAC and Frank–Wolfe updates to incorporate curvature information without matrix inversion.
  2. QSD achieves lower validation loss than Muon and several variants on GPT pretraining, with around 6% training-time overhead.
Critical weaknesses
  1. The paper reports up to 8.49% wall-clock speedup at matched validation loss in table 5, but Muon does not reach the target losses in the reported runs in tables 3 and 4. The wall-clock speedups also appear to be calculated from the reported token speedups and training overhead, rather than independently measured. The paper does not provide the corresponding Muon training results or clearly explain how the token speedups were calculated.
  2. More importantly, the paper does not assess whether the performance improvements require the full K-FAC-based quadratic optimization or could be achieved through simpler methods. Proposition 4.1 shows that even isotropic curvature produces nonuniform spectral updates, and table 7 suggests that much of the improvement is associated with the nonuniform spectrum. However, the paper does not compare with this simpler isotropic-curvature baseline. So, it is unclear how much of the improvement comes from the proposed curvature estimation and iterative optimization.
Questions for authors
  1. How are the token and wall-clock speedups in table 5 calculated?
  2. How much of QSD's improvement comes from curvature-aware optimization instead of simply using a nonuniform spectrum? Could the authors compare full QSD with the simpler isotropic-curvature baseline?
5 comments

We thank the reviewer for recognizing QSD's curvature-aware formulation, its efficient implementation, and its improved validation performance with modest computational overhead. We also appreciate the constructive feedback and address the concerns below.

W1 & Q1: The paper reports up to 8.49% wall-clock speedup at matched validation loss in table 5, but Muon does not reach the target losses in the reported runs in tables 3 and 4. The wall-clock speedups also appear to be calculated from the reported token speedups and training overhead, rather than independently measured. The paper does not provide the corresponding Muon training results or clearly explain how the token speedups were calculated. How are the token and wall-clock speedups in table 5 calculated?

We thank the reviewer for pointing out the need to clarify our training-efficiency evaluation. In the original submission, we independently trained Muon from scratch under different extended token budgets and compared the final validation loss of each run with that achieved by QSD under the original fixed budget. We then selected the Muon training budget that yielded a comparable final validation loss. Since these extended-training experiments were conducted on different machines, directly comparing their wall-clock times could introduce hardware-dependent variations. Therefore, we estimated the wall-clock speedups by combining the token speedups with the measured training-time overheads in Tables 3 and 4, which were obtained under the same hardware environment. We agree that this estimation procedure should have been explicitly stated.

To address this concern, we conducted additional extended-training experiments on the same machine, directly measuring the wall-clock times of Muon and QSD under comparable final validation losses. We now calculate the wall-clock speedups from these measured times rather than estimating them from token speedups and computational overhead. Specifically, we define the speedups as

Token speedup=(NMuonNQSD−1)×100\text{Token speedup} = \left(\frac{N_{\text{Muon}}}{N_{\text{QSD}}}-1\right)\times 100%,

Wall-clock speedup=(TMuonTQSD−1)×100\text{Wall-clock speedup} = \left(\frac{T_{\text{Muon}}}{T_{\text{QSD}}}-1\right)\times 100%,

where NN denotes the total number of training tokens and TT denotes the total training time required to reach the matched validation loss.

ModelTarget lossMuon final lossMuon TokensMuon Time (s)Token SpeedupWall-clock Speedup
GPT-124M3.31513.31522.875B8491.8↑ 15.0%↑ 8.40%
GPT-350M3.00183.00128B61157.8↑ 14.3%↑ 7.32%

For GPT-124M, Muon achieved a validation loss of 3.3152 after 2.875B tokens and 8,491.8 seconds, closely matching the QSD target loss of 3.3151. For GPT-350M, Muon achieved a validation loss of 3.0012 after 8B tokens and 61,157.8 seconds. These comparisons yield token speedups of 15.0% and 14.3%, and directly measured wall-clock speedups of 8.40% and 7.32%, respectively.

The new measurements are consistent with the efficiency improvements reported in the original submission, while removing the need to estimate wall-clock speedups from token savings and training overhead. We will update Table 5 with the directly measured results and clarify the evaluation protocol in the revised version.

W2 & Q2: More importantly, the paper does not assess whether the performance improvements require the full K-FAC-based quadratic optimization or could be achieved through simpler methods. Proposition 4.1 shows that even isotropic curvature produces nonuniform spectral updates, and table 7 suggests that much of the improvement is associated with the nonuniform spectrum. However, the paper does not compare with this simpler isotropic-curvature baseline. So, it is unclear how much of the improvement comes from the proposed curvature estimation and iterative optimization. How much of QSD's improvement comes from curvature-aware optimization instead of simply using a nonuniform spectrum? Could the authors compare full QSD with the simpler isotropic-curvature baseline?

We thank the reviewer for suggesting this important comparison. To distinguish the benefit of structured curvature estimation from that of a nonuniform update spectrum, we conducted an additional ablation using an isotropic curvature approximation (QSD-Iso.). As shown in the following table, QSD-Iso. reduces the average validation loss by only 0.0043, indicating that a nonuniform update spectrum without structured curvature information provides marginal gain. In contrast, full QSD can substantially reduce the loss by 0.0158. Thus it implies that the structured curvature estimation indeed plays a crucial role in the full QSD.

MethodRun 1Run 2Run 3Avg. loss
Muon3.33183.33123.33193.3316
QSD-Iso.3.32693.32833.32683.3273 (-0.0043)
QSD3.31513.31613.31613.3158 (-0.0158)

We hope these clarifications and experiments could address your concerns and questions. We sincerely thank you again for your valuable insights, which have helped us improve the clarity and depth of our paper.

Best regards,

The authors

Dear Reviewer hn52,

Thank you again for your thoughtful and constructive feedback on our submission. We sincerely appreciate your time and effort throughout the review process.

As the November 18 discussion deadline approaches, we would like to kindly follow up on our previous responses and additional experiments. We hope they have adequately addressed your concerns.

If you have had a chance to review them, we would greatly appreciate hearing whether any concerns remain unresolved. We would be more than happy to provide further clarification or additional information before the discussion period ends.

Thank you again for your time and consideration.

Best regards,

The Authors

Reviewer_FzaZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Quadratic Spectral Descent (QSD), an optimizer that extends Muon’s linear objective to a curvature-aware quadratic objective under the same spectral-norm constraint. This formulation allows both nonuniform singular-value allocation and adaptation of the update’s singular directions. The practical algorithm combines Kronecker-factored curvature estimates with GGN-based scale calibration and a warm-started Frank–Wolfe solver, whose linear subproblems use Muon-style matrix-sign operations. The paper provides structural analyses of the optimal updates and convergence guarantees for the quadratic surrogate. Experiments on GPT-124M and GPT-350M pretraining show improved validation loss over Muon, with additional comparisons against NorMuon and Newton-Muon at the 124M scale. QSD incurs approximately 6% additional training time at a fixed token budget while achieving up to an 8.49% wall-clock speedup at matched validation loss.

Critical strengths
  • Clear and well-motivated formulation. QSD extends Muon’s linear objective to a quadratic model while preserving the spectral-norm constraint. This provides a principled way to adapt both the singular values and singular directions of the update.

  • Practical use of curvature information. Combining Kronecker-factored curvature with warm-started Frank–Wolfe iterations avoids curvature-matrix inversion and reuses Muon-style matrix-sign operations. The resulting implementation achieves a modest measured training-time overhead in the evaluated settings.

  • Interpretable theoretical analysis. The analysis explains when curvature favors nonuniform spectral allocation and when it changes the optimal singular directions. The optimality certificate and convergence guarantees also provide a clear mathematical foundation for the inner solver.

  • Consistent empirical gains with supporting ablations. QSD improves validation loss over Muon at both evaluated model scales and outperforms NorMuon and Newton-Muon in the 124M experiments. The matched-loss wall-clock results demonstrate practical benefits, while the direction and spectrum ablations help explain the observed improvements.

Critical weaknesses
  • Limited evidence connecting the proposed mechanism to the gains. Since Muon already exhibits favorable curvature properties, the additional benefit of explicit curvature modeling deserves stronger validation. Tables 7 and 13 provide useful evidence, but do not fully distinguish direction-specific curvature adaptation from update rescaling or smoothing. Comparing first-order gain, second-order curvature cost, and actual loss reduction at the same checkpoint would strengthen the explanation.

  • Insufficient validation of the practical curvature approximation. QSD relies on an empirical-Fisher K-FAC proxy with scalar calibration against GGN directional curvature. Matching curvature along one update direction does not establish accuracy along other directions explored by the solver. Ablations without calibration and with a simpler isotropic curvature model would help establish which components are necessary.

  • Incomplete comparison with closely related methods. The introduction emphasizes the distinction between QSD’s explicit quadratic objective and the curvature-aware geometries of Mousse and FISMO, but neither method is included in the experiments. Consequently, the results establish improvements over Muon and selected variants, while leaving the practical advantage over these closely related curvature-aware approaches unresolved.

Questions for authors
  1. Can you compare Muon and QSD at the same training checkpoint in terms of first-order gain, second-order curvature cost, and actual one-step loss reduction? A comparison with matched update norms would help distinguish curvature adaptation from rescaling effects.

  2. How much of the improvement depends on GGN-based scale calibration? Could you report results without calibration and with a simpler isotropic curvature model to clarify the benefit of the structured curvature estimate?

  3. How does QSD compare with Mousse and FISMO under comparable tuning and compute budgets? These comparisons would directly test the practical benefit of optimizing an explicit quadratic objective rather than modifying the update geometry.

4 comments

We sincerely thank the reviewer for the positive assessment of QSD's formulation, theoretical analysis, and empirical results, below are our response to the weaknesses and questions raised by the reviewer.

W1 & Q1: Limited evidence connecting the proposed mechanism to the gains. Since Muon already exhibits favorable curvature properties, the additional benefit of explicit curvature modeling deserves stronger validation. Tables 7 and 13 provide useful evidence, but do not fully distinguish direction-specific curvature adaptation from update rescaling or smoothing. Comparing first-order gain, second-order curvature cost, and actual loss reduction at the same checkpoint would strengthen the explanation. Can you compare Muon and QSD at the same training checkpoint in terms of first-order gain, second-order curvature cost, and actual one-step loss reduction? A comparison with matched update norms would help distinguish curvature adaptation from rescaling effects.}

Thank you for this valuable suggestion. To better understand the mechanism underlying QSD's improvement, we conducted an additional one-step analysis at identical model checkpoints. Specifically, we compared Muon and QSD in terms of the first-order gain ⟨g,U⟩\langle g,U\rangle, the second-order curvature cost 12⟨U,HU⟩\frac{1}{2}\langle U,HU\rangle, and the actual loss reduction L(W)−L(W−U)L(W)-L(W-U), evaluated over 36 checkpoint-batch pairs (9 checkpoints ×\times 4 batches).

Here, U=ηDU=\eta D denotes the actual parameter update, including the learning-rate scaling. We use the independently tuned learning rates of 0.01 for Muon and 0.02 for QSD. To control for differences in applied update magnitude, we rescale each Muon update to match the Frobenius norm of the corresponding QSD update while preserving its direction. Values are in units of 10−310^{-3} nats, reported as mean ±\pm SEM.

MethodUpdate norm∥U∥F\|U\|_FFirst-order gain ↑Curvature cost ↓Actual reduction ↑
MuonOriginal1.7843.80 ± 4.2622.22 ± 2.8821.44 ± 1.52
QSDOriginal2.4436.96 ± 3.487.82 ± 1.1229.07 ± 2.42
MuonMatched Fro. (to QSD)2.4457.10 ± 5.4337.34 ± 4.5919.49 ± 1.13

The results reveal a clear trade-off between first-order gain and second-order curvature cost. Despite achieving a smaller first-order gain, QSD incurs a substantially lower curvature cost and achieves a 35.6% larger actual loss reduction than Muon with their original updates.

More importantly, this advantage persists after layer-wise norm matching. At the same update norms, QSD reduces the curvature cost by 79.1% (7.82 vs. 37.34) and achieves a 49.2% larger actual loss reduction (29.07 vs. 19.49), despite having a smaller first-order gain.

These findings provide additional evidence for the proposed mechanism: QSD sacrifices some first-order gain in exchange for a substantially smaller curvature penalty, resulting in a greater actual loss reduction. The matched-norm comparison further suggests that the improvement is not merely attributable to differences in update magnitude, but is consistent with QSD producing directions with more favorable curvature properties.

W2 & Q2: Insufficient validation of the practical curvature approximation. QSD relies on an empirical-Fisher K-FAC proxy with scalar calibration against GGN directional curvature. Matching curvature along one update direction does not establish accuracy along other directions explored by the solver. Ablations without calibration and with a simpler isotropic curvature model would help establish which components are necessary. How much of the improvement depends on GGN-based scale calibration? Could you report results without calibration and with a simpler isotropic curvature model to clarify the benefit of the structured curvature estimate?

Thank you for the insightful suggestion. We conducted two additional ablations on GPT-124M pretraining to assess the contributions of GGN-based scale calibration and structured curvature estimation. Each configuration was evaluated over three runs.

MethodRun 1Run 2Run 3Avg. loss
Muon3.33183.33123.33193.3316
QSD w/o calibration3.32273.32123.32373.3225 (-0.0091)
QSD-Iso.3.32693.32833.32683.3273 (-0.0043)
QSD3.31513.31613.31613.3158 (-0.0158)

As shown in the table, removing GGN-based calibration increases the validation loss from 3.3158 to 3.3225, demonstrating its contribution to QSD's performance. Nevertheless, QSD without calibration still outperforms Muon (3.3225 vs. 3.3316).

To assess the contribution of structured curvature estimation, we also evaluated QSD-Iso., which replaces the structured curvature approximation with an isotropic model. QSD-Iso. achieves an average validation loss of 3.3273, outperforming Muon (3.3316), while full QSD further reduces the loss to 3.3158. This indicates that the structured curvature approximation can provide significant benefits beyond a simple isotropic model.

Together, these ablations demonstrate that both GGN-based scale calibration and structured curvature estimation contribute to QSD's performance, while neither component alone fully accounts for its improvement over Muon.

W3 & Q3: Incomplete comparison with closely related methods. The introduction emphasizes the distinction between QSD’s explicit quadratic objective and the curvature-aware geometries of Mousse and FISMO, but neither method is included in the experiments. Consequently, the results establish improvements over Muon and selected variants, while leaving the practical advantage over these closely related curvature-aware approaches unresolved. How does QSD compare with Mousse and FISMO under comparable tuning and compute budgets? These comparisons would directly test the practical benefit of optimizing an explicit quadratic objective rather than modifying the update geometry.

Thank you for this valuable suggestion. We expanded our experiments to include Mousse, FISMO, and DynMuon (a spectrum-shaping algorithm) on GPT-124M pretraining, using matched training-token budgets and comparable hyperparameter tuning.

MethodRun 1Run 2Run 3Avg. loss
Muon3.33183.33123.33193.3316
FISMO3.32963.33003.32933.3296 (-0.0020)
Mousse3.32493.32253.32593.3244 (-0.0072)
DynMuon3.32833.32843.32843.3284 (-0.0032)
QSD3.31513.31613.31613.3158 (-0.0158)

As shown in the table, QSD achieves the lowest average validation loss of 3.3158, outperforming Mousse (3.3244), FISMO (3.3296), and DynMuon (3.3284). These results demonstrate the effectiveness of QSD compared with existing curvature-aware and Muon-based optimization methods.

Regarding FISMO, its official implementation is not publicly available. We contacted the authors to request the code but have not been responded yet. Therefore, we implemented FISMO based on the algorithmic descriptions provided in the original paper.


Once again, thank you for your valuable feedback. We believe your suggestions will improve the clarity and impact of our work. We hope our responses address your concerns satisfactorily.

Best regards,

The authors

Thank you for the detailed responses and additional experiments. I have reviewed the matched-norm analysis, curvature ablations, and comparisons with related methods. These additions satisfactorily address my main concerns and strengthen the empirical support for QSD. I will maintain my original rating.

We sincerely thank the reviewer for their positive feedback and for recognizing the value of our additional analyses and experiments. We greatly appreciate your constructive suggestions, which have helped us strengthen the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses

Dear PCs, SACs, ACs, and Reviewers,

We sincerely thank you for the time, effort, and patience you have devoted to reviewing our work; your feedback has been extremely helpful in improving the paper. Considering there are some common concerns regarding our work, we'd like to provide brief responses at the very front.

At its core, our work aims to answer a fundamental question: what is the performance limit of second-order optimizer under the spectral-norm constraint? or how much additional gain can second-order information (Hessians) can bring with a practical computational cost?

While Muon provides the optimal update for the linearized objective, we propose to address the quadratic model and develop the QSD algorithm. We believe our work establishes a principled optimality benchmark for the second-order optimizer under the spectral-norm constraint, rather than relying on heuristic spectral shaping.

Moreover, as suggested by the reviewers, we have added curvature ablations and comparisons against other curvature-aware Muon variants, including FISMO and MOUSSE. Furthermore, we also added a larger-scale model (GPT-770M) experiment.

Taken together, these results highlight that QSD serves not only as an improved optimizer, but also a principled and computationally tractable framework for investigating the performance limit of second-order optimization under spectral-norm constraints. We hope the above explanations and additional results are helpful in your evaluation.

Thanks again for your excellent work!

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.