acceptodds
Under review as a conference paper at ICLR 2027

Muon as a Kronecker-Factored Curvature Approximation without Accumulation

Abstract

Kronecker-factored methods such as K-FAC, Shampoo, and SOAP approximate layer curvature with input and output factors that they accumulate across steps, whereas Muon orthogonalizes the current momentum and stores no curvature matrix. The algebraic link between Muon and Shampoo without accumulation does not show whether Muon's momentum statistics follow the curvature of trained networks. We study Muon as an accumulation-free Kronecker-factored curvature approximation with two substitutions: the input factor is read from the Gram of the current momentum, and the output factor is replaced by the identity. A signal and noise decomposition explains when the momentum Gram follows the leading curvature subspace and separates subspace recovery from spectral accuracy. Measured against a sequence-level generalized Gauss-Newton reference in GPT-2, the momentum Gram captures most of the leading input curvature energy, whereas the output factor is nearly diagonal pairwise but has unequal diagonal scales and collective structure. Controlled quadratic and nonlinear experiments show that estimation noise, unequal output scales, and output coupling cause distinct failures. These results support a conditional curvature interpretation of Muon and specify where it falls short.

Primary area:
optimization
Keywords:
Muon, Kronecker-factored curvature, accumulation-free preconditioning, momentum Gram, second-order optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DsivOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper interprets Muon as an accumulation-free Kronecker-factored curvature approximation, using the momentum Gram as the input curvature factor and the identity as the output factor. The authors provide a signal-noise analysis, evaluate curvature approximation accuracy against GGN estimates in neural networks, and conduct controlled experiments investigating estimation noise, output-scale heterogeneity, and output coupling.

Critical strengths
  1. Systematic empirical analysis. The paper evaluates Muon's implicit curvature approximation against a GGN reference, distinguishing subspace alignment, spectral accuracy, and output-side curvature structure. These measurements provide useful insights beyond existing algebraic interpretations.

  2. Informative controlled experiments. The experiments separately investigate estimation noise, output-scale heterogeneity, and output coupling, helping identify different limitations of Muon's implicit curvature approximation.

Critical weaknesses
  1. Insufficient differentiation from prior work. The structured-Fisher interpretation of Muon has already been investigated by [1]. The present paper provides more extensive empirical diagnostics, but its conceptual advances over existing curvature-based interpretations are not sufficiently clear.

  2. Limited theoretical justification. The signal-noise analysis assumes stationary, independent gradients and does not account for momentum accumulated along the optimization trajectory. Moreover, the identity μ⊤μ=RAR\mu^\top\mu=RAR does not necessarily imply alignment with leading curvature directions, as this depends on the structure of AA. Consequently, the theoretical analysis provides only a conditional explanation for the curvature alignment observed in experiments.

  3. Unclear connection between curvature approximation and optimization performance. Although the paper finds that lower spectral mismatch often corresponds to better training loss, changing the curvature estimation window also affects the update direction and optimization dynamics. It is therefore unclear whether the performance differences are actually caused by more accurate curvature estimation. While the controlled experiments demonstrate the effects of different curvature structures, the extent to which Muon's curvature approximation errors affect its performance on real language models remains unclear.

[1] Wenbo Gong, Meyer Scetbon, Chao Ma, and Ted Meeds. Towards efficient optimizer design for LLM via structured Fisher approximation with a low-rank extension. In International Conference on Learning Representations (ICLR), 2026.

Questions for authors
  1. Could the authors clarify the main conceptual advances over the structured-Fisher interpretation of Muon in Gong et al. (2025), particularly beyond the empirical characterization of curvature approximation accuracy?
Reviewer_rweYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether Muon can be understood as a Kronecker-factored curvature approximation without a separate curvature accumulator. Using the polar decomposition identity, the authors interpret Muon as replacing the input curvature factor with the momentum Gram matrix and the output factor with the identity. A signal–noise decomposition provides conditions under which the momentum Gram may track leading curvature directions, while distinguishing subspace alignment from spectral accuracy. The authors evaluate these approximations against a sequence-level generalized Gauss–Newton reference in a modified GPT-2, with additional measurements on ViT-Tiny and Gated DeltaNet. Controlled quadratic and nonlinear experiments examine failures associated with estimation noise, unequal output scales, and output coupling. The contribution is primarily diagnostic rather than a new optimizer.

Critical strengths

1.The GGN measurements provide useful evidence of the curvature information retained by Muon. Distinguishing subspace alignment from spectral accuracy clarifies what momentum-based preconditioning captures and misses. 2.SOAP is a relevant strong comparator. The controlled experiments distinguish estimation noise, output-scale mismatch, and output coupling, while supplementary architectures broaden the empirical evidence.

Critical weaknesses

1.The contribution beyond existing Muon–Shampoo and curvature interpretations is mainly descriptive. The paper does not establish that its diagnostics predict effective corrections in realistic training, leaving their practical significance uncertain. 2.The connection between curvature accuracy and optimization remains confounded by different training trajectories. Single-seed window sweeps and learning-rate optima at search boundaries further limit the robustness of the evidence.

Questions for authors

1.Can the authors validate one actionable prediction of the diagnostics—for example, whether output-scale correction benefits query/key projections more than the comparatively uniform output/down projections? 2. Do the main conclusions hold when curvature estimates are compared at identical weights and update momentum, and when decisive experiments are repeated across seeds with expanded learning-rate grids?

Reviewer_moWXOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies Muon as an accumulation-free Kronecker-factored curvature approximation, interpreting the momentum Gram as an input factor and the identity as an output factor. It develops a signal–noise decomposition of the momentum Gram, compares the implied factors with a sequence-level generalized Gauss–Newton reference in GPT-2, and includes additional measurements on ViT-Tiny and Gated DeltaNet. Controlled quadratic and nonlinear experiments examine failures associated with input estimation, output scales, and output coupling.

Critical strengths
  1. The analysis distinguishes leading-subspace recovery from spectral accuracy, and unequal output scales from output coupling. The measurements reveal differences across modules, including greater output-scale heterogeneity in query/key projections.
  2. The controlled experiments make a useful effort to separate different failure modes. In particular, the linear-regression experiment varies input higher moments while preserving covariance, and the output experiments vary diagonal scales and coupling separately.
Critical weaknesses
  1. The significance of the diagnostics for understanding Muon remains insufficiently established. Even with the reported experimental results, it is unclear how much they advance our ability to explain, predict, or improve Muon’s behavior. Section 5.3 provides some evidence linking spectral mismatch to loss, but the window experiments leave curvature accuracy entangled with momentum and trajectory effects. The stronger conclusion that the measurements “indicate which correction each module needs” is not validated through targeted corrections on the real networks.
  2. The theoretical novelty appears limited. The polar-update identity is already known, and the signal–noise decomposition in Eq. (6) follows from standard second-moment calculations. The additional expression μ⊤μ=RAR\mu^\top\mu=RAR in Eq. (8) does not establish leading-curvature alignment without further restrictions on the residual-dependent AA. Although the authors explicitly describe the explanation as conditional, the analysis does not establish sufficiently concrete conditions on AA to make this mechanism predictive.
  3. The curvature interpretation relies on several approximations, including the Kronecker model, the observed-label moment correspondence C≈RC\approx R, and stationary local-quadratic assumptions. The exact polar identity does not justify these approximations, and agreement in leading subspaces does not establish accuracy of the resulting preconditioned update. A direct comparison with reference curvature whitening may help clarify their combined effect on update quality.
Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.