acceptodds
Under review as a conference paper at ICLR 2027

FISMO: FIsher-Structured Momentum-Orthogonalized Optimizer

Abstract

Training large-scale neural networks requires solving nonconvex optimization problems, where the choice of optimizer strongly affects both convergence behavior and computational efficiency. While adaptive methods like Adam have long dominated practice, the recently proposed Muon optimizer achieves superior performance through orthogonalized momentum updates that enforce isotropic geometry with uniform singular values. However, this strict isotropy discards potentially valuable curvature information encoded in gradient spectra, motivating optimization methods that balance geometric structure with adaptivity. We introduce FISMO (FIsher-Structured Momentum-Orthogonalized), which generalizes isotropic updates to incorporate anisotropic curvature through Fisher information geometry. By reformulating the optimizer update as a trust-region problem constrained by a Kronecker-factored Fisher metric, FISMO achieves structured preconditioning that adapts to local loss landscape geometry while maintaining computational tractability. We establish convergence guarantees for FISMO in nonconvex settings, proving an deterministic rate and an stochastic rate for the expected gradient norm. Moreover, we theoretically show that FISMO can achieve sharper finite-time contraction than Muon-style isotropic orthogonalization through a pre-asymptotic analysis. Empirical evaluation on image classification and language modeling benchmarks demonstrates that FISMO achieves superior training efficiency and final performance compared to established baselines.

Primary area:
optimization
Keywords:
Deep Learning Optimization, Fisher Information Geometry, Orthogonalized Momentum

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_KSbyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper designs a new algorithm for neural network optimization. The main idea is to replace the Frobenious norm constraint in the factored NGD update with the spectral norm constraint. Based on this desing choice the authors develop FISMO algorithm, analyze its convergence and perform preliminary numerical experiments.

Critical strengths

The main strengths are useful closed-form update rule in Theorem 4.2 and promising preliminary experiments.

Critical weaknesses

The proofs in this paper contain many mathematical mistakes. The most severe mistakes are the following:

  • Theorem 4.1 is incorrect: function J(P,Q)J(P,Q) is not convex in P for fixed Q and not convex in Q for fixed P. The proof of this statement is missing, and a simple 1-d counterexample is available: $ J(p,q) = (f + \mu)/(pq) + \ln p + \ln q. $ Choosing q=1q = 1 and f+μ>0f + \mu > 0 implies (J(p,q))p′′=[2(f+μ)−p]/p3≱0(J(p,q))_p'' = [2(f+\mu) - p] / p^3 \not\geq 0.

  • The proof of Lemma F.8 is incorrect: the inequality (65) is plugged into (64) with an opposite sign. Similarly, the proof of Lemma F.9 is incorrect: the inequality on lines 1649-1650 has wrong direction.

  • The proof of Lemma F.10 is incorrect. In particular, the expectation on lines 1671-1673 can be non-zero: it is not justified that G~t\tilde{G}_t is an unbiased estimator of H~t\tilde{H}_t, since GtG_t, PtP_t, and QtQ_t may be correlated. This issue further propagatesn to Lemma F.13, Theorem F.14 and Theorem 5.8. The relation on lines 1652-1656 may be also incorrect even in the noiseless setting, since C~t\tilde{C}_t is an EMA of H~t\tilde{H}_t and G~t\tilde{G}_t is just the current transformed gradient.

The errors above invalidate the key steps in the convergence analysis of Algorithm 1. Consequently, the main convergence results are not established by the proofs in the paper.

Besides, the main design choice of this paper is replacing frobenius norm in the factored NGD (line 192) with the spectral norm. However, while the original Frobenius norm constraint follows from Fisher geometry, the choice of spectral norm here is not adequately justified. In particular, the justification on lines 194-200 is weak: Frobenius norm already bounds the "largest directional amplification", and there is no explanation of why the "updates that are less dominated by large singular directions" are preferable in this setting. In contrast, for Muon, there are many theoretical results showing, that orthogonalization can be beneficial under different structural assumptions.

Questions for authors

n/a

Reviewer_2xnrOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes FISMO, a matrix optimizer that combines Kronecker-factored empirical Fisher preconditioning with Muon-style orthogonalized momentum. The authors derive the update from a preconditioned spectral-norm trust region, provide nonconvex convergence and local contraction analyses, and evaluate the method on a 124M-parameter GPT-2 model and CIFAR-10.

Critical strengths
  1. Combining Fisher-structured preconditioning with momentum orthogonalization is an interesting direction for matrix optimizers. The trust-region/LMO formulation provides a geometric interpretation of the update.
  2. The paper is generally well organized, and the main algorithm and its motivation are explained clearly.
Critical weaknesses
  1. The novelty and theoretical claims need stronger support. The KL/log-det Kronecker approximation and coupled preconditioner equations in Theorem 4.1 very closely match the idealized KL-Shampoo formulation of https://openreview.net/forum?id=pQQuC1nIQq (ICLR 2026), which is neither discussed nor compared against. The main difference appears to be orthogonalization in the whitened space, but its benefit is not isolated. The change from the Fisher-induced Frobenius-norm constraint to a spectral-norm constraint in Section 4.1 also needs clearer motivation. I think a clear discussion with this prior work is required.
  2. The experimental evidence is too limited to establish practical benefits. The language-model experiment uses only 2,000 steps on a 124M-parameter model, and the paper does not report essential details such as token counts, batch sizes, learning-rate schedules, or baseline tuning. FISMO also requires matrix inverses/inverse square roots and orthogonalization, yet wall-clock time and memory overhead are not reported. The claimed training efficiency and scalability are therefore not supported by the experiments.
Questions for authors
  1. Could the authors clarify the precise difference from KL-Shampoo and provide a KL-Shampoo baseline and an ablation without orthogonalization?
  2. Could the authors provide complete experimental configurations and longer language-model runs, along with wall-clock/memory comparisons and sensitivity studies for damping and preconditioner update frequency? Also, I think a proper experiment is needed to examine the performance of the method.
Reviewer_KA6VOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FISMO combines two-sided matrix preconditioning with momentum orthogonalization. The paper motivates the preconditioners through a Kronecker approximation of an empirical Fisher matrix, derives a closed-form linear minimization oracle in a whitened spectral-norm ball, and proposes EMA updates with damping and trace normalization for the FISMO algorithm. It claims deterministic O(T−1/2)O(T^{-1/2}) and stochastic O(T−1/4)O(T^{-1/4}) convergence rates, together with a pre-asymptotic advantage over isotropic updates. Experiments report favorable loss or accuracy curves on GPT-2/OpenWebText and SimpleDLA/CIFAR-10.

Critical strengths
  1. The paper is clearly presented and easy to understand.
  2. The optimizer design is overall well motivated. The algorithm is derived from Optimal Kronecker Approximation of Fisher matrix(theorem 4.1) and the optimal solution of the optimization problem 3 that seeks the steepest descent direction within a trust region (theorem 4.2).
Critical weaknesses
  1. Several issues exist in the proof/theorem.
    ∙\bullet According to FISMO, Pt,QtP_t,Q_t depend on the minibatch gradient GtG_t used in the momentum update. Thus, although Gt−∇L(Wt−1)G_t-\nabla L(W_{t-1}) is conditionally centered, its whitened version need not be. Specifically, E[N~t∣Gt−1]=0\mathbb{E}[\tilde N_t|\mathcal G_{t-1}]=0 need not hold. Lemma F.10 nevertheless use this equality to eliminate the cross term to derive eqn. 92.
    ∙\bullet Theorem 4.1’s blockwise convexity claim is false: consider the scalar case, and suppose G=Q=1G=Q=1. Then J=(1+μ)/p+log⁡pJ=(1+\mu)/p+\log p has second derivative (2(1+μ)−p)/p3<0(2(1+\mu)-p)/p^3<0 for p>2(1+μ)p>2(1+\mu).
  2. Compared to Muon, FISMO has extra computation cost even though the author propose to update the preconditioning matrix Pt,QtP_t, Q_t every 10 iterations. It's more reasonable to also provide training loss/validation accuracy vs running time curve to verify the effectiveness of FISMO. Besides, the extra memory consumption could also be prohibitive in practice.
Questions for authors
  1. The motivation to use spectral norm in eqn. 3 instead of frobenius norm in line 192 is still unclear to me, which needs more explanation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.