Second-order and orthogonal-preconditioned optimizers (Shampoo, SOAP, Muon) deliver strong speedups at pretraining scale yet fail catastrophically in low-rank (LoRA) fine-tuning. This paper's contribution is the exact mechanism of that failure—two amplification laws—not an optimizer. Across five model families (GPT-2, Llama-3.2, Qwen3, Phi-4-mini, Mistral-7B; 124M–7B) and two data domains, the covariances that spectral preconditioners invert are severely rank-deficient: median effective rank 1.1–5.5 against dimensions 768–4096. We prove two exact laws: with a constant eigenvalue floor ϵ the amplification equals ϵ−2α exactly—independent of learning rate, data, and spectrum—and with a relative ridge γλmax it scales as (γλmax)−2α, forcing the ridge to Θ(1): no interior sweet spot exists. A pre-registered program of 575 runs verifies the laws to seven significant digits and shows free inversion diverges at every learning rate in LoRA and full-parameter training, with the divergence horizon exceeding 30-step calibration probes. The theory identifies the eigenbasis as the only stable channel of spectral information and predicts its own repair: discarding eigenvalue magnitudes while keeping the eigenbasis. The resulting rotation-only variant (SOAP-X)—Muon's principle at pretraining scale, reached here as the theory's prediction for deficient flows—matches or beats AdamW, Muon, and capped-inversion baselines in 6 of 8 configurations, including full-parameter training and a 5000-step horizon; on MMLU-5-shot the diverged optimizer's checkpoint collapses to random-level accuracy (24.89–25.28%) while every stable optimizer finishes 32–43 points above it. A 7B boundary check confirms the cold-start law is dimension-free and finds SOAP-X itself diverging mid-training—a pre-registered miss that three diagnosis blocks then trace to fp32 GPU eigendecomposition returning non-orthogonal eigenvectors on these near-singular covariances; one QR re-orthogonalization per refresh repairs it, verified across seeds and a 5× horizon at 7B. The laws apply to rank-deficient gradient flows (fine-tuning), not pretraining, where covariances fill in and spectral methods succeed.
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.Reviewer_qhT8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
The paper argues that second-order and orthogonal-preconditioned optimizers (Shampoo, SOAP, Muon) fail in low-rank (LoRA) fine-tuning, and that its primary contribution is the theory explaining this failure.
It reports that the covariances these preconditioners invert are severely rank-deficient, and proves two exact amplification laws, one for a constant eigenvalue floor and one for a relative ridge.
The theory identifies the eigenbasis as the only stable channel of spectral information and predicts a rotation-only variant (SOAP-X) that discards eigenvalue magnitudes, which matches or beats AdamW and Muon in most configurations.
Empirically, a divergence of SOAP-X at the largest scale is fixed by QR re-orthogonalization.
Critical strengths
Quantified rank deficiency of the inverted covariances.
This work reports the participation ratio reff of the instrumented covariances at step 30, across every tested condition.
Across five model families (124M–7B) and two data domains, the median effective rank is 1.1–5.5 against dimensions 768–4096, and half of all LoRA-side covariances have λmin exactly 0.0 in float32.
The authors state that the object that spectral preconditioners invert is overwhelmingly null space or numerical tail.
Critical weaknesses
The central claim is stated more broadly than the paper's own SOAP-AC results support.
The Conclusion states that what is new is "the exact account of why every eigenvalue-retaining family member must fail here".
However, SOAP-AC is defined as rescaling "the update of SOAP(α) back to the α=0 RMS", which removes the amplification while retaining the inversion (Appendix A).
On Phi-4-mini, SOAP-AC(1/8) has 0 of 3 runs divergent over 5000 steps (Table 6).
Appendix A also describes SOAP-AC(1/2) as stable.
Questions for authors
Definition of SOAP-AC:
We noticed two descriptions of SOAP-AC in the paper. Section 3 states that "SOAP-AC(α) caps the inverted spectrum at ϵ−α".
Appendix A.1 states that SOAP-AC "rescales the update of SOAP(α) back to the α=0 RMS".
Could the authors clarify whether these two descriptions refer to the same operation, and which one was used in the reported experiments?
Clarification on the form of the ridge law:
The abstract states "We prove two exact laws", the second being that with a relative ridge γλmax the amplification "scales as (γλmax)−2α".
Theorem 3 states "A≈(γλmax)−α× (opposite-side scale)", and the predictions in Appendix A.5 are evaluated as (γλmax)−αλopp−α. Could the authors clarify which expression is the intended statement of the law, and whether it is meant to be exact or approximate? If the expressions are intended to coincide, a brief note on the conditions under which they do would help readers.
This paper explores instabilities arising from spectral preconditioning methods in rank-deficient gradient flows, particularly in the context of LoRA fine-tuning. The authors analyze the gradient covariance spectrum and derive amplification formulas for "eigenvalue-floor" and "relative-ridge" preconditioners. Based on this analysis, they advocate a "rotation-only" update approach—retaining the covariance matrix eigenvectors while discarding eigenvalue magnitudes. Through extensive experiments across several language model families, the study examines issues such as spectral deficiency, optimizer divergence, sensitivity to learning rates, training duration, and numerical failures during the eigenvalue decomposition process. Furthermore, the paper demonstrates that the numerical instability observed in 7B-parameter models can be resolved through QR re-orthogonalization.
Critical strengths
This is an important, insightful study of optimizer instability. The paper explores a practically significant failure mode that arises when using spectral preconditioning during low-rank fine-tuning. Using covariance measurements and controlled numerical diagnostics, the research explains why certain optimizer configurations become unstable and offers valuable, in-depth insights.
The paper presents an extensive empirical study, accompanied by a highly informative failure analysis. The experiments encompass a diverse range of model architectures, data domains, optimization settings, and training durations. Of particular value is the analysis of failure cases involving 7B-parameter models; the authors find that eigenvectors lose orthogonality during eigenvalue decomposition in finite-precision arithmetic and propose a targeted numerical correction to address this issue.
Critical weaknesses
The study's core theoretical proposition is narrower in scope than the text suggests, and its innovation is relatively limited. When spectral scaling factors converge to the same lower eigenvalue bound, a precise amplification law emerges; in this regime, parameter updates are effectively equivalent to a constant rescaling of the baseline updates. This "precision" holds only under specific "cold-start" configurations and does not generalize to typical rank-deficient gradient flow scenarios. More importantly, the claim that "no stable interior ridge parameter exists" relies on specific scale-dependent assumptions and empirical numerical settings. The paper fails to prove that spectral preconditioning is generally infeasible under rank-deficient conditions, which significantly diminishes the value of its primary theoretical contribution.
The paper insufficiently explains the link between algebraic amplification and training instability. A large operator amplification factor does not necessarily imply a large update magnitude along the actual gradient direction; furthermore, even if the update magnitude is large, it does not automatically lead to divergence in the optimization dynamics. Although empirical evidence shows that the tested implementations do fail, the theory does not provide a sufficiently general characterization of stability. Moreover, the "rotation-only" design principle closely resembles existing methods, limiting its innovation to practical algorithmic implementation.
Questions for authors
Setting aside the specific "cold-start" scenario where both spectral factors equal their respective numerical lower bounds, what is the strongest theoretical conclusion this paper can offer? Can the authors distinguish between "worst-case operator amplification" and the amplification that occurs during the actual gradient update process?
Under what specific assumptions does the "relative-ridge" based analysis rule out the existence of a stable, interior value for the ridge parameter? Given that the lower bound presented in the paper depends on the learning rate, parameter scale, and the covariance spectrum, why should the conclusion (i.e., γ=Θ(1)) hold true beyond the specific configurations tested?
Does this paper establish a formal derivation linking "spectral amplification" to "divergence in training dynamics," or is this connection based entirely on empirical observation? If the latter, then the paper’s core theoretical claims should be stated with more restricted scope.
What additional theoretical or practical insights does SOAP-X provide compared to existing "rotation-only" or "orthogonalized-update" methods? Specifically, which experimental results demonstrate that "preserving the covariance eigenbasis" is superior to "employing orthogonalization strategies that do not rely on the covariance"?
The submission studies spectral preconditioning in rank-deficient gradient regimes, particularly LoRA and full-parameter language-model fine-tuning. It defines a family that applies inverse covariance eigenvalue powers to an Adam-normalized signal in a rotated basis. The paper claims an exact amplification law under a constant eigenvalue floor, a relative-ridge law that rules out a stable nontrivial preconditioner, and a resulting justification for discarding eigenvalue magnitudes. A substantial experimental program compares the resulting variants with AdamW and Muon, examines calibration horizons, and investigates a separate numerical eigensolver failure at 7B scale.
The topic is relevant, and a few observations are useful. However, the principal conclusions exceed what the definitions, proofs, and comparisons establish. My recommendation is driven by two issues: the experiments do not establish the claimed failure attribution to canonical SOAP/Shampoo, and the main theoretical claims contain consequential errors. The conditional identity for an entirely floored cache is valid, but it does not imply the stated general result for rank-deficient covariances.
Critical strengths
The paper examines a practically relevant question: how covariance estimation, numerical floors, and delayed basis refreshes affect optimization during fine-tuning. Its separation of update magnitude from direction is a useful diagnostic perspective.
The reported experiment is extensive. This work reports 575 runs across several models and settings, includes longer training horizons, and documents unsuccessful predictions and boundary cases. These observations could support a useful empirical study after the scope of the conclusions is corrected.
Critical weaknesses
W1 The algorithm definitions do not support the attribution to canonical SOAP and Shampoo
Section 3 and Appendix A.1 define the analyzed update as
Δ=−ηPfl(DR)−αufl(DL)−αQ⊤,u=v+ϵam
Here the inverse eigenvalue factors are applied to an already Adam-normalized signal. Canonical SOAP instead applies Adam in the covariance eigenbasis and projects the resulting update back; its algorithm does not add these inverse eigenvalue factors after Adam normalization [1, Algorithm 3]. Consequently, the rotation-only core of the submission’s α=0 member is already present in SOAP. Exact equivalence still depends on the treatment of moments, refreshes, and normalization.
Likewise, selecting the Shampoo exponent α=1/4 does not make this hybrid update equivalent to Shampoo. The signal to which the preconditioner is applied matters. Established Distributed Shampoo implementations also include layer-wise learning-rate grafting, which changes precisely the absolute-update-scale behavior at issue here [2, Section 2.4]. These implementation choices must be specified before failures of the hybrid can be attributed to Shampoo.
The problem is the transfer of its failures and repairs to established algorithms without an explicit equivalence argument or a direct comparison preserving their relevant safeguards. The manuscript needs a precise mapping from each named variant to its update rule and to a specified canonical implementation. The present evidence supports conclusions about the tested variants; it does not establish a general failure of SOAP or Shampoo.
This concern does not depend on whether the authors claim to introduce a new optimizer. It affects the claimed explanatory contribution and the interpretation of the experiments.
W2 The main amplification theorem omits an essential support condition
Theorem 2 states that, under A1–A3, the first nonzero gradient after a covariance refresh has amplification exactly ϵ−2α (p. 4, lines 167–172). Rank deficiency alone is insufficient for this equality.
For a fixed nonzero normalized signal u, let ri=max(λiR,ϵ) and lj=max(λjL,ϵ). Orthogonal invariance gives the actual identity
A2=∑i,j∣ui,j∣2∑i,j∣ui,j∣2ri−2αlj−2α
For α>0, this implies A≤ϵ−2α. Equality requires every nonzero entry of u to lie in a row and a column for which the corresponding floors both bind. The existence of zero or small eigenvalues elsewhere does not impose this condition.
A simple counterexample uses the smooth, nonnegative objective f(W)=W112/2 on 2×2 matrices with W11(0)=1. The first gradient is E11. With zero-initialized EMA statistics and β=0.95, refresh the eigensystems after this gradient and retain them for ten steps. The cached covariances are then
L=R=0.05E11.
All gradients and consistently maintained moments remain supported on entry (1,1). Choose a sufficiently small learning rate so that the next gradient is nonzero. At that first nonzero gradient after the refresh, u=aE11 for some a=0, and
A=0.05−2α=20≈4.4721 when α=1/4.
The theorem instead predicts 106 for ϵ=10−12. Both covariance matrices are exactly rank deficient throughout. Thus this example falls within the stated assumptions and the expressly included full-parameter setting.
Appendix A.3 proves a narrower identity: if both cached covariance spectra are entirely zero when refreshed, and a nonzero update occurs before the next refresh, then both floors bind everywhere and the ratio is indeed ϵ−2α. I agree with that conditional identity. The main theorem and applicability boundary must either adopt those conditions or supply a support-dependent result. Machine-precision agreement in that special case cannot validate the broader theorem.
W3 The EMA effective-window argument is not a valid steady-state rank bound
The finite-history statement
rank(Ct)≤min(d,rt)
is correct. Appendix A.4 then replaces the number of accumulated summands by W=(1−β)−1 in an asserted steady-state null-fraction bound (p. 13, lines 664–689). An EMA decays old contributions; it does not exactly truncate them after W steps.
For a concrete counterexample, take r=1,d=1000, and β=0.95, and repeatedly cycle through the 40 unit row gradients e1⊤,…,e40⊤. At the end of each complete cycle, the 40 positive covariance eigenvalues are proportional to 1,β,…,β39. Their participation ratio is
reff=(1−β)(1+β)(1+β40)(1−β40)≈30.1176>W=20.
This holds at every completed cycle after zero initialization and in the limiting steady cycle. The exact rank is 40, and the null fraction is 0.96, below the claimed lower bound 1−rW/d=0.98. The covariance remains rank deficient throughout.
If “after W EMA steps” in the main theorem means literally t=W, the finite-time rank inequality remains valid. The error is the appendix’s extension to a steady-state effective window and its interpretation of W as a participation-ratio ceiling. The measured small participation ratios may be useful observations, but they do not prove that ceiling. Moreover, a low participation ratio by itself does not establish that any eigenvalue is below a specified absolute numerical floor.
W4 The ridge and stability arguments do not establish the claimed impossibility result
Theorem 3 and Appendix A.5 claim that every stable ridge must make the preconditioner uniform, so stable nontrivial spectral preconditioning is impossible (pp. 4 and 13–14). I do not find a valid derivation of this conclusion.
First, the calculation drops the opposite-side scale. Write the paper’s one-sided expression as A=(γλmax)−αS, where S is that scale, and set w=∥W∥rms. Even accepting the proposed condition ηcA≲w, rearrangement gives
γ≳λmax(ηcS/w)1/α
The displayed expression in the paper omits S. If the signal occupies directions where both relative floors bind and both sides use the same γ, the corresponding expression is instead
γ≳λmaxRλmaxL(ηc/w)1/(2α)
These are materially different dependencies and cannot be interchanged without justification.
Second, substituting the stated ranges η/w∈[10−3,10−2], λmax∈[10−5,10−4], and α=1/4 into the paper’s printed one-sided expression gives approximately 10−8–10−3 when c=1. That calculation does not establish the claimed order-one threshold. A corrected two-sided calculation may yield different thresholds, but the required calculation is missing.
Third, an order-one lower bound is insufficient to prove uniformity. Uniformity follows at γ≥1; for 0<γ<1, eigenvalues above γλmax remain distinct. In particular, the reported threshold near 0.6 does not itself establish loss of all spectral information.
Finally, the update-to-weight-scale inequality is not derived as a necessary condition for optimization stability. Independence of the ratio A from the learning rate does not make the actual displacement ηA∥u∥F independent of the learning rate. Appendix A.6 explicitly gives a nonempty small-rate “stable set” immediately after asserting that lowering the rate cannot restore stability. The experiments can support a statement about impractically small rates in the tested configurations, but the one-step identity does not establish impossibility of stable inversion or the stated divergence-time scaling without additional assumptions.
These errors matter because Appendix A.6’s optimality claim for the rotation-only member relies on Theorem 3. I recognize that the main text explicitly limits rotation safety to removal of the eigenvalue-amplification channel; my concern is with the stronger impossibility and optimality claims.
W5 Inconsistent definitions obstruct interpretation and reproduction
The presentation problems affect the scientific content. Section 3 describes SOAP-AC as capping the inverted spectrum at ϵ−α, which is already implied by the constant eigenvalue floor. Appendix A.1 instead defines SOAP-AC through rescaling the complete update to the α=0 RMS. Those operations are different.
Similarly, Section 3 describes SOAP-X as matching AdamW’s update RMS, while the formal family defines it through α=0. The relationship among that definition, any separate AdamW reference, and the normalization ablations needs to be specified. The main and appendix versions of the amplification theorem also have different scopes, as detailed above.
These inconsistencies make it difficult to determine which object each experiment and claim concerns. The manuscript needs one authoritative update definition for each variant and consistent references to those definitions throughout. The large experiment count and precise checks of the special cold-start identity do not resolve the mismatch between the tested algorithms and the broader conclusions.
[1] Vyas et al, SOAP: Improving and Stabilizing Shampoo using Adam
[2] Shi et al, A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scale.
Questions for authors
Canonical algorithms. Which exact published SOAP and Shampoo implementations are represented by the tested variants? Please identify differences in the preconditioned signal, moment handling, normalization or grafting, and basis-refresh procedure. Which existing comparisons establish the same failures for the canonical algorithms with their relevant safeguards retained?
Scope of Theorem 2. Is the intended theorem restricted to an entirely floored stale cache? If so, can the main statement and full-parameter applicability claim be narrowed accordingly? Otherwise, what additional assumption excludes the rank-deficient quadratic counterexample in W2?
EMA bound. How does the steady-state bound in Appendix A.4 handle the repeated 40-direction example in W3? Please distinguish exact rank, participation ratio, and the number of eigenvalues below an absolute floor.
Ridge and stability. Can the ridge calculation be supplied with both sides’ scales retained, and can the necessary stability condition be justified? What establishes a threshold of at least one, rather than merely a threshold of order one, and what rules out an interior ridge preserving relative spectral information?
Update normalization. What are the exact operations used for SOAP-AC and SOAP-X in each reported experiment, and how do they reconcile Section 3 with Appendix A.1? This clarification is necessary to interpret whether the observed improvement comes from removing the extra inverse factors, controlling overall amplitude, or another implementation change.
From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.