acceptodds
Under review as a conference paper at ICLR 2027

Spectral Deflation for Factorization-Free Matrix Filtering in Muon and Semidefinite Programming

Abstract

GPU implementations of the Muon optimizer and of first-order semidefinite programming (SDP) solvers share one computational pattern: a matrix factorization is replaced by a fixed-depth polynomial filter applied after normalization. When a few dominant spectral components carry most of the input's scale, normalization pushes the remaining spectrum toward zero, where the filter is least accurate. We propose spectral deflation: estimate the dominant components, remove them, filter the normalized residual with the unchanged filter, and restore them. Deflation preserves the target matrix function, and we prove that it strictly reduces the finite-step error of the classical Newton–Schulz family. Implemented with batched randomized SVD for Muon and warm-started subspace tracking for ADMM, deflation consistently improves GPT-2 pretraining over the corresponding Muon baselines with both the Newton–Schulz and Polar Express mappings, also in wall-clock time, and lowers the KKT residuals of factorization-free ADMM on large-scale SDPs within a similar projection time.

Primary area:
optimization
Keywords:
Deflation, Matrix Function, Muon, Large Language Model, Semidefinite Programming, ADMM, Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kt5LOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes spectral deflation to improve the accuracy of fixed-depth polynomial filtering methods, particularly Muon and first-order SDP solvers. The motivation is that, when a few dominant singular values account for most of the Frobenius norm, normalization pushes the remaining singular values close to zero, making it difficult for a fixed number of Newton-Schulz iterations to bring them close to one. To address this issue, the authors propose removing the dominant singular components, normalizing the residual matrix, and then reintroducing the removed components before applying the polynomial filter. They also provide theoretical results showing that spectral deflation reduces the finite-step approximation error for the classical Newton-Schulz family. The authors evaluate the proposed method on GPT-2 Large pretraining using FineWeb, comparing Deflated Muon against Muon with both Newton-Schulz and Polar Express mappings. They report improvements in validation loss and wall-clock performance, along with several ablation studies. They further demonstrate the effectiveness of spectral deflation in first-order SDP solvers, showing lower KKT residuals on large-scale SDP problems under similar projection-time budgets.

Critical strengths

The motivation is interesting and well justified. For example, in Muon, normalization by the Frobenius norm can push smaller singular values close to zero when a few dominant singular values account for most of the matrix Frobenius norm. Consequently, these small singular values may remain significantly below one even after a fixed number of Newton-Schulz iterations. To address this issue, the authors propose removing the dominant singular components, normalizing the residual matrix, and then reintroducing the removed singular directions. The proposed idea is simple and intuitive, and it can be incorporated into existing polynomial filtering methods without modifying the polynomial itself.

Critical weaknesses
  • The performance gains of Deflated Muon appear relatively modest. For example, in Figure 3(a), while Deflated Muon achieves a lower validation loss than standard Muon with the Newton-Schulz mapping, Muon with Polar Express achieves the best validation loss among all four methods when comparing their respective optimal learning rates. This raises concerns about the practical benefits of deflation, especially considering the additional computational overhead of Algorithm 3. Furthermore, several experiments appear to use a fixed learning rate of 0.01, including Figure 3(b) and possibly Tables 1 and 2. However, Figure 3(a) suggests that 0.01 is not the optimal learning rate for any of the methods. In particular, the performance gap between Muon and Deflated Muon is relatively large at this learning rate, whereas the gap becomes smaller or even reverses near their respective optimal learning rates. Therefore, I believe that a fair comparison should evaluate each method with its individually optimized hyperparameters, particularly learning rates, while accounting for the additional computational overhead of deflation. Without such comparisons, it remains unclear whether Deflated Muon provides meaningful practical benefits over well-tuned baselines.
  • It is unclear whether the proposed algorithm needs to be as complicated as it currently is. Figure 7 in Appendix C shows that the singular value spectra are often dominated by a small number of large singular values. Furthermore, Table 5 in Appendix F shows that using a very small deflation window, with an average deflation rank of approximately 1.5, already achieves performance comparable to that of larger deflation windows. This raises the question of whether a much simpler approach, such as estimating only the top singular triplet using power iteration and removing that component before normalization, could achieve similar performance without requiring the batched randomized SVD in Algorithm 3. Such a baseline would help clarify whether the additional computational complexity is necessary. Additionally, the necessity of the deflation gate is not sufficiently justified. Since the gate is evaluated after estimating the singular values using Algorithm 3, most of the computational cost has already been incurred, regardless of whether deflation is applied. While the gate may help avoid unnecessary or potentially harmful deflation, the paper does not provide an ablation showing what happens when the gate is removed. It would be helpful to understand why this gate is necessary and how much it contributes to the overall performance.
Questions for authors
  1. Could the authors compare the methods using their individually optimized hyperparameters, particularly learning rates and the number of polynomial iterations, under equal training-time budgets? Figure 3(a) suggests that the learning rate of 0.01 used in Figure 3(b) is suboptimal for several methods. It would also be helpful to clarify which learning rates were used in Tables 1 and 2 and Figure 4, and whether they were tuned separately for each method.
  2. Table 5 suggests that removing only a very small number of dominant singular components is sufficient to achieve most of the performance improvement. Could the authors clarify how the proposed method compares against a simpler rank-1 deflation approach, where only the top singular triplet is estimated using power iteration? Would such an approach achieve comparable performance with lower computational overhead?
  3. What happens if the deflation gate in Algorithm 1 is removed and deflation is always applied? Since the gate is evaluated after running Algorithm 3, the computational cost of estimating the dominant singular components has already been incurred. Could the authors explain why the gate is necessary and provide an ablation comparing performance with and without it?
  4. Theorem 1 appears to analyze the classical Newton-Schulz polynomial rather than the retuned coefficients used in practical Muon. Could the authors clarify whether a similar error reduction guarantee holds for the actual Muon coefficients?
  5. Figure 4 extends the experiments to approximately the Chinchilla-optimal training budget. However, modern LLM pretraining often uses several times the Chinchilla-optimal number of tokens. Is there any evidence that the advantage of Deflated Muon persists under substantially longer training horizons, especially when each baseline is evaluated with its optimal hyperparameters?
1 comment

We thank the reviewer for the constructive suggestions. NS/PE denote Newton–Schulz/Polar Express; D denotes Deflated Muon; Δ = D − Muon.

Q1. Tuned hyperparameters and equal time

Tables 1–2 and Figures 3(b), 4 use lr = 0.01 for all methods; Figure 3(a) uses the common grid {0.002, …, 0.01}. We agree that lr = 0.01 alone does not establish an advantage over tuned baselines.

We now select learning rate and depth separately within the tested grids. Each method receives Muon L = 5's 1905-step time budget, including deflation overhead; step counts follow measured timings, with schedules adjusted accordingly:

MethodLlrValidation loss
Muon NS60.0063.3560
D NS50.0063.3548
Muon PE50.0053.3472
D PE30.0073.3403

D PE has the lowest loss, 0.0069 below the best baseline. NS losses are comparable, with D using five rather than six iterations. On GPT-2 Large, 4 × H200, step time is 1763 ms for Muon L = 5 and 1782 ms for D L = 5 (+1.1%), faster than Muon L = 7.

Q2. Rank-1 simplification and unknown head width

Table 5's small ranks are a posteriori statistics, available after spectral estimation. The head width is unknown beforehand and varies by matrix and training stage. Batched randomized SVD estimates the leading components and selects the rank online without prescribing it in advance.

We tested three batched power-iteration steps for the top triplet, with rank-1 deflation always on. For stability, the residual is (I−uu⊤)M(I−vv⊤)(I-uu^\top)M(I-vv^\top) before normalization:

Mapping, lr = 0.01MuonD, randomized SVDRank-1
NS3.38133.35943.3575
PE3.36753.36003.3535

Rank-1 recovers the gain, but its measured overhead is 24 ms/step versus 20–25 ms for randomized SVD. Logging all 216 matrices shows mean active ranks of 10.0/19.1/3.0/7.9/8.4/9.4 for Q/K/V/O/MLP input/output, with an overall range of 1–31. For K, the early-to-late rank increases from 14 to 28. We retain randomized SVD for adaptive rank selection at comparable measured cost; rank-1 is a useful simpler comparison.

Q3. Gate ablation

The gate does not avoid estimation cost; it selects when and how much to deflate. Always-on deflation uses:

k=\min\\{\max\\{\\#\\{s_i > 0.1s_1\\},1\\},r-1\\}.

Mapping, lr = 0.01Gated lossAlways-on lossMean rank, gated/always-on
NS3.35943.35788.8/19.7
PE3.36003.36089.5/23.3

The loss changes (−0.0016/+0.0008) are comparable to observed run-to-run variation. The gate primarily limits deflated rank in these runs; this ablation does not show it is necessary for the gain.

Q4. Practical coefficients

The floor statement is polynomial-independent; Eq. (10)'s strict ratio bound is specific to classical NS. With exact components under Theorem 1,

\alpha_k=\lVert M\rVert_F/\lVert R_k\rVert_F.$$ Muon uses $p(x)=3.4445x-4.7750x^3+2.0315x^5$, with p(1) = 0.701, so classical convergence to one does not apply. Define $$B_L(\ell)=\max_{x\in[\ell,1]}|1-p^{\circ L}(x)|.$$ Interval inclusion gives $B_L(\ell_{\mathrm{def}})\le B_L(\ell)$. For L = 5, $$B_5(\ell)\le\max\\{0.3182,1-453.0\ell\\}.$$ This envelope improves until saturation near ℓ = 1.5×10⁻³; it does not guarantee improvement in every direction. Directions whose baseline and deflated iterates stay in [0, 0.55] through the final iterate improve under the larger initialization. Across 24 recorded matrices, relative Frobenius error decreases from 0.583 to 0.423 with the retuned map, versus 0.879 to 0.799 for the classical quintic. Figures 2 and 10–11 already use the retuned map. ## Q5. Longer training GPT-2 Large at 10B tokens (Figure 4, lr = 0.01) ends at 2.9500/2.9274 for Muon/D NS (Δ = −0.0226) and 2.9382/2.9236 for PE (Δ = −0.0146). The gap does not shrink between the reported 1B and 10B settings at this learning rate. We also trained GPT-2 Small (124M) on 10B tokens, about 4× its compute-optimal budget, selecting each method's lr from {0.0025, 0.005, 0.01}: |Mapping|Muon|D|Δ| |---|---|---|---| |NS|3.2476|3.2419|−0.0057| |PE|3.2402|3.2387|−0.0016| Both retain lower observed losses at the longer relative horizon, with a smaller PE difference. We have uploaded the revised manuscript.
Reviewer_dCryOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Muon and recent GPU solvers for semidefinite programs (SDPs) both replace a matrix factorization with a short, fixed sequence of matrix polynomials. Muon uses it to approximate the polar factor of each momentum matrix. ADMM for SDPs uses it to approximate the matrix sign function, which gives the projection onto the PSD cone. Before the polynomials are applied, the input is divided by a norm. When a few large singular values (or eigenvalues) dominate that norm, all the other values are pushed close to zero. A fixed-depth polynomial cannot lift them to one in a few steps, so those directions are poorly handled.

The fix is simple. Estimate the few dominant components, subtract them, and normalize only the remainder. The small values are then scaled up by the ratio of the two norms, so the same polynomial handles them much better. The removed part is added back exactly, so the target function does not change. For Muon, the dominant components come from a batched randomized SVD that processes all same-shape momentum matrices at once. For ADMM, a subspace is tracked across iterations with one block step per projection. A gate turns deflation on only when the spectrum is head-dominated, and a padding constant keeps the head directions inside the filter's input range.

The theory (Theorem 1, Proposition 2) shows that, with exact deflation and the classical Newton-Schulz polynomials, deflation raises the smallest normalized value and gives a smaller worst-case error than the baseline. Experiments pretrain GPT-2 Large on FineWeb with 1B and 10B tokens. Deflated Muon reaches a lower validation loss than Muon for both the tuned Newton-Schulz map and Polar Express, across learning rates, momentum values, and iteration counts. On four max-cut and min-bisection SDPs from the Mittelmann set, with FP16 projections, ADMM with the deflated projection reaches a lower KKT residual for a similar projection time.

Critical strengths

S1. A simple, general idea that keeps the target exact. The paper identifies a shared weakness of fixed-depth polynomial filters: normalization by the whole matrix squeezes the small part of the spectrum. The fix is a wrapper that leaves the polynomial unchanged, so it works with Newton-Schulz, Polar Express, and the composite SDP filter alike. Unlike many Muon variants, it does not change the target update (Eq. 2). This makes it easy to adopt and easy to combine with other improvements to the polynomials.

S2. Careful engineering and an open account of the details. The two estimators fit the GPU setting well: batching momentum matrices by shape avoids kernel-launch overhead, and the warm-started subspace fits the slowly changing ADMM iterates. The appendix measures the estimation error against exact decompositions (Table 3, Figure 9), proves the lemmas in full, and reports cases where the gate does not fire or fires because of estimation error (Appendix F, attention K at step 1500). The ablations over window, threshold, and deflation cutoff (Tables 4 to 6) show the method is not sensitive to its own constants.

Critical weaknesses

W1. The language-model gains are small, come from single runs, and lack the controls needed to attribute them to deflation. This is my biggest concern.

  • No table or figure reports more than one seed or any error bar. The gap between Muon and Deflated Muon is a few hundredths in validation loss for Newton-Schulz and smaller for Polar Express. The same default configuration appears in several tables (Tables 1, 2, 4, 5, 6), and its reported loss differs from table to table by an amount that is not much smaller than some of the claimed gains. For Polar Express at the highest momentum value in Table 2, the two methods are equal. The caption still says Deflated Muon wins at every momentum value. The claim that three deflated Polar Express iterations beat five baseline iterations (Table 1) is also within this spread.
  • The window and threshold sweeps (Tables 5 and 6) change the number of deflated directions by a large factor, yet the loss barely moves. This is consistent with a robust method, but it is also what one would see if the differences were mostly noise. Several seeds per setting would separate the two.
  • There is no compute-matched baseline. Deflation adds a randomized SVD, an extra subtraction, and a reconstruction at every step. The obvious alternatives at the same cost are one or two more polynomial iterations, or a different polynomial design (for example CANS, which is cited). Table 1 stops at five iterations, so the reader cannot compare. The per-step overhead is also not reported as a number. The wall-clock claim rests on the curves in Figures 1, 3, and 4 only.
  • The evaluation uses one model (GPT-2 Large) on one dataset. Section 3.4 cites work showing that momentum spectra become less head-dominated during training, and Table 4 shows much of the benefit is already there when deflation is used for only the first part of training. It is not clear whether the gain holds for other sizes or architectures.
  • The SDP study is narrow. All four instances are max-cut or min-bisection relaxations. Only FP16 projections are tested, and only against the Kang et al. filter. The curves report projection time, not total solver time, and the plotted residual leaves out the two PSD-feasibility terms (Appendix E.4). It is not shown how the method compares with ADMM using an exact or partial eigendecomposition, which is the usual reference.

W2. The theory and the stated mechanism do not fully explain the reported gains.

  • Theorem 1 bounds the spectral-norm error, which is set entirely by the smallest singular value. The ratio bound (Eq. 10) depends on the floor before and after deflation. For the momentum spectra in Figure 7, the smallest normalized values are tiny even after the lift by the deflation factor. In that regime the bound is very close to one, so the theorem says almost nothing about the matrices that are actually used. The accuracy gains shown in Figure 2 are in Frobenius error, driven by the middle of the spectrum, which the theory does not address.
  • The theory covers the classical Newton-Schulz map with exact deflated components. The experiments use Muon's retuned map, which does not converge to one, and Polar Express, with estimated components. Appendix D gives a perturbation sketch for inexact components, but no result covers the maps that are actually trained with. The padding constant also differs between the two maps and is not ablated.
  • The paper's story is that a more accurate polar factor gives better training. Some of its own results point elsewhere. In Table 1, deflation already helps with zero iterations, where the output is the rescaled input and not a polar factor at all. With Polar Express, the deflated loss gets worse from three to five iterations, even though the factor becomes more accurate. With the retuned map, deflation also changes how the head and tail directions are weighted in the final update, and it likely changes the update norm. So the gain may come from a different update shape or an effective step-size change, not from accuracy. This matters, because the paper contrasts itself with methods that "change the update's scaling or geometry" (Appendix A). Two controls would settle it: Muon with an exact SVD polar factor, and a report of the update norm for both methods.
  • The novelty is modest. Deflation of dominant components is classical, as the paper notes (Saad et al., Eiermann et al., Bloch et al.). The new parts are the setting and the GPU estimators. That is a fair contribution, but the framing as a "general framework" is larger than what is shown.
Questions for authors

Clarifications:

Q1. Seeds and variance. Can you rerun the main 1B-token comparison (both maps, default settings) with at least three seeds and report mean and spread? If the gap stays clearly larger than the seed-to-seed spread for both maps, my main concern would be resolved. Please also explain why the default configuration has different loss values in Tables 1, 2, 4, 5, and 6.

Q2. Compute-matched baseline. What is the per-step time of Deflated Muon and Muon for GPT-2 Large? Can you compare against Muon with six or seven polynomial iterations, or with CANS, at matched wall-clock time?

Q3. Accuracy versus update shape. Can you train with an exact SVD polar factor as an upper bound? If exact orthogonalization does not beat Deflated Muon, the gain likely comes from something other than accuracy. Please also report the update norm (Frobenius or RMS) of both methods over training, and explain the improvement at zero iterations in Table 1.

Q4. Theory in the relevant regime. Can you evaluate the bound in Eq. 10 on the recorded momentum matrices of Section 3.1? Is there a version of Theorem 1 for Frobenius error, or for Muon's retuned map, which does not converge to one?

Q5. SDP scope. Is the Lanczos run on the residual (Algorithm 2, line 6) included in the reported projection time? What is the total solver time? Do the gains hold in FP32, on SDPs from other families, and against ADMM with an exact or partial eigendecomposition?

Q6. Scale. Do the gains hold for a smaller and a larger model, or for a different architecture? Since Table 4 suggests much of the benefit comes early, does it shrink with longer training beyond the 10B-token run?

Minor:

Q7. The padding constant is set to different values for the two maps. Please give a short sensitivity study for it, and say how it was chosen.

Q8. The caption of Table 2 says Deflated Muon wins at every momentum value, which does not hold for Polar Express at the highest value. Please correct it. The default momentum value used in the main runs should also be stated in Section 3.

Q9. The reproducibility statement says code is in the supplementary material. Please confirm that the batched randomized SVD and the deflated ADMM code will both be released, with the SDP filter coefficients.

1 comment

We thank the reviewer for the constructive questions. NS/PE denote Newton–Schulz/Polar Express; D denotes Deflated Muon. Δ = D − Muon. Unless noted, ± is sample SD over four seeds (1, 2, 3, 42); p-values use two-sided paired t-tests.

Q1. Seeds and default discrepancies

GPT-2 Large, 1B tokens, lr = 0.01, β = 0.95, L = 5:

MapMuonDPaired Δ (p)
NS3.3817 ± 0.00553.3597 ± 0.0013−0.0220 ± 0.0065 (0.007)
PE3.3673 ± 0.00193.3618 ± 0.0029−0.0055 ± 0.0021 (0.013)

The 95% CIs are [−0.032, −0.012] and [−0.009, −0.002]. Each ablation batch reran the same default and seed; the 0.002–0.005 spread reflects GPU run-to-run variation. Within-batch gains were −0.020 to −0.023 (NS) and −0.007 to −0.010 (PE).

Q2. Compute matching

On 4 × H200, Muon L = 5/6/7 takes 1763/1773/1799 ms/step; D L = 5 takes 1782 ms (+1.1%). At lr = 0.01, allocating Muon L = 5's 1905-step time budget and adjusting schedules gives four-seed mean losses:

MethodMatched-time loss
Muon NS, L = 6/7/9/103.3650/3.3754/3.3809/3.3804
Muon PE, L = 53.3673
CANS, L = 53.3657
D NS, L = 53.3632

D receives 1884 steps. Its margins over NS L = 6 and CANS are small (0.0018/0.0025). CANS uses the public five-polynomial coefficients with our PE implementation's two bfloat16 safety factors.

Q3. Accuracy versus update shape

Exact-SVD polar loss is 3.3699 (four-seed mean), above both deflated means in Q1. Accuracy alone therefore does not explain the gain.

The D/Muon update-RMS ratio on identical momentum inputs falls from 1.81 (steps 1–100) to 1.04 (1001–1905). Matching D's norm to Muon's per matrix/step retains the gain: 3.3572 versus D 3.3594 and Muon 3.3813 (seed 42). Normalization only, M/(1.1σ^max⁡)M/(1.1\hat\sigma_{\max}), gives 3.3818; random subspaces give 3.3794.

At L = 0, the initialization already sets the head to γ⁻¹ and amplifies the residual by αk=∥M∥F/∥Rk∥F\alpha_k=\|M\|_F/\|R_k\|_F. Four-seed losses are 4.096 versus 4.189, showing benefit before polynomial iteration.

Q4. Relevant theory

Eq. (10) evaluates to 0.9991–1.0000 on 24 recorded matrices; measured spectral-error ratios are 0.998–1.000. Tiny floors (10⁻⁶–10⁻⁴) leave the weakest directions unresolved.

For exact heads under Theorem 1, q=rank⁡(M)q=\operatorname{rank}(M), and G=pd∘LG=p_d^{\circ L}, squared Frobenius error is EF2=k[1−G(γ−1)]2+∑i=k+1q[1−G(σi/∥Rk∥F)]2.E_F^2=k[1-G(\gamma^{-1})]^2+\sum_{i=k+1}^{q}[1-G(\sigma_i/\|R_k\|_F)]^2. Classical monotonicity reduces every tail term. At γ = 1 the head term vanishes; for γ > 1 it must be included (below 2×10⁻¹⁶ per head direction for the quintic, 1 ≤ γ ≤ 1.1, L ≥ 2).

For Muon's retuned map, p(1) = 0.701. Its finite-step envelope B5(ℓ)=max⁡x∈[ℓ,1]∣1−p∘5(x)∣≤max⁡{0.3182,1−453ℓ}B_5(\ell)=\max_{x\in[\ell,1]}|1-p^{\circ5}(x)|\le\max\{0.3182,1-453\ell\} does not increase when the floor becomes min⁡{αkℓ,γ−1}\min\{\alpha_k\ell,\gamma^{-1}\}. This is not a universal matrix-error ratio guarantee. Recorded Frobenius errors fall 0.583→0.423 (retuned) and 0.879→0.799 (classical).

Q5. SDP scope

We tested eight instances with the same ADMM and five projection variants. Timing includes tracking, gating, residual Lanczos, filtering, restoration, and symmetrization.

On G55mc, 10,000 iterations take 186/217/1879 s for filter FP16/deflated FP16/exact FP64, with η = 2.8e-3/8.1e-4/5.5e-4. Time to 10⁻³ is unreached/108/939 s. Partial eigendecomposition saves no time here, with about half the eigenvalues positive.

FP32 does not improve every checkpoint but improves PSD feasibility: on G55mc, λ_min(X) changes from −2.2e-3 to −2.7e-6. After a common deflated-FP16 phase to η < 10⁻³, deflated FP32 reaches 5.0e-6–5.3e-6 on the two n = 512 instances, versus 7.8e-5–1.0e-4 without deflation. On G55mc it reaches 2.9e-5 versus 1.1e-4 (exact: 3.1e-5).

The four added DIMACS torus cases broaden the instance/size range; all eight remain max-cut/min-bisection. Deflated FP16 reaches 10⁻³ on six; the FP16 filter on none.

Q6. Scale and horizon

With separately tuned learning rates, GPT-2 Small loss reductions at 1B tokens are 0.0073/0.0010 (NS/PE), and at 10B are 0.0057/0.0016. The latter is about 4× its compute-optimal budget. GPT-2 Large's 10B gains are 0.0226/0.0146 (lr = 0.01).

Q7. Padding

γ⁻¹ provides an overshoot margin. Defaults are γ = 1.01 (NS), 1.1 (PE). Recorded polar errors vary little over γ = 1–1.5. NS retains its gain throughout; PE diverges within three steps at γ = 1/1.01 but gives 3.3600/3.3594 at γ = 1.1/1.5. The larger PE margin addresses training stability.

Q8. Caption and default momentum

PE at β = 0.98 gives 3.3783 ± 0.0024/3.3772 ± 0.0023 (Muon/D; p = 0.26). We agree the caption should describe comparable performance. Default β = 0.95.

Q9. Release

Deflated Muon and ADMM code are in the supplement and will be released with FP16/FP32 filter coefficients, run scripts, and added experiments. Instances are public via Mittelmann/DIMACS.

We have uploaded the revised manuscript.

Reviewer_NzhtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes spectral deflation. The dominant spectral components of the input are peeled off (leading k singular triplets for the polar factor, k eigenpairs of largest magnitude for the PSD projection), the normalized residual alone is passed through an unchanged fixed-depth polynomial filter, and the head is restored through an exact identity. Because the target is unchanged and only the normalized spectrum seen by the filter moves, the spectral floor rises by a factor αk. The authors pair this with two estimators - a batched randomized SVD for Muon, a warm-started subspace tracker for factorization-free ADMM - and prove that exact deflation lifts the floor from l to min(αk l, γ-1), which contracts the finite-step Newton-Schulz error. Experiments cover GPT-2 Large pretraining at 1B and 10B tokens and four large SDPs from Mittelmann's collection.

Critical strengths

1.The mechanism is clean and leaves the target alone. Equation (2) is an exact decomposition, so deflation changes only the normalization the filter sees. That separates it from the redesign-the-polynomial line (CANS, Polar Express) and the change-the-geometry line (NorMuon, MuonEq), and it explains why the same wrapper drops into two apparently unrelated settings. The positioning is unusually clear. 2.The two instantiations are not copy-paste. Muon has many same-shape matrices and redraws a sketch every step; ADMM has one large slowly varying matrix and can carry its subspace forward. The paper says this out loud and picks different estimators accordingly, which is more thought than most papers give this step. 3.The theory is honest about its own scope. The proof goes through the Bjorck-Bowie residual identity with explicit constants and the padding parameter folded into the theorem. Appendix E.3 adds a bound for the fixed filter actually used (Proposition 5) and an idealized minimax comparison (Theorem 6), which draws a line between the analysis and the implementation instead of blurring it. 4.The ablations are thorough. L, β, w, τ and the deflation cutoff are all swept; estimator accuracy is quantified separately (Table 3, Figure 9, Ritz-value error below 3e-7); even the practical question of enabling deflation for only part of training is tested. The w sweep in Table 5, where the number of removed pairs varies from 1.5 to 8.8 with essentially no change in validation loss, is quietly one of the more informative results in the paper.

Critical weaknesses

1.The statistical evidence is thin. Every language-model run is a single seed on a single data subset, and the gains are on the order of 0.02 nats (3.3794 to 3.3568, roughly 0.7%). For Polar Express at β = 0.98, Table 2 reports 3.3786 against 3.3785 - a tie - while the text claims deflation "outperforms Muon at every β". That sentence should be removed, and I would want 2-3 seeds with some measure of spread before "consistently improves" carries the weight the abstract places on it. 2.The obvious alternative - just run more iterations - is never matched. Deflation adds a batched rSVD (r' = 2r, about 0.05m columns), a QR and a few GEMMs per step, roughly the cost of several polynomial iterations. Table 1 shows deflated L = 3 already beats the L = 5 baseline; so what happens if the baseline gets L = 7 or 8 at the same wall-clock? Figure 1a plots against time, which is the right axis, but only one baseline point is drawn. Adding a cost-matched, stronger baseline would settle the "same filter budget" claim far better than any further sweep. 3.There is a tension in the paper that I could not resolve. Figure 2 shows relative polar-factor error at L = 5 still around 0.4 (the plotted range is 0.4 to 1.0), while end-to-end training improves by 0.02 nats. Either the metric is dominated by a long tail of tiny singular values and has little to do with training, or polar accuracy is simply not Muon's bottleneck. Both readings undercut the causal story, and neither is discussed. 4.No control rules out implicit regularization. Subtract the same rank-k subspace but replace it with random orthonormal directions, or merely rescale the initialization, and see whether the gain survives. Without such a control, "better spectral approximation leads to better training" remains an assumption rather than a finding. 5.The SDP evidence is too thin for the claim. Four instances, all max-cut or min-bisection from a single collection at n = 5,000 and 7,000, a fixed budget of 104 iterations, and residuals that stop at 1e-4 to 1e-2. Practitioners care about 1e-6 and below, and the FP16 projection's accuracy floor sits near 1e-4, so I cannot tell whether deflation still helps in the regime that matters. There is also no comparison against exact eigendecomposition projection, against SCS or cuADMM, or against the approximate-ADMM line. Figure 5 shows that the same filter does better with deflation, which is not the same claim as the solver being competitive.

Questions for authors

1.What are the means and variances across seeds for the 1B-token runs? Is the Polar Express β = 0.98 difference anything other than noise? 2.At a matched wall-clock budget, what does the baseline reach with L = 9 or 10? 3.What happens with a random-subspace subtraction of the same rank, or with only a change of normalization scale? 4.How do the 0.4-level errors in Figure 2 map onto the 0.02-nat end-to-end gain? Does this indicate that Muon is insensitive to polar-factor accuracy? 5.How does the deflated projection behave for residuals at or below 1e-6, and how does it compare with exact projection or other first-order solvers?

1 comment

We thank the reviewer for the constructive questions. NS/PE denote Newton–Schulz/Polar Express; D denotes Deflated Muon. Δ = D − Muon. Unless noted, ± is sample SD over four seeds (1, 2, 3, 42), controlling initialization and data order; p-values use two-sided paired t-tests.

Q1. Seeds and β = 0.98

GPT-2 Large, 1B tokens, lr = 0.01, β = 0.95, L = 5:

MapMuonDPaired Δ95% CIp
NS3.3817 ± 0.00553.3597 ± 0.0013−0.0220 ± 0.0065[−0.032, −0.012]0.007
PE3.3673 ± 0.00193.3618 ± 0.0029−0.0055 ± 0.0021[−0.009, −0.002]0.013

Both intervals exclude zero. For PE at β = 0.98, Δ = −0.0010 ± 0.0015 (p = 0.26), within seed variation. We agree that this case is comparable performance, not a clear improvement.

Q2. Deeper, compute-matched baselines

NS step times are 1763/1838/1847 ms for Muon at L = 5/9/10, versus 1782 ms for D at L = 5. At lr = 0.01:

MethodEqual-step lossMatched-time loss
Muon, L = 93.3693 ± 0.00393.3809 ± 0.0036
Muon, L = 103.3655 ± 0.00123.3804 ± 0.0017
D, L = 53.3597 ± 0.00133.3632 ± 0.0029

At equal steps, deeper baselines receive 3–4% more time. The matched budget equals Muon L = 5 over 1905 steps; Muon L = 9/10 and D receive 1826/1818/1884 steps, with schedules adjusted accordingly. D's paired advantages are 0.0177 ± 0.0014 (p = 0.0001) and 0.0172 ± 0.0030 (p = 0.001).

At grid-selected lr = 0.006, single matched-budget runs retain this ordering: 3.3718/3.3689 for Muon L = 9/10 versus 3.3548 for D L = 5.

Q3. Random-subspace and normalization controls

For NS, lr = 0.01, seed 42:

UpdateValidation loss
Muon3.3813
D3.3594
Random subspaces3.3794
Normalization only3.3818

The random control substitutes uniform random subspaces for the dominant ones, with the same per-matrix, per-step rank selected by the unchanged estimator. Normalization only uses M/(1.1σ^max⁡)M/(1.1\hat\sigma_{\max}) without removal/restoration. Neither reproduces the gain, supporting the role of the dominant components.

Q4. Polar error versus training gain

Figure 2 weights all 1280 singular directions equally. About 90% of pooled squared error lies in directions with σi<10−3σ1\sigma_i<10^{-3}\sigma_1; those above 10−2σ110^{-2}\sigma_1 contribute only 3–5%. Thus, error near 0.4 mainly reflects unresolved weak directions. Deflation approximately halves pooled squared error and improves alignment with the exact polar factor from 0.77 to 0.90.

Exact-SVD polar updates at every step reduce validation loss from 3.3817 ± 0.0055 to 3.3699 ± 0.0021. D reaches 3.3597 ± 0.0013, a further paired difference of −0.0102 ± 0.0030. Accuracy matters, but alone does not explain the full training benefit.

Q5. SDP accuracy and projection references

With the same ADMM, σ = 1, and 10⁵ iterations on torusg3-8, toruspm3-8-50, torusg3-15, and G55mc, deflation lowers the FP16 residual floor from 4.3e-3–6.1e-3 to 6.8e-4–8.3e-4 (5–8×). Neither tested FP16 filter reaches 10⁻⁶.

For G55mc after 10,000 iterations:

Projectionms/projectionFinal ηTime to η < 10⁻³
Filter, FP1617.82.8e-3Not reached
Deflated, FP1620.98.1e-4108 s
Exact, FP641875.5e-4939 s

Times to 10⁻³ are also 281 versus 2630 s on G60mc and 110 versus 374 s on torusg3-15, for deflated versus exact projection.

For higher accuracy, we switch from deflated FP16 to exact FP64 at η < 10⁻³. Time to 10⁻⁶ decreases from 240 to 235 s on torusg3-8 and from 159 to 154 s on toruspm3-8-50. On G55mc, time to 10⁻⁴ decreases from 5873 to 5133 s, and to 5×10⁻⁵ from 7619 to 6877 s.

These results support deflated FP16 as an efficient initial phase with exact refinement. We have not compared other first-order solvers; our claims concern projection within the tested ADMM.

We have uploaded the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.