- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes spectral deflation to improve the accuracy of fixed-depth polynomial filtering methods, particularly Muon and first-order SDP solvers. The motivation is that, when a few dominant singular values account for most of the Frobenius norm, normalization pushes the remaining singular values close to zero, making it difficult for a fixed number of Newton-Schulz iterations to bring them close to one. To address this issue, the authors propose removing the dominant singular components, normalizing the residual matrix, and then reintroducing the removed components before applying the polynomial filter. They also provide theoretical results showing that spectral deflation reduces the finite-step approximation error for the classical Newton-Schulz family. The authors evaluate the proposed method on GPT-2 Large pretraining using FineWeb, comparing Deflated Muon against Muon with both Newton-Schulz and Polar Express mappings. They report improvements in validation loss and wall-clock performance, along with several ablation studies. They further demonstrate the effectiveness of spectral deflation in first-order SDP solvers, showing lower KKT residuals on large-scale SDP problems under similar projection-time budgets.
Critical strengths
The motivation is interesting and well justified. For example, in Muon, normalization by the Frobenius norm can push smaller singular values close to zero when a few dominant singular values account for most of the matrix Frobenius norm. Consequently, these small singular values may remain significantly below one even after a fixed number of Newton-Schulz iterations. To address this issue, the authors propose removing the dominant singular components, normalizing the residual matrix, and then reintroducing the removed singular directions. The proposed idea is simple and intuitive, and it can be incorporated into existing polynomial filtering methods without modifying the polynomial itself.
Critical weaknesses
- The performance gains of Deflated Muon appear relatively modest. For example, in Figure 3(a), while Deflated Muon achieves a lower validation loss than standard Muon with the Newton-Schulz mapping, Muon with Polar Express achieves the best validation loss among all four methods when comparing their respective optimal learning rates. This raises concerns about the practical benefits of deflation, especially considering the additional computational overhead of Algorithm 3. Furthermore, several experiments appear to use a fixed learning rate of 0.01, including Figure 3(b) and possibly Tables 1 and 2. However, Figure 3(a) suggests that 0.01 is not the optimal learning rate for any of the methods. In particular, the performance gap between Muon and Deflated Muon is relatively large at this learning rate, whereas the gap becomes smaller or even reverses near their respective optimal learning rates. Therefore, I believe that a fair comparison should evaluate each method with its individually optimized hyperparameters, particularly learning rates, while accounting for the additional computational overhead of deflation. Without such comparisons, it remains unclear whether Deflated Muon provides meaningful practical benefits over well-tuned baselines.
- It is unclear whether the proposed algorithm needs to be as complicated as it currently is. Figure 7 in Appendix C shows that the singular value spectra are often dominated by a small number of large singular values. Furthermore, Table 5 in Appendix F shows that using a very small deflation window, with an average deflation rank of approximately 1.5, already achieves performance comparable to that of larger deflation windows. This raises the question of whether a much simpler approach, such as estimating only the top singular triplet using power iteration and removing that component before normalization, could achieve similar performance without requiring the batched randomized SVD in Algorithm 3. Such a baseline would help clarify whether the additional computational complexity is necessary. Additionally, the necessity of the deflation gate is not sufficiently justified. Since the gate is evaluated after estimating the singular values using Algorithm 3, most of the computational cost has already been incurred, regardless of whether deflation is applied. While the gate may help avoid unnecessary or potentially harmful deflation, the paper does not provide an ablation showing what happens when the gate is removed. It would be helpful to understand why this gate is necessary and how much it contributes to the overall performance.
Questions for authors
- Could the authors compare the methods using their individually optimized hyperparameters, particularly learning rates and the number of polynomial iterations, under equal training-time budgets? Figure 3(a) suggests that the learning rate of 0.01 used in Figure 3(b) is suboptimal for several methods. It would also be helpful to clarify which learning rates were used in Tables 1 and 2 and Figure 4, and whether they were tuned separately for each method.
- Table 5 suggests that removing only a very small number of dominant singular components is sufficient to achieve most of the performance improvement. Could the authors clarify how the proposed method compares against a simpler rank-1 deflation approach, where only the top singular triplet is estimated using power iteration? Would such an approach achieve comparable performance with lower computational overhead?
- What happens if the deflation gate in Algorithm 1 is removed and deflation is always applied? Since the gate is evaluated after running Algorithm 3, the computational cost of estimating the dominant singular components has already been incurred. Could the authors explain why the gate is necessary and provide an ablation comparing performance with and without it?
- Theorem 1 appears to analyze the classical Newton-Schulz polynomial rather than the retuned coefficients used in practical Muon. Could the authors clarify whether a similar error reduction guarantee holds for the actual Muon coefficients?
- Figure 4 extends the experiments to approximately the Chinchilla-optimal training budget. However, modern LLM pretraining often uses several times the Chinchilla-optimal number of tokens. Is there any evidence that the advantage of Deflated Muon persists under substantially longer training horizons, especially when each baseline is evaluated with its optimal hyperparameters?
We thank the reviewer for the constructive suggestions. NS/PE denote Newton–Schulz/Polar Express; D denotes Deflated Muon; Δ = D − Muon.
Q1. Tuned hyperparameters and equal time
Tables 1–2 and Figures 3(b), 4 use lr = 0.01 for all methods; Figure 3(a) uses the common grid {0.002, …, 0.01}. We agree that lr = 0.01 alone does not establish an advantage over tuned baselines.
We now select learning rate and depth separately within the tested grids. Each method receives Muon L = 5's 1905-step time budget, including deflation overhead; step counts follow measured timings, with schedules adjusted accordingly:
D PE has the lowest loss, 0.0069 below the best baseline. NS losses are comparable, with D using five rather than six iterations. On GPT-2 Large, 4 × H200, step time is 1763 ms for Muon L = 5 and 1782 ms for D L = 5 (+1.1%), faster than Muon L = 7.
Q2. Rank-1 simplification and unknown head width
Table 5's small ranks are a posteriori statistics, available after spectral estimation. The head width is unknown beforehand and varies by matrix and training stage. Batched randomized SVD estimates the leading components and selects the rank online without prescribing it in advance.
We tested three batched power-iteration steps for the top triplet, with rank-1 deflation always on. For stability, the residual is (I−uu⊤)M(I−vv⊤) before normalization:
Rank-1 recovers the gain, but its measured overhead is 24 ms/step versus 20–25 ms for randomized SVD. Logging all 216 matrices shows mean active ranks of 10.0/19.1/3.0/7.9/8.4/9.4 for Q/K/V/O/MLP input/output, with an overall range of 1–31. For K, the early-to-late rank increases from 14 to 28. We retain randomized SVD for adaptive rank selection at comparable measured cost; rank-1 is a useful simpler comparison.
Q3. Gate ablation
The gate does not avoid estimation cost; it selects when and how much to deflate. Always-on deflation uses:
k=\min\\{\max\\{\\#\\{s_i > 0.1s_1\\},1\\},r-1\\}.
The loss changes (−0.0016/+0.0008) are comparable to observed run-to-run variation. The gate primarily limits deflated rank in these runs; this ablation does not show it is necessary for the gain.
Q4. Practical coefficients
The floor statement is polynomial-independent; Eq. (10)'s strict ratio bound is specific to classical NS. With exact components under Theorem 1,
\alpha_k=\lVert M\rVert_F/\lVert R_k\rVert_F.$$ Muon uses $p(x)=3.4445x-4.7750x^3+2.0315x^5$, with p(1) = 0.701, so classical convergence to one does not apply. Define $$B_L(\ell)=\max_{x\in[\ell,1]}|1-p^{\circ L}(x)|.$$ Interval inclusion gives $B_L(\ell_{\mathrm{def}})\le B_L(\ell)$. For L = 5, $$B_5(\ell)\le\max\\{0.3182,1-453.0\ell\\}.$$ This envelope improves until saturation near ℓ = 1.5×10⁻³; it does not guarantee improvement in every direction. Directions whose baseline and deflated iterates stay in [0, 0.55] through the final iterate improve under the larger initialization. Across 24 recorded matrices, relative Frobenius error decreases from 0.583 to 0.423 with the retuned map, versus 0.879 to 0.799 for the classical quintic. Figures 2 and 10–11 already use the retuned map. ## Q5. Longer training GPT-2 Large at 10B tokens (Figure 4, lr = 0.01) ends at 2.9500/2.9274 for Muon/D NS (Δ = −0.0226) and 2.9382/2.9236 for PE (Δ = −0.0146). The gap does not shrink between the reported 1B and 10B settings at this learning rate. We also trained GPT-2 Small (124M) on 10B tokens, about 4× its compute-optimal budget, selecting each method's lr from {0.0025, 0.005, 0.01}: |Mapping|Muon|D|Δ| |---|---|---|---| |NS|3.2476|3.2419|−0.0057| |PE|3.2402|3.2387|−0.0016| Both retain lower observed losses at the longer relative horizon, with a smaller PE difference. We have uploaded the revised manuscript.