acceptodds
Under review as a conference paper at ICLR 2027

Cost-Free Spectral Estimation for Adaptive Newton–Schulz in Matrix Optimizers

Abstract

Matrix optimizers such as Muon transform each momentum matrix through an approximate orthogonalization, typically implemented by a small number of Newton–Schulz matrix multiplications. The quality and cost of this approximation depend strongly on the singular-value spectrum of its input, yet existing implementations use the same fixed polynomial routine for every layer and throughout training. We show that this uniform treatment is unnecessary: the computations in the Newton–Schulz method already reveal enough information to make the method adaptive. The Gram matrices formed inside Newton–Schulz iterations yield spectral moments through inexpensive scalar reductions, requiring no additional matrix multiplications. From these moments, we recover an estimate of the empirical singular-value distribution and use it to select a polynomial routine specialized to the current matrix. This turns Newton–Schulz orthogonalization into a spectrum-adaptive procedure that responds to differences across both layers and training time. On saved momentum matrices, spectral estimation substantially reduces orthogonalization error at a fixed iteration budget or reaches the same accuracy with fewer iterations, and in GPT pretraining up to 1B parameters it lowers the validation loss of two matrix optimizers. Our results suggest that matrix-function operations inside optimizers need not be designed for a conservative worst-case spectrum: they can cheaply measure the spectrum they are already processing and specialize computations accordingly.

Primary area:
optimization
Keywords:
deep learning optimization, LLM pretraining, matrix optimizers, Newton–Schulz iterations, spectrum estimation, Muon

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tYBROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a spectrum-adaptive procedure for Newton–Schulz matrix-function computations. It extracts spectral moments from Gram matrices already formed during the iteration, fits a discrete spectrum using a maximum-entropy method, and uses the fitted spectrum to select a polynomial routine for each matrix. The method is evaluated for polar-factor computation in Muon and inverse-square-root computation in OKLS. Offline experiments show lower approximation error than a fixed routine in many settings. GPT pretraining experiments at 160M, 300M, and 1B non-embedding parameter scales report lower validation loss for the adaptive method than for the corresponding fixed-routine baselin

Critical strengths

The method uses spectral information from intermediate matrices that Newton–Schulz already computes. It avoids extra matrix multiplications and offers a practical way to adapt polynomial routines to differences between matrices. The idea has some level of novelty.

Critical weaknesses
  1. The practical evidence is limited in scale and statistical strength. The largest pretraining model has 1B non-embedding parameters, and all runs use the same dataset and training setup. The reported validation-loss improvements are small The H200 timings for frontier-model matrix shapes are per-matrix measurements rather than end-to-end training results at those scales. These experiments support the method’s potential, but do not establish that its gains and low overhead will persist in larger distributed training.

  2. The experimental evaluation is incomplete and the tuning protocol is insufficiently documented. Several important hyperparameters, including the learning rate and momentum coefficients, have unusually precise and nonstandard values (For example, the peak learning rates are 0.01202 for Muon and 0.09434 for OKLS, the momentum coefficients are beta=0.9586 for Muon). However, the paper does not sufficiently explain how these values were selected, including the search space, tuning budget, validation criterion, random seeds, or whether all methods received comparable tuning budgets. Some choices appear consistent with automated hyperparameter tuning, but this cannot be verified from the paper. Consequently, it is difficult to determine whether the reported improvements come from the proposed spectral estimation and routine-selection method or from optimizer-specific hyperparameter tuning.

Questions for authors

1.Can the authors report the full polar-factor and inverse-square-root errors, including the singular values below the scoring cutoff, and discuss how these errors affect the resulting optimizer updates?

2.Can the authors provide pretraining results on larger models (3B or larger), and end-to-end timing or throughput measurements for the training runs? This would help establish whether the small loss improvements are robust and clarify the practical overhead beyond the per-matrix measurements.

3.How were the optimizer settings in Table 19 selected? Please clarify which settings were inherited from previous work,and which were changed or retuned for this work, and the tuning procedure or sensitivity of the results to these choices. Since currently these information is not provided, it looks like that these hypperparameters are result of auto-searching.

Reviewer_SD5kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The Newton–Schulz iteration used in Muon approximates the polar factor of momentum matrices. Its approximation difficulty depends strongly on the matrix spectrum, yet existing implementations commonly apply the same fixed polynomial routine across matrices. This paper proposes a spectrum-adaptive procedure that responds to differences across layers and training time, substantially reducing offline orthogonalization error at a fixed iteration budget. End-to-end GPT pretraining from 160M to 1B parameters shows small but consistent validation-loss improvements over fixed routines for both Muon and OKLS.

Critical strengths
  1. The paper presents a simple and useful insight: commonly used Newton–Schulz implementations apply fixed, spectrum-agnostic polynomial routines despite their strongly spectrum-dependent approximation quality. The intermediate computation can instead be used to select adaptive routines.
  2. The empirical studies are extensive, including offline validation, end-to-end pretraining, and detailed cost accounting. Comprehensive diagnostics and ablation studies are provided.
  3. The visualizations and presentation are clear and effective.
Critical weaknesses
  1. `Cost-free' in the title may be overstated. Although additional matrix multiplications are avoided, the method still requires scalar reductions, synchronization, CPU optimization, and periodic deeper measurement passes. End-to-end throughput overhead is estimated but not directly measured.
  2. The necessity of frequent dynamic adaptation is not clearly established. Reusing the first checkpoint’s per-matrix selection performs nearly as well as recomputing the current selection. Moreover, the end-to-end validation-loss improvements are relatively small and reported without repeated-seed uncertainty.
  3. Stronger baselines are needed to isolate the benefit of the proposed method. Improved normalization and static per-type choices recover part of the offline gain but are not evaluated end-to-end. The exact-spectrum oracle should also use the same scoring objective and measurement period as the proposed method.
Questions for authors
  1. Can the authors compare against stronger production-oriented fixed routines, such as publicly specified multi-stage coefficient schedules, using the same matrix-multiplication or wall-clock budget?
  2. Can the target rule provide a certified error bound? Controlled worst-case spectra containing isolated small singular values would help evaluate when the moment-based estimator fails.
  3. Can the proposed method and exact-spectrum oracle be compared using the same scoring objective and measurement period? Table 4 includes a fixed K=8 baseline at 160M; numerical results and measured wall-clock time may be reported at larger scales and with OKLS.
  4. The main RMS score excludes singular values below the cutoff and can improve average error while worsening worst-case error. Full-spectrum error and the implications of spectral-tail errors for optimizer stability should be reported.
Reviewer_aiFeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work addresses the problem that the dynamics of Newton–Schulz iterations in Muon optimizer vary depending on the input spectrum of the momentum. NS reads spectral moments from the already computed Gram matrix during iterations and its square without additional matrix multiplication. Then, it finally estimates the spectral distribution using maximum-entropy fitting. By predicting the approximation error from this distribution, the proposed method chooses the most accurate routine at a fixed depth or the shortest routine that is expected to achieve the target error. In terms of theory, it proposes conditions and an error bound that allow for the prediction of candidate scores without reconstructing the spectrum. This procedure significantly reduces polar approximation error in the momentum matrix and shows a small but consistent improvement in average validation loss compared to the fixed routine during Muon and OKLS from 160M to 1B pre-training.

Critical strengths
  1. The authors extract spectral moments information from intermediate gradient Gram matrices which is already computed during NS iterations, without additional matrix multiplications or expensive SVD. This provides an efficient way to adapt polynomial routines to individual matrices with minimal computational overhead.

  2. Rather than reconstructing the entire spectral distribution, the proposed method mainly considers predicting the approximation error relevant to candidate choice. Theorem 2 provides theoretical intuition on this approach by relating prediction accuracy to several quantities.

  3. The proposed framework can be applicable to both polar approx. in Muon and also inverse square root computation in OKLS.

Critical weaknesses

I have some concerns about the paper.

  1. A fundamental concern I have on this work is whether more accurate spectral flattening necessarily leads to better optimization performance. While the proposed method effectively reduces the approximation error of NS iterations, it remains unclear why making the singular values closer to one should consistently improve LLM pre-training. In fact, I think that rough orthogonalization introduces structured spectral reweighting, which may potentially benefit optimization by suppressing less informative directions. Similarly, small perturbations to the ideal polar update may improve training dynamics rather than degrade them. In this context, this paper does not sufficiently investigate the possibility or establish a clear connection between orthogonalization accuracy and downstream optimization or generalization. I believe that a more comprehensive analysis of this relationship, such as comparing exact orthogonalization with partial flattening or perturbed updates, would greatly strengthen the motivation of the paper.

  2. While I appreciate the ablation studies in appendix, the benefits of per-matrix and time-dependent adaptation remain unclear. For example, reusing polynomial routine or choosing one routine per matrix achieves performance comparable to the proposed method. In this context, I have some doubt about whether fine-grained adaptation is actually necessary.

Questions for authors

Please refer to the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.