- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a spectrum-adaptive procedure for Newton–Schulz matrix-function computations. It extracts spectral moments from Gram matrices already formed during the iteration, fits a discrete spectrum using a maximum-entropy method, and uses the fitted spectrum to select a polynomial routine for each matrix. The method is evaluated for polar-factor computation in Muon and inverse-square-root computation in OKLS. Offline experiments show lower approximation error than a fixed routine in many settings. GPT pretraining experiments at 160M, 300M, and 1B non-embedding parameter scales report lower validation loss for the adaptive method than for the corresponding fixed-routine baselin
Critical strengths
The method uses spectral information from intermediate matrices that Newton–Schulz already computes. It avoids extra matrix multiplications and offers a practical way to adapt polynomial routines to differences between matrices. The idea has some level of novelty.
Critical weaknesses
-
The practical evidence is limited in scale and statistical strength. The largest pretraining model has 1B non-embedding parameters, and all runs use the same dataset and training setup. The reported validation-loss improvements are small The H200 timings for frontier-model matrix shapes are per-matrix measurements rather than end-to-end training results at those scales. These experiments support the method’s potential, but do not establish that its gains and low overhead will persist in larger distributed training.
-
The experimental evaluation is incomplete and the tuning protocol is insufficiently documented. Several important hyperparameters, including the learning rate and momentum coefficients, have unusually precise and nonstandard values (For example, the peak learning rates are 0.01202 for Muon and 0.09434 for OKLS, the momentum coefficients are beta=0.9586 for Muon). However, the paper does not sufficiently explain how these values were selected, including the search space, tuning budget, validation criterion, random seeds, or whether all methods received comparable tuning budgets. Some choices appear consistent with automated hyperparameter tuning, but this cannot be verified from the paper. Consequently, it is difficult to determine whether the reported improvements come from the proposed spectral estimation and routine-selection method or from optimizer-specific hyperparameter tuning.
Questions for authors
1.Can the authors report the full polar-factor and inverse-square-root errors, including the singular values below the scoring cutoff, and discuss how these errors affect the resulting optimizer updates?
2.Can the authors provide pretraining results on larger models (3B or larger), and end-to-end timing or throughput measurements for the training runs? This would help establish whether the small loss improvements are robust and clarify the practical overhead beyond the per-matrix measurements.
3.How were the optimizer settings in Table 19 selected? Please clarify which settings were inherited from previous work,and which were changed or retuned for this work, and the tuning procedure or sensitivity of the results to these choices. Since currently these information is not provided, it looks like that these hypperparameters are result of auto-searching.