acceptodds
Under review as a conference paper at ICLR 2027

FIRMO: Robust Matrix-Aware Optimization for Neural Networks

Abstract

The optimization of large language models (LLMs) remains a critical challenge, particularly as model scaling exacerbates sensitivity to algorithmic imprecision and training instability. Recent advances in optimizers improve convergence efficiency through momentum orthogonalization but suffer from two key robustness limitations: dimensional fragility in orthogonalization precision and vulnerability to outlier-induced noise. To address these robustness challenges, we introduce FIRMO, an optimizer that enhances training stability through dual robustness mechanisms. First, we develop a dimension-robust orthogonalization scheme that calibrates the full Newton–Schulz composition by matrix shape, achieving higher fidelity than Muon across the evaluated shapes under the same five-step budget. Second, we introduce an optimization-robust framework grounded in proximal optimization that adaptively suppresses extreme input magnitudes while preserving entries within the threshold and retaining matrix-wise orthogonalization. Experiments on a 1B-parameter LLM demonstrate faster convergence in training steps and lower final loss than Muon in 10B-token runs. After 100B-token pre-training, FIRMO achieves 60.12% average zero-shot accuracy across nine tasks, versus 59.59% for Muon and 59.05% for AdamW. Our work establishes a unified framework for robust and precise matrix-aware optimization by jointly exploiting matrix-shape structure and adaptive input-scale control.

Primary area:
optimization
Keywords:
optimization, orthogonalized optimizers, large language model pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_pP2DOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

FIRMO augments Muon with offline, shape-specific Newton–Schulz coefficients and percentile-based clipping before orthogonalization. Evaluation includes 1B-model pretraining, nine-task evaluation after 100B tokens, component ablations, and a clipping-focused CIFAR-10 experiment.

Critical strengths

The 100B-token experiment provides meaningful end-to-end validation. Testing orthogonalization on gradients separate from the calibration sample also provides useful evidence beyond performance on the fitted spectra.

Critical weaknesses

Limited novelty without decisive comparisons. Improved NS polynomials and pre-orthogonalization magnitude control have direct precedents in CANS/Polar Express and AdaMuon, acknowledged in §2.2 but absent from the main comparisons. A global coefficient triplet refitted on the same calibration data is also missing. Consequently, the experiments do not establish whether shape conditioning and this particular combination offer advantages over existing alternatives. This is central to assessing the contribution, not merely a request for additional baselines.

Insufficient evidence for robustness and reliability. Section 4.5 selects the calibration mixture using larger-model stability results that are not presented, while the displayed experiment favors a different mixture. A central robustness argument therefore cannot be verified. The 0.53-percentage-point average accuracy improvement over Muon is also reported without uncertainty estimates, leaving its reliability unclear.

Questions for authors

Clarification—comparative evidence: Are existing results available against CANS/Polar Express with identical clipping, AdaMuon, or globally refitted coefficients that establish the value of FIRMO’s specific design?

Clarification—robustness evidence: What are the configurations and results of the larger-model stability experiments invoked in §4.5?

Reviewer_vErzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FIRMO, an extension of Muon combining shape-conditioned Newton–Schulz (NS) coefficients with percentile-based input clipping. The coefficients are calibrated offline with a mix of empirical (using a 0.5B calibration tokens) and random matrix spectra pulled by shape. The coefficients remain fixed during training and are not updated. The authors demonstrate improved orthogonalization accuracy. in addition they report large cifar10 accuracy gap from muon (with a weak accuracy that is substantially below 90%) and report modest improvements in language-model pretraining loss and downstream accuracy.

Critical strengths
  1. The approach is simple and preserves Muon's five-step NS iteration budget while quantile clipping is relativly cheap and calibration costs are not discussed but remain external to the training loop.

  2. Table 1 and Figure 2 demonstrate substantial improvements in finite-step orthogonalization accuracy thanks to the coefficient calibraion process.

  3. The experiments include 100B-token LLM pretraining, component ablations, and a separate CIFAR-10 evaluation that are usefull.

Critical weaknesses
  1. Limited novelty and missing comparison with prior art. Optimizing NS polynomials has already been explored in CANS and Polar Express. FIRMO's main methodological distinction is offline calibration using shape-specific spectral distributions, combined with percentile clipping. The paper does not compare against these closely related approaches or establish the importance of shape-specific calibration relative to more general polynomial optimization.

  2. Questionable calibration robustness and hyperparameter selection (Section 4.5). a. Figure 4(b) shows that the 1:1 calibration mixture achieves lower training loss than the selected 1:3 mixture. The authors justify their choice using stability observations from other configurations that are not reported. This leaves the reliability of the calibration procedure unclear. b. The clipping experiments raise similar concerns. Table 3 shows substantial variation across percentiles on CIFAR-10, with accuracy increasing from 85.75% at (p=0.95) to 88.44% at (p=0.85), whereas the LLM experiments favor (p=0.90). These results suggest possible configuration dependence, but no repeated runs or sensitivity analysis establish whether the selected hyperparameters are robust. The CIFAR10 accuracy seems prticulalry low for the baseline and the authors do not explain it.

  3. Insufficient understanding of the proposed mechanisms (Sections 3.2–3.3, 4.2–4.3).

a. Figure 2 establishes improved approximation of exact SVD polarization, but the training experiments do not isolate the contribution of calibrated NS or compare against exact polarization. b. Percentile clipping can change the singular-value distribution and rotate informative singular directions. It might improve finite-step NS accuracy by modifying conditioning; however it could also introduce segnificant directional bias. The paper measures neither effect. Consequently, it remains unclear whether clipping improves optimization through outlier suppression, changes in NS approximation accuracy, or modifications to the update direction.

  1. Limited statistical evidence and reproducibility (Sections 3.2, 4.1–4.6).

a. The reported LLM improvements are modest: 0.53 percentage points in average zero-shot accuracy over Muon and approximately 0.01 lower training loss. No repeated-run statistics, confidence intervals, or random seeds are provided. The CIFAR-10 results also lack uncertainty estimates.

b. Important calibration details are missing, including the numerical coefficient optimization procedure and precise input normalization. Algorithm 1 specifies clipping before normalization but does not define the normalization operator. Section 4.3 additionally reports a final loss of 2.5407, apparently inconsistent with Figure 3.

Taken together, the experiments provide preliminary proof-of-concept evidence, but are insufficient to establish a reproducible and robust optimizer improvement.

Questions for authors
  1. How does shape-specific calibration compare with CANS and Polar Express?

  2. Why is the 1:3 calibration mixture preferred over the better-performing 1:1 configuration, and where is the supporting stability evidence?

  3. How reliable are the gains across random seeds and calibration samples redraws? Can the authors explain the different preferred clipping percentiles in LLM and CIFAR-10 experiments, and report uncertainty estimates for the expeiments?

  4. Does clipping improve NS approximation accuracy? How well correlated are the clipped directions with the uncliped ones post SVD polarozation? Can the authors measure changes in conditioning, leading singular-vector alignment, and polarization error before and after clipping, and isolate the effect of calibrated NS on convergence?

  5. Can the authors provide the complete calibration settings and proceedure and clarify the reported final-loss discrepancy?

Reviewer_H84UOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

the authors develop a new optimizer aiming to improve the training stability. essentially, the authors introduce two modifications for Muon: (1) for each matrix shape they calibrate coeffs of the Newton-Schulz polynomial; (2) during training, before orthogonalization step of Muon, their method does percentile-based entrywise clipping of momentum / gradients. they show that their FIRMO optimizer (Muon + (1) + (2)) improves upon baselines in scalable training and on the downstream evals.

Critical strengths

S1: very easy to adopt the FIRMO optimizer. one can use coefficients from their table.

S2: the method is well-explained, and the authors show that their method attains the lowest relative error compared to Newton-Schulz

Critical weaknesses

W1: weak baselines. the authors mention CANS (Grishina et al., 2025), and Polar Express (Amsel et al., 2025), but do not compare against these works.

W2: no evidence that Newton–Schulz precision matters for training. Fig. 3 shows only SoftThresh and the full method. also no proper tuning of baselines.

Questions for authors
  1. could the authors compare against Polar Express and CANS, at least at the same 5-step budget, both on fidelity and on loss?

  2. could the authors report a wall-clock time for their method, and how the clipping affects it?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.