acceptodds
Withdrawn from ICLR 2027

Rethinking Spectral Constraints in Muon: Beyond Norm Preservation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Training large neural networks requires optimizers that make effective progress at a practical computational cost. Muon controls the scale of its updates, but does not directly constrain the weights themselves. Spectral constraints address this gap by controlling the scale of weight matrices. However, how these constraints are enforced matters: keeping the norm unchanged does not necessarily preserve the model's ability to learn. We revisit spectral constraints in Muon through their effects on parameter updates and computational cost. We establish conditions for norm preservation and show how certain constraint designs can unnecessarily restrict the changes available to the model. Guided by this analysis, we propose two Muon-based optimizers. Spectral Block Correction (SBC) selectively corrects update blocks while preserving the spectral norm. Regularized Spectral Projection (RSP) shrinks trial weights and caps only singular values exceeding a prescribed bound. Under the reported single-seed configurations, SBC reduces final BERT-Large validation loss on Wikipedia by 14.3% versus AdamW. On Imagenette, SBC and RSP reduce final ViT-L/16 validation cross-entropy by 16.2% and 13.2%, respectively, versus Muon. SBC also reduces the mean cross-width learning-rate transfer penalty by 69.6% relative to Muon. These results highlight that spectral control must account for learning and per-step cost, not norm preservation alone.

Primary area:
optimization
Keywords:
Muon, spectral constraints, matrix optimization, subspace methods, neural network training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7xp9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the distinction between preserving a weight matrix's spectral norm and preserving its freedom to update. It derives a sharp step-size bound for two-sided spectral protection and shows that repeated exact protection can lock leading singular directions. The authors propose Spectral Block Correction (SBC), which repairs a selected spectral block before radial restoration and displacement checking, and Regularized Spectral Projection (RSP), which combines shrinkage with projection onto a spectral-norm ball. Experiments cover BERT-Large, ViT-L/16, learning-rate transfer across widths, and SBC ablations. The supplementary archive provides historical implementations, raw logs, transfer grids, and complete ablation trajectories.

Critical strengths

The analysis identifies a concrete failure mode of hard spectral protection. Proposition 2 shows that repeatedly excluding leading singular directions can lock them, even when the weight norm is preserved.

SBC repairs a coupled spectral block and checks the final displacement after rescaling. Appendix B explains why both operations are needed, giving the update rule a clear justification.

The supplementary archive substantially improves traceability. The supplied main-run endpoints, transfer calculations, and response statistics agree with the corresponding raw records, and historical implementation differences are documented.

Critical weaknesses

1.The main comparisons leave the reliability of the gains unclear. BERT Muon skips 644 updates, while SBC uses a different initialization.

2.The transfer ranking is sensitive to source-rate selection. Unequal search grids further limit the comparison.

3.The full ablations do not establish a lasting benefit from block correction. The results support the early advantage but leave the added complexity insufficiently justified. RSP also lacks component ablations.

Questions for authors

1.Does SBC consistently improve on SSO and controlled Muon with shared initialization, stable precision, comparable tuning, and several seeds?

2.Does the transfer ranking survive repeated source evaluations or a near-optimal source-rate rule? Can the missing NR records be supplied?

3.What explains the loss of SBC's early CE advantage? Can RSP ablations separate the effects of shrinkage and projection?

4.What causes the ViT timing gap? Can you compare time-to-quality and performance under matched wall-clock budgets?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 7xp9

Thank you for your comments. Below, we clarify our contribution and present additional experiments and analyses.

1. Shared initialization, numerical stability, and controlled comparisons (Q1)

To address the shortfall in successful updates caused by overflows in the original BERT Muon run, we added a controlled Muon-NS5 experiment with matrix learning rate 0.02, weight decay 0.1, momentum 0.9, no Nesterov momentum, and five Newton–Schulz iterations. This run completed all 2,035 successful updates with zero overflows and zero skipped updates.

MethodSuccessful updatesVal. loss (MLM+NSP)Val. MLM loss
Muon-NS5, matrix WD=0.12,0358.36927.6682
RSP2,0347.93867.4010
SBC2,0357.06226.7915

The new Muon run is no longer affected by overflows or a shortfall in successful updates. Under the reported configurations, the original SBC/RSP validation endpoints remain lower. The SBC/RSP entries are the original runs; SBC's initialization difference remains documented in Appendix E.2. We will clearly distinguish newly added records from historical ones.

The new cross-width experiments also provide a more direct controlled comparison with shared initialization. At widths 256, 512, and 1024, SBC, SSO, and controlled Muon-NS5 load the same original initialization file within each width, use BF16, and complete the same 1,000 successful updates without overflow. In these settings, SBC has lower final validation MLM loss than Muon-NS5 at all three widths and lower loss than SSO at widths 256 and 1024; the full results are presented in the next section. These new observations do not depend on the initialization difference or Muon's update shortfall in the original BERT main experiment.

For learning-rate selection, the original main experiments used optimizer-appropriate configurations selected from training experience. The new cross-width experiments retain rates selected at the source width without target-width retuning, while the new RSP component experiments compare all component combinations over the same three-rate grid. We will describe these selection protocols separately so that each comparison is tied to the factors it controls.

The additional experiments strengthen the evidence through shared initialization, numerical stability, matched successful-update budgets, and controlled component comparisons. In particular, SBC achieves lower final validation MLM loss than controlled Muon-NS5 at all three widths with shared initialization, providing further support for its optimization performance under these controlled settings.

2. Learning-rate transfer, source-rate selection, and NR records (Q2)

The original transfer experiment organizes candidate learning rates geometrically. For each optimizer, the rate is selected using validation MLM loss at source width 256 and then reused unchanged at widths 512, 1024, and 2048. Target-width validation results do not enter source-rate selection; they are used only to calculate transfer penalties relative to each method's target-width grid minimum. Thus, this protocol tests whether a learning rate selected solely from the source model remains effective as model width increases.

Regarding your question about near-optimal source rates, the sensitivity analysis in the original paper indeed shows that candidate rates with very similar source-width losses can yield different transfer-penalty rankings. We will explicitly qualify this ranking by the reported grids and source-selection rule, distinguishing the practical effectiveness of a fixed source rate from an invariant ranking across all near-optimal source rates.

To further assess fixed source rates over longer training, we extended the cross-width evaluation to 1,000 successful updates. SBC, SSO, and Muon-NS5 use source learning rates of approximately 0.01009076, 0.00848528, and 0.012, respectively, with each method retaining its rate across all four widths.

All 12 main conditions use 24-layer BERT, BF16, batch size 384, sequence length 512, a fixed validation set, and seed 830101. Every condition completes 1,000 successful updates without overflow. Final validation MLM losses are:

WidthSBCSSOMuon-NS5
2566.9523666.9726606.983990
5126.4815386.4807906.523921
10246.1509646.1612036.224148
20486.0503516.0786636.169502

SBC has lower final MLM loss than the tested Muon-NS5 baseline at all four widths and lower loss than SSO at three widths. At the two larger widths, 1024 and 2048, SBC also has lower MLM loss than both baselines at all five common validation checkpoints: 200, 400, 600, 800, and 1,000 updates. These results show that, in the tested settings, SBC continues to use a fixed source learning rate effectively as model width and training duration increase.

Within widths 256, 512, and 1024, the three methods share the original initialization files. The SBC result at width 2048 uses an externally reconstructed run with the same seed and initialization recipe; tensor equality has not been independently verified, and we will label this run's provenance separately. The new results report actual validation MLM losses to supplement the longer-horizon behavior of fixed rates, rather than conflating them with transfer penalties under the original 200-update protocol.

The original paper includes NR in the cross-width transfer comparison, covering eight learning-rate conditions at each of four widths. NR retains radial restoration and final-displacement checking but omits block correction, providing a simpler control for assessing its contribution. Under the grids and source-width selection protocol reported in the original paper, SBC's mean transfer penalty is 0.0311, compared with 0.0659 for NR. This supplies a direct comparison of cross-width performance against simple radial restoration and, together with the new longer-horizon fixed-rate results, provides evidence at different training horizons.

3. SBC's stage-specific advantage and RSP component contributions (Q3)

3.1 RSP: separating shrinkage and projection under matched settings

To address your request for RSP component ablations, we added small-ViT experiments on CIFAR-100 comparing Muon, shrink-only, projection-only, and full RSP. The four methods share initialization, the data split, training schedule, momentum settings, and auxiliary parameter-group settings. All use momentum 0.9, no Nesterov momentum, and five Newton–Schulz iterations.

Each method is tested on the same learning-rate grid {0.01,0.02,0.04}, with seed 861001. When enabled, shrinkage uses strength 0.05 and projection uses a spectral-radius multiplier of 1.1. Each condition trains for 50 epochs and completes 8,750 successful updates without overflow.

Final validation CE for all 12 conditions is:

MethodLR=0.01LR=0.02LR=0.04
Muon2.66842.49152.4985
Shrink-only2.12611.85151.6986
Projection-only2.21451.79741.5116
Full RSP2.06091.72321.4543

Full RSP achieves the lowest final validation CE at all three common learning rates. It also achieves the lowest CE when each method selects its best setting within the grid using the same final-validation-CE criterion.

These results support the contributions of both components: compared with shrink-only using the same shrinkage mechanism, adding spectral projection further improves validation CE at all three learning rates; compared with projection-only, adding shrinkage also improves CE at all three rates. Thus, the full combination's benefit cannot be attributed solely to either component, nor is it observed only at one selected learning rate.

3.2 SBC: block correction improves optimization progress at the observed stage

For SBC, we further analyzed the paper's existing three-method CIFAR-100 records. The model is a six-layer, width-192 ViT with matrix learning rate 0.02, batch size 256, and seed 831001. SBC-K1 fixes the correction block size to one. NR retains the common Muon proposal, radial restoration, and final-displacement checking, but omits block correction.

At the epoch-30 observation point, retaining the original training configuration and learning-rate schedule, all three methods have completed 5,250 successful updates:

MethodTrain CEVal. CETop-1 (%)Top-5 (%)
SBC0.93561.575757.4885.14
SBC-K11.29821.581856.5684.98
NR1.51511.714852.8282.84

Training CE is the mean over epoch 30 of the original training trajectory, and validation metrics use the same validation set; the analysis retains the original training configuration and learning-rate schedule.

At the matched update budget, full SBC outperforms both controls in training CE, validation CE, Top-1 accuracy, and Top-5 accuracy. Relative to NR, SBC reduces validation CE by approximately 8.11% and improves Top-1 accuracy by 4.66 percentage points. Because NR retains radial restoration and final-displacement checking, this comparison directly supports the role of additional block correction at this training stage. The comparison with SBC-K1 further examines the difference between adaptive coupled-block correction and fixed one-dimensional correction.

Regarding the later crossings of the validation curves, we assess block correction through its concrete benefits along the optimization trajectory. The controls above compare correction strategies within a common framework of radial restoration and displacement checking: at the matched epoch-30 update budget, full SBC improves both training fit and validation performance. Thus, the effect of block correction is reflected in observable optimization progress, rather than merely in satisfying a spectral-norm constraint.

Section 4.3 further examines this benefit using recorded elapsed time: SBC has lower validation CE than fixed one-dimensional SBC-K1 correction at all listed budgets of 60, 90, and 120 minutes, and lower CE than NR at 60 and 90 minutes. Together with the matched-update results, these comparisons support the value of adaptive coupled-block correction from the perspectives of update progress and recorded elapsed time.

4. Computational overhead, update efficiency, and performance under fixed time budgets (Q4)

4.1 Per-step computational overhead and its sources

The original paper reports timing results from a 200-step BERT training test on a single V100 GPU. The mean complete training-step times for Muon, SSO, and RSP are 25.19, 67.79, and 67.01 seconds, respectively. Their corresponding optimizer times are 1.31, 41.82, and 46.30 seconds per step, while the remaining training work takes approximately 20.71–25.97 seconds. This breakdown shows that the additional time for SSO and RSP relative to Muon primarily comes from optimizer computation, while RSP's overall per-step cost is close to SSO's.

SBC additionally performs near-boundary coupled-block correction, displacement checking after radial restoration, and candidate backtracking when needed, leading to a different computational cost. The original table's SBC value of 119.60 seconds per step is an estimate, separately identified from the directly measured values for the other methods. This reflects the designs' trade-off: SBC emphasizes finer spectral-geometric control, whereas RSP uses a simpler shrinkage-and-projection procedure.

4.2 Update efficiency at a common validation target

The original paper also reports the number of ViT-L/16 updates needed to reach a common validation quality. Taking Muon's final validation CE at 3,000 updates as the target, the interpolated estimates for SSO, RSP, and SBC are 1,906, 2,027, and 1,768 updates, respectively. AdamW does not reach this target within the reported 3,000 updates. Relative to Muon, SBC and RSP require approximately 41.1% and 32.4% fewer updates, respectively.

These results show that additional spectral processing can improve update efficiency under the reported ViT configuration. We present per-step computational cost separately from the number of updates required to reach a target, to clarify the trade-off between optimization progress and additional computation. The BERT timings and ViT update-efficiency results come from different experiments and are not combined into a claim of end-to-end ViT speedup. Please refer to the original paper's training-step timing table, its table of ViT updates to common validation targets, and the appendix subsection “Validation Crossings and Timing” for the results and measurement definitions.

4.3 Additional comparison under fixed elapsed-time budgets

We further analyzed the paper's existing small-ViT CIFAR-100 records, comparing validation CE for SBC, SBC-K1 (correction block size fixed to k=1k=1), and NR (without block correction) at budgets of 60, 90, and 120 minutes. For each method, we use the last completed validation result within the budget, rather than its best historical value, and do not interpolate at unobserved times.

Elapsed-time budgetSBC val. CESBC-K1 val. CENR val. CE
60 minutes2.26112.39122.2976
90 minutes2.06032.22972.1325
120 minutes1.99642.12821.9222

At elapsed-time budgets of 60, 90, and 120 minutes, SBC reduces validation CE relative to SBC-K1 by approximately 5.44%, 7.60%, and 6.19%, respectively, achieving better validation performance at every listed budget. At 60 and 90 minutes, SBC also achieves the lowest validation CE among all three methods. These results complement the matched-update quality comparison with a recorded elapsed-time perspective, further supporting the practical value of adaptive coupled-block correction over fixed one-dimensional correction and demonstrating SBC's performance advantage at the reported shorter time budgets.

Thank you for the additional experiments. However, the sustained benefit of block correction remains insufficiently established, so I maintain my rating of 2.

Reviewer_eRcgOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes two Muon variants that seek to control the norm of weight matrices. The authors show that weight norms remain bounded under certain conditions, and experimental results claim improvements over Muon on a few training tasks.

Critical strengths

I cannot find a significant strength of the paper unless the weaknesses below are addressed.

Critical weaknesses

Unless I am missing something, there seems to be critically missing related work [1, 2]. [3] is cited, but its implications are not fully discussed. In particular, one can provably control the spectral norm of weight matrices by simply adding decoupled weight decay to Muon. The AdamW and Muon baselines for the experiments use no weight decay or unusually small weight decay, as seen in Table 6. It is difficult to argue for any significance of the presented results without, as a very first step, a direct comparison to Muon with a standard weight decay value, e.g. 0.10.1 [3]. The experimental results are also significantly weakened by the use of a single seed and inconsistent momentum configurations between Muon and SBC/RSP.

References

[1] Lizhang Chen, Jonathan Li, Qiang Liu. Muon Optimizes Under Spectral Norm Constraints. TMLR 2026.

[2] Maria-Eleni Sfyraki, Jun-Kun Wang. Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise. ICML 2026.

[3] Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, Volkan Cevher. Training Deep Learning Models with Norm-Constrained LMOs. ICML 2025.

[4] Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, Zhilin Yang. Muon is Scalable for LLM Training. 2025.

Questions for authors
  1. Could the authors compare SBC/RSP with Muon with weight decay 0.10.1?
  2. The authors explicitly state that some hyperparameters (most importantly, learning rates) were not tuned. Could the authors properly tune all hyperparameters for the experiments?
4 comments
AuthorsOpenReview ↗

Response to Reviewer eRcg

Thank you for your comments. Below, we clarify our contribution and present additional experiments and analyses.

1. Direct comparison with Muon using weight decay 0.1 (Q1)

Following your suggestion, we added a controlled Muon-NS5 experiment on BERT-Large with matrix weight decay 0.1. This run uses a matrix learning rate of 0.02, momentum 0.9, no Nesterov momentum, and five Newton–Schulz steps. It completes all 2,035 successful updates, with zero overflows and zero skipped updates.

MethodSourceSuccessful updatesValidation loss (MLM+NSP)Validation MLM loss
Muon-NS5, matrix WD=0.1New run2,0358.36927.6682
RSPOriginal run2,0347.93867.4010
SBCOriginal run2,0357.06226.7915

This additional result directly supplies the weight-decay control you requested and removes overflow-related update shortfalls from this Muon run. Under the reported configurations, both SBC and RSP retain lower total validation loss and lower validation MLM loss than this additional baseline.

The SBC and RSP entries are from the original manuscript; SBC's initialization difference is documented in Appendix E.2. We will clearly identify the sources and configurations of these results and include the additional Muon baseline in the revised manuscript.

2. Related work and our contribution: distinctions and connections between existing constraint theory and update-geometry analysis

Thank you for pointing out these related works. We would like to clarify that our central question is not whether the spectral norm of a weight matrix can be controlled, but how the specific enforcement of a constraint affects available update directions and optimization progress. These questions are closely related, but their objects of study and theoretical conclusions are different.

Chen et al. [1] explain the implicit spectral-norm constraints of Muon with weight decay within the Lion-𝒦 framework and study the corresponding convergence properties. Sfyraki and Wang [2] analyze Lion/Muon with weight decay through stochastic Frank–Wolfe and consider heavy-tailed noise. These works provide important foundations for understanding the constraint structure and optimization properties of these optimizers. Our work examines a different, concrete question: even when the spectral norm is controlled, can a particular constraint-enforcement rule still exclude useful update directions and thereby restrict optimization? Accordingly, our work does not assume that Muon with weight decay cannot control the spectral norm. Instead, it investigates update-geometry questions that existing norm-control results do not directly answer: how specific spectral-protection rules restrict safe step sizes, exclude available update components, and even induce singular-direction locking, and uses this analysis to design corresponding spectral-correction mechanisms.

Pethick et al. [3] study optimization methods based on linear minimization oracles over norm balls, architecture-adapted geometry, and hyperparameter transfer. Our work shares this norm-constrained optimization background but focuses on how specific protection and correction operations affect updates. Correspondingly, our theoretical results concern step-size restrictions under hard protection, the trade-off between the extent of protection and layer responses, and singular-direction locking under repeated protection. Our algorithmic designs construct spectral-correction mechanisms to address these geometric issues. This emphasis differs from establishing a norm-constrained optimization framework and studying its training properties.

Specifically, Proposition 1 characterizes a uniform safe step-size restriction under exact two-sided hard protection. For directions satisfying the corresponding left- and right-subspace exclusion conditions and having spectral norm at most one, the step size that guarantees no increase beyond the original spectral norm uniformly over this entire class is bounded by

σ1(W)−σk+1(W).\sigma_1(W)-\sigma_{k+1}(W).

We also analyze the relationship between the extent of protection and the layer's input response: protecting more singular directions may enlarge the safe step-size range while removing more update components. Thus, determining whether the weight norm is controlled is insufficient to characterize effective updates under this rule.

Proposition 2 further shows that, when the spectral norm is fixed and exact two-sided exclusion is repeatedly enforced, the initial dominant singular directions are constrained by an invariant; recomputing the protected subspaces at every step need not release this locking. This conclusion concerns the specified hard-protection rule and identifies a situation in which the norm is correctly preserved while freedom to update is restricted. It is an analysis of concrete update dynamics, rather than another proof that weight norms are bounded.

This focus also directly informs the method designs. SBC jointly corrects a coupled block near the spectral boundary, rather than zeroing all update components involving the corresponding directions, and checks the actual executed displacement after radial restoration. RSP combines shrinkage with explicit spectral projection, treats shrinkage strength and the spectral upper bound as separate design quantities, and caps only violating singular values during projection. These designs concern how constraints are enforced, which update components require correction, and how the correction affects the actual update, rather than merely whether a norm-controlling decay term is included.

We therefore view these works and ours as complementary: prior research establishes constraint interpretations and optimization properties of the relevant optimizers, whereas our work characterizes geometric restrictions introduced by specific constraint-enforcement rules and proposes corresponding update designs. Existing norm-control results do not imply that the update-geometry questions analyzed in our paper have already been resolved. In the revised Related Work section, we will add and discuss [1] and [2], expand our discussion of [3], and explicitly present the distinctions and connections in research questions, theoretical conclusions, and algorithmic operations.

3. Learning-rate selection and component benefits (Q2)

For the main experiments, we selected learning rates appropriate to each optimizer and task based on training experience and reported the full configurations in Appendix E.2 and Table 6. On BERT-Large, the matrix learning rate is 0.02 for Muon, SSO, SBC, and RSP. On ViT-L/16, it is 0.02 for Muon and 0.01 for SSO, SBC, and RSP. AdamW uses 0.0001 and 0.001 for the two tasks, respectively.

To further examine learning-rate selection and component effects, we added small-ViT experiments on CIFAR-100 comparing Muon, shrink-only, projection-only, and full RSP over the same learning-rate grid {0.01,0.02,0.04}. All conditions share the initialization and matched training configurations, with seed 861001. Each condition trains for 50 epochs and completes 8,750 successful updates without overflow. The table below reports final validation CE for all 12 conditions.

MethodLR=0.01LR=0.02LR=0.04
Muon2.66842.49152.4985
Shrink-only2.12611.85151.6986
Projection-only2.21451.79741.5116
Full RSP2.06091.72321.4543

Full RSP achieves the lowest final validation CE at all three common learning rates. It also achieves the lowest CE when each method selects its best configuration within this grid using the same final-validation-CE criterion.

In particular, full RSP outperforms shrink-only, which uses the same shrinkage mechanism, at all three learning rates. This directly supports the additional role of spectral projection beyond shrinkage; the benefit of the complete combination in this component experiment cannot simply be attributed to adding shrinkage regularization. Full RSP also outperforms projection-only at all three learning rates, supporting the combined design.

Thus, the additional experiments provide comparisons under a common learning-rate selection protocol and directly separate the contributions of shrinkage and projection. The benefit of the full combination is observed at multiple common learning rates, rather than only at one selected configuration. The BERT experiment in Section 1 addresses the requested WD=0.1 baseline, while these component experiments examine the respective roles of shrinkage, projection, and learning-rate selection.

4. Momentum and update-direction configurations

Regarding momentum configurations, it is important to distinguish the controlled comparisons from the native Muon implementation in the original manuscript.

The original BERT experiments already use momentum 0.9, no Nesterov momentum, and a common five-step Newton–Schulz direction construction for controlled Muon-NS5, SBC, and RSP. The additional Muon run with WD=0.1 uses the same settings. Thus, these controlled BERT comparisons do not have the stated differences in momentum coefficient or Nesterov configuration.

The original ViT Muon baseline uses the native implementation, with momentum 0.95 and Nesterov momentum enabled; these settings are separately documented in the appendix. The additional CIFAR-100 component experiments use momentum 0.9, no Nesterov momentum, and five Newton–Schulz steps for all four methods, while matching initialization, the training schedule, and auxiliary parameter-group configurations.

Consequently, the additional component experiments isolate shrinkage and projection on a common update foundation, and the observed differences cannot be attributed to different momentum coefficients or Nesterov settings across methods. We will further highlight the distinction between controlled and native implementations in the revision to clarify the configurations used in each experiment.

5. Expanded experimental coverage: tasks, components, learning rates, and model widths

Beyond the original BERT-Large/Wikipedia and ViT-L/16/Imagenette experiments, we expanded the empirical evaluation across baseline configurations, component mechanisms, learning rates, and model widths.

For baseline configurations, we added a Muon-NS5 run with matrix WD=0.1 that completes 2,035 successful updates without overflow. For components and learning rates, we completed 12 RSP ablation conditions covering four methods and three learning rates. Each condition trains for the full 50 epochs, and all final validation results are reported under the same protocol.

For SBC's mechanism, we further analyzed the paper's existing three-method CIFAR-100 records. At the 5,250-update checkpoint, retaining the original training configuration and learning-rate schedule with seed 831001, SBC, SBC-K1, and NR have validation CE values of 1.5757, 1.5818, and 1.7148, respectively, and Top-1 accuracies of 57.48%, 56.56%, and 52.82%. These comparisons examine both the presence of block correction and the use of an adaptive block size, supporting SBC's optimization benefits at this stage of training.

For model width and training duration, we extended the fixed-source-learning-rate cross-width evaluation to 1,000 successful updates. It covers SBC, SSO, and Muon-NS5 at widths 256, 512, 1024, and 2048, yielding 12 main conditions with seed 830101, all completed without overflow. Each method retains its own source learning rate across all four widths. SBC has lower final MLM loss than the tested Muon-NS5 baseline at all four widths and lower loss than SSO at three widths. At the two larger widths, 1024 and 2048, its MLM loss is lower than both baselines at all five common validation checkpoints.

I thank the authors for their response. Much of the response appears to be AI-generated, which makes it difficult to accurately assess the contributions of this paper. I maintain my rating and strongly encourage the authors to improve the quality of the writing.

Reviewer_3bV8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how spectral constraints affect Muon’s optimization dynamics, showing that hard protection of dominant singular directions can restrict effective updates and cause singular-direction locking. To address these limitations, it proposes two optimizers: Spectral Block Correction (SBC), which selectively corrects updates while preserving spectral norm, and Regularized Spectral Projection (RSP), which combines weight shrinkage with spectral clipping. Experiments on BERT-Large, ViT-L/16, and cross-width learning-rate transfer demonstrate promising improvements in validation performance.

Critical strengths

The paper identifies a concrete and consequential failure mode of hard spectral protection. Propositions 1–2 and Appendix A connect a sharp uniform step-size restriction, loss of available update components, and an invariant that can prevent a model from reaching a target on the same spectral sphere. This is a useful contribution to understanding constrained matrix optimizers and gives a principled motivation for allowing more flexible updates.

Critical weaknesses

The central performance and learning-rate transfer claims lack sufficiently robust experimental support:

The BERT comparison has material confounds. Muon completes only 1,391 successful updates, versus SBC’s 2,035, because of 644 overflow skips; SBC also has a different initialization-artifact hash (Appendix E.2, Table 7). The overflow discrepancy specifically compromises the Muon comparison. Moreover, in the ViT experiment, the improvement over the strongest constrained baseline is much smaller: SBC achieves 0.9604 cross-entropy versus SSO’s 0.9739, a 0.30 percentage-point accuracy difference, while RSP underperforms SSO (Table 1).

The transfer result is also sensitive to source-rate selection. Table 11 shows that an alternative SSO learning rate increases source-width loss by only 0.0002184 but reduces its mean transfer penalty from 0.1127 to 0.0159, below SBC’s selected-rate penalty of 0.0311. SBC also improves with an alternative source rate. Thus, the result demonstrates sensitivity to nearly tied source losses; it does not establish a robust ordering among methods. Unequal grids and the 200-update evaluation horizon further limit the strength of the headline transfer percentages. These limitations are disclosed, but they substantially reduce the evidentiary weight of the empirical contribution.

Questions for authors
  1. Were the learning rates adequately tuned for each optimizer in the main BERT-Large and ViT-L/16 experiments? Table 6 reports fixed learning-rate configurations, but no comprehensive or comparable learning-rate sweeps are provided. Could the authors clarify how these learning rates were selected?

  2. Introducing two distinct optimizers in a single paper is somewhat confusing. Could the authors clarify which optimizer is preferable, or under what conditions each optimizer is expected to perform better? Are there any trade-offs between SBC and RSP in terms of optimization performance, computational efficiency, or practical applicability?

  3. Validation of practical benefit: Can you provide directly measured time or GPU-hours to a common validation target on the same task and hardware, including SBC and a simpler normalization control such as NR? This would materially affect whether the additional spectral computations appear worthwhile.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 3bV8

Thank you for the helpful comments. We present the additional experiments and analyses addressing your concerns below.

1. BERT overflows, successful updates, and the main results

To address the reduced number of successful updates in the original Muon run, we added a controlled Muon-NS5 experiment with matrix learning rate 0.02, weight decay 0.1, momentum 0.9, no Nesterov momentum, and five Newton–Schulz steps. This run completed all 2,035 successful updates with zero overflows and zero skipped updates.

Method and sourceUpdatesOverflowsMLM+NSPMLM
Muon-NS5, original, WD=01,3916449.12698.0909
Muon-NS5, new, WD=0.12,03508.36927.6682
RSP, original2,03417.93867.4010
SBC, original2,03507.06226.7915

The new Muon baseline removes the overflow and update-shortfall factors from that run and improves upon the original Muon endpoint. The validation losses reported for SBC and RSP remain lower. The SBC/RSP entries are the original runs, and SBC's initialization difference remains documented in Appendix E.2; these numbers are results under the reported configurations, rather than effect estimates from a shared-initialization paired experiment.

For the ViT comparison with SSO, SBC achieves validation CE 0.9604 versus 0.9739, with Top-1 accuracy 73.65% versus 73.35%. We interpret this as an incremental improvement over a strong constrained baseline, rather than a large margin. RSP does not outperform SSO in this setting. The roles of the two designs additionally concern constraint enforcement, component effects, and computational trade-offs, as discussed below.

2. Learning-rate selection and additional component experiments (Q1)

For the main experiments, we selected learning rates appropriate to each optimizer and task based on training experience, with the full configurations reported in Appendix E.2 and Table 6. On BERT-Large, the matrix learning rate is 0.02 for Muon, SSO, SBC, and RSP. On ViT-L/16, it is 0.02 for Muon and 0.01 for SSO, SBC, and RSP. AdamW uses 0.0001 and 0.001, respectively.

To further examine component effects across learning rates, we added small-ViT experiments on CIFAR-100 comparing Muon, shrink-only, projection-only, and full RSP. All four methods share the initialization, matched training settings, and learning-rate grid {0.01,0.02,0.04}. With seed 861001, every condition completes 50 epochs and 8,750 successful updates without overflow. The final validation CE for all 12 conditions is:

MethodLR=0.01LR=0.02LR=0.04
Muon2.66842.49152.4985
Shrink-only2.12611.85151.6986
Projection-only2.21451.79741.5116
Full RSP2.06091.72321.4543

Full RSP achieves the lowest final validation CE at every tested common learning rate. It also achieves the lowest CE when each method selects its best setting within this grid using the same final-validation-CE criterion. Thus, in this component experiment, the benefit of combining shrinkage and projection is observed across all three tested learning rates, rather than only at one selected setting.

3. Original geometric search and new longer-horizon transfer evaluation

The original learning-rate transfer experiment organizes candidate rates geometrically, examining multiplicative changes rather than arbitrary linear increments. For each optimizer, we select the learning rate using validation MLM loss at source width 256 and reuse it unchanged at widths 512, 1024, and 2048. Target-width grid minima serve only as references for computing transfer penalties; they are not used to select the source learning rate. The source-rate sensitivity analysis in the original appendix provides additional context for this selection protocol.

We extended the cross-width evaluation from 200 to 1,000 successful updates to assess performance over a longer training horizon. SBC, SSO, and Muon-NS5 use the source-width learning rates 0.010090756983044574, 0.00848528137423857, and 0.012, respectively, keeping each rate fixed across all four widths.

All 12 conditions use 24-layer BERT, batch size 384, sequence length 512, BF16, seed 830101, and the same fixed validation set. Every condition completes 1,000 successful updates without overflow. Validation MLM loss at update 1,000 is:

WidthSBCSSOMuon-NS5
2566.9523666.9726606.983990
5126.4815386.4807906.523921
10246.1509646.1612036.224148
20486.0503516.0786636.169502

SBC has lower final MLM loss than the tested Muon-NS5 baseline at all four widths and lower loss than SSO at three widths. At the two larger widths, 1024 and 2048, SBC has lower MLM loss than both baselines at all five common validation checkpoints: 200, 400, 600, 800, and 1,000 updates.

These results show that, in the tested settings, SBC can continue to use a fixed source learning rate effectively as model width and training duration increase. The original experiment evaluates learning-rate selection and transfer penalties on the geometric grids; the new experiment supplies longer-horizon validation performance at fixed source rates. These are complementary evaluations.

4. Why two optimizers, and when to use each (Q2)

SBC and RSP are two designs motivated by the same theoretical insight, rather than two unrelated optimizers. Our central finding is that preserving the spectral norm does not necessarily preserve freedom to update: rigid protection of dominant singular directions can restrict useful updates and induce direction locking. This motivates more flexible ways to retain useful updates while controlling spectral scale.

SBC more directly implements the geometric analysis for a fixed spectral radius. It jointly corrects a near-boundary singular-direction block and checks the actual final displacement after radial restoration. This more detailed enforcement entails block correction, candidate checking, and possible backtracking. Under the source-rate selection protocol reported in the paper, SBC achieves a mean learning-rate transfer penalty of 0.0311, compared with 0.0703 for RSP. The new experiments above further demonstrate SBC's longer-horizon cross-width performance relative to SSO and Muon-NS5.

RSP emphasizes a simpler computational realization of the same principle. It combines trial-weight shrinkage with projection onto a spectral-norm ball, allows the weight norm to decrease within the ball, and avoids SBC's block-correction and candidate-backtracking procedures. RSP also has a proximal interpretation and feasibility guarantees; its role is to simplify constraint enforcement, not to dispense with theoretical justification. The component results in Section 2 additionally show validation-CE benefits from combining shrinkage and projection beyond either component alone.

Accordingly, SBC is the corresponding choice when maintaining a fixed spectral radius and controlling update geometry precisely are central requirements. RSP is a more direct option when norm shrinkage is acceptable, a spectral upper bound is sufficient, and simplifying the computational procedure is a priority. SBC's reported learning-rate transfer performance and RSP's computational simplification reflect these different emphases, rather than a claim that one method dominates across all tasks.

5. NR control: quality at a 30-epoch budget and time to target (Q3)

We further analyzed the existing CIFAR-100 experimental records from the paper, comparing SBC, SBC-K1, and NR at a 30-epoch evaluation budget while retaining the originally reported training configuration and learning-rate schedule. The model is a six-layer, width-192 ViT, with 45,000 training images, 5,000 validation images, matrix learning rate 0.02, batch size 256, and seed 831001. NR retains the common Muon proposal, radial restoration, and final-displacement checking, but omits block correction; SBC-K1 fixes the block size to one.

After the same 5,250 successful updates, the results are as follows. Training CE is the mean over epoch 30; all validation metrics use the same validation set.

MethodTrain CEVal. CETop-1 (%)Top-5 (%)
SBC0.93561.575757.4885.14
SBC-K11.29821.581856.5684.98
NR1.51511.714852.8282.84

At this matched update budget, SBC achieves the lowest training and validation CE and the highest Top-1 and Top-5 accuracy. Relative to NR, validation CE decreases by approximately 8.11%, and Top-1 accuracy increases by 4.66 percentage points, supporting the optimization contribution of block correction at this stage of training.

By further analyzing the paper's existing training and validation records, we also computed elapsed times to common validation targets. Validation is performed once per epoch, and the target is considered reached at the first recorded checkpoint with validation CE no greater than the threshold. Each entry below gives “epoch / cumulative minutes,” using the logged end-to-end time without interpolation.

Validation CE targetSBCSBC-K1NR
2.505 / 38.485 / 39.267 / 35.63
2.0013 / 109.4519 / 159.8123 / 119.25
1.9016 / 136.2121 / 177.0026 / 134.74
1.8019 / 163.0625 / 211.1827 / 139.88
1.7522 / 189.8827 / 228.1730 / 155.08
NR epoch-30 CE (≈ 1.7148)23 / 198.7927 / 228.1730 / 155.08

At every listed common target, SBC requires fewer epochs than NR. Relative to SBC-K1, SBC requires fewer or equal epochs and has lower recorded elapsed time at every listed target. Its time comparison with NR depends on the target quality, illustrating the trade-off between improved progress per update and the additional per-update cost of finer block correction. These historical timings are reported as observed in the original logs, rather than as a newly executed matched-hardware benchmark.

Together with the quality comparison at a fixed update budget, these controls show that SBC's additional structure can improve optimization progress at this training stage while exposing its computational cost. This is consistent with the two methods' intended roles: finer spectral-geometric control for SBC, and a simpler constraint-enforcement procedure for RSP.

Thank you to the authors for their response. However, I still have the following concerns and therefore maintain my current Weak Reject rating.

First, I do not find it reasonable to conduct a learning-rate search over such a narrow range (0.01, 0.02, 0.04). In particular, the best results for shrink-only, projection-only, and full RSP all occur at LR = 0.04, which lies at the boundary of the search range. This suggests that the optimal learning rates may lie outside the tested range, making it difficult to draw convincing conclusions from the current comparisons.

Second, the authors state that: “For the main experiments, we selected learning rates appropriate to each optimizer and task based on training experience.” Could the authors clarify what specific training experience informed these choices? What is the rationale for selecting these particular learning rates for different optimizers?

Furthermore, why do the authors disable Nesterov momentum in most of their experiments? The official Muon implementation recommends using Nesterov momentum, yet the authors predominantly use configurations without it. Could the authors explain the rationale behind this choice?

Finally, the authors propose two optimizers, SBC and RSP. *Which optimizer would the authors recommend for practical training, and why? It would be helpful to provide a clearer recommendation based on their respective optimization performance, computational overhead, and practical trade-offs.

Thanks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.