- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the distinction between preserving a weight matrix's spectral norm and preserving its freedom to update. It derives a sharp step-size bound for two-sided spectral protection and shows that repeated exact protection can lock leading singular directions. The authors propose Spectral Block Correction (SBC), which repairs a selected spectral block before radial restoration and displacement checking, and Regularized Spectral Projection (RSP), which combines shrinkage with projection onto a spectral-norm ball. Experiments cover BERT-Large, ViT-L/16, learning-rate transfer across widths, and SBC ablations. The supplementary archive provides historical implementations, raw logs, transfer grids, and complete ablation trajectories.
Critical strengths
The analysis identifies a concrete failure mode of hard spectral protection. Proposition 2 shows that repeatedly excluding leading singular directions can lock them, even when the weight norm is preserved.
SBC repairs a coupled spectral block and checks the final displacement after rescaling. Appendix B explains why both operations are needed, giving the update rule a clear justification.
The supplementary archive substantially improves traceability. The supplied main-run endpoints, transfer calculations, and response statistics agree with the corresponding raw records, and historical implementation differences are documented.
Critical weaknesses
1.The main comparisons leave the reliability of the gains unclear. BERT Muon skips 644 updates, while SBC uses a different initialization.
2.The transfer ranking is sensitive to source-rate selection. Unequal search grids further limit the comparison.
3.The full ablations do not establish a lasting benefit from block correction. The results support the early advantage but leave the added complexity insufficiently justified. RSP also lacks component ablations.
Questions for authors
1.Does SBC consistently improve on SSO and controlled Muon with shared initialization, stable precision, comparable tuning, and several seeds?
2.Does the transfer ranking survive repeated source evaluations or a near-optimal source-rate rule? Can the missing NR records be supplied?
3.What explains the loss of SBC's early CE advantage? Can RSP ablations separate the effects of shrinkage and projection?
4.What causes the ViT timing gap? Can you compare time-to-quality and performance under matched wall-clock budgets?
4 comments
SBC has lower final MLM loss than the tested Muon-NS5 baseline at all four widths and lower loss than SSO at three widths. At the two larger widths, 1024 and 2048, SBC also has lower MLM loss than both baselines at all five common validation checkpoints: 200, 400, 600, 800, and 1,000 updates. These results show that, in the tested settings, SBC continues to use a fixed source learning rate effectively as model width and training duration increase.
Within widths 256, 512, and 1024, the three methods share the original initialization files. The SBC result at width 2048 uses an externally reconstructed run with the same seed and initialization recipe; tensor equality has not been independently verified, and we will label this run's provenance separately. The new results report actual validation MLM losses to supplement the longer-horizon behavior of fixed rates, rather than conflating them with transfer penalties under the original 200-update protocol.
The original paper includes NR in the cross-width transfer comparison, covering eight learning-rate conditions at each of four widths. NR retains radial restoration and final-displacement checking but omits block correction, providing a simpler control for assessing its contribution. Under the grids and source-width selection protocol reported in the original paper, SBC's mean transfer penalty is 0.0311, compared with 0.0659 for NR. This supplies a direct comparison of cross-width performance against simple radial restoration and, together with the new longer-horizon fixed-rate results, provides evidence at different training horizons.
3. SBC's stage-specific advantage and RSP component contributions (Q3)
3.1 RSP: separating shrinkage and projection under matched settings
To address your request for RSP component ablations, we added small-ViT experiments on CIFAR-100 comparing Muon, shrink-only, projection-only, and full RSP. The four methods share initialization, the data split, training schedule, momentum settings, and auxiliary parameter-group settings. All use momentum 0.9, no Nesterov momentum, and five Newton–Schulz iterations.
Each method is tested on the same learning-rate grid {0.01,0.02,0.04}, with seed 861001. When enabled, shrinkage uses strength 0.05 and projection uses a spectral-radius multiplier of 1.1. Each condition trains for 50 epochs and completes 8,750 successful updates without overflow.
Final validation CE for all 12 conditions is:
| Method | LR=0.01 | LR=0.02 | LR=0.04 |
|---|---|---|---|
| Muon | 2.6684 | 2.4915 | 2.4985 |
| Shrink-only | 2.1261 | 1.8515 | 1.6986 |
| Projection-only | 2.2145 | 1.7974 | 1.5116 |
| Full RSP | 2.0609 | 1.7232 | 1.4543 |
Full RSP achieves the lowest final validation CE at all three common learning rates. It also achieves the lowest CE when each method selects its best setting within the grid using the same final-validation-CE criterion.
These results support the contributions of both components: compared with shrink-only using the same shrinkage mechanism, adding spectral projection further improves validation CE at all three learning rates; compared with projection-only, adding shrinkage also improves CE at all three rates. Thus, the full combination's benefit cannot be attributed solely to either component, nor is it observed only at one selected learning rate.
3.2 SBC: block correction improves optimization progress at the observed stage
For SBC, we further analyzed the paper's existing three-method CIFAR-100 records. The model is a six-layer, width-192 ViT with matrix learning rate 0.02, batch size 256, and seed 831001. SBC-K1 fixes the correction block size to one. NR retains the common Muon proposal, radial restoration, and final-displacement checking, but omits block correction.
At the epoch-30 observation point, retaining the original training configuration and learning-rate schedule, all three methods have completed 5,250 successful updates:
| Method | Train CE | Val. CE | Top-1 (%) | Top-5 (%) |
|---|---|---|---|---|
| SBC | 0.9356 | 1.5757 | 57.48 | 85.14 |
| SBC-K1 | 1.2982 | 1.5818 | 56.56 | 84.98 |
| NR | 1.5151 | 1.7148 | 52.82 | 82.84 |
Training CE is the mean over epoch 30 of the original training trajectory, and validation metrics use the same validation set; the analysis retains the original training configuration and learning-rate schedule.
At the matched update budget, full SBC outperforms both controls in training CE, validation CE, Top-1 accuracy, and Top-5 accuracy. Relative to NR, SBC reduces validation CE by approximately 8.11% and improves Top-1 accuracy by 4.66 percentage points. Because NR retains radial restoration and final-displacement checking, this comparison directly supports the role of additional block correction at this training stage. The comparison with SBC-K1 further examines the difference between adaptive coupled-block correction and fixed one-dimensional correction.
Regarding the later crossings of the validation curves, we assess block correction through its concrete benefits along the optimization trajectory. The controls above compare correction strategies within a common framework of radial restoration and displacement checking: at the matched epoch-30 update budget, full SBC improves both training fit and validation performance. Thus, the effect of block correction is reflected in observable optimization progress, rather than merely in satisfying a spectral-norm constraint.
Section 4.3 further examines this benefit using recorded elapsed time: SBC has lower validation CE than fixed one-dimensional SBC-K1 correction at all listed budgets of 60, 90, and 120 minutes, and lower CE than NR at 60 and 90 minutes. Together with the matched-update results, these comparisons support the value of adaptive coupled-block correction from the perspectives of update progress and recorded elapsed time.
4. Computational overhead, update efficiency, and performance under fixed time budgets (Q4)
4.1 Per-step computational overhead and its sources
The original paper reports timing results from a 200-step BERT training test on a single V100 GPU. The mean complete training-step times for Muon, SSO, and RSP are 25.19, 67.79, and 67.01 seconds, respectively. Their corresponding optimizer times are 1.31, 41.82, and 46.30 seconds per step, while the remaining training work takes approximately 20.71–25.97 seconds. This breakdown shows that the additional time for SSO and RSP relative to Muon primarily comes from optimizer computation, while RSP's overall per-step cost is close to SSO's.
SBC additionally performs near-boundary coupled-block correction, displacement checking after radial restoration, and candidate backtracking when needed, leading to a different computational cost. The original table's SBC value of 119.60 seconds per step is an estimate, separately identified from the directly measured values for the other methods. This reflects the designs' trade-off: SBC emphasizes finer spectral-geometric control, whereas RSP uses a simpler shrinkage-and-projection procedure.
4.2 Update efficiency at a common validation target
The original paper also reports the number of ViT-L/16 updates needed to reach a common validation quality. Taking Muon's final validation CE at 3,000 updates as the target, the interpolated estimates for SSO, RSP, and SBC are 1,906, 2,027, and 1,768 updates, respectively. AdamW does not reach this target within the reported 3,000 updates. Relative to Muon, SBC and RSP require approximately 41.1% and 32.4% fewer updates, respectively.
These results show that additional spectral processing can improve update efficiency under the reported ViT configuration. We present per-step computational cost separately from the number of updates required to reach a target, to clarify the trade-off between optimization progress and additional computation. The BERT timings and ViT update-efficiency results come from different experiments and are not combined into a claim of end-to-end ViT speedup. Please refer to the original paper's training-step timing table, its table of ViT updates to common validation targets, and the appendix subsection “Validation Crossings and Timing” for the results and measurement definitions.
4.3 Additional comparison under fixed elapsed-time budgets
We further analyzed the paper's existing small-ViT CIFAR-100 records, comparing validation CE for SBC, SBC-K1 (correction block size fixed to ), and NR (without block correction) at budgets of 60, 90, and 120 minutes. For each method, we use the last completed validation result within the budget, rather than its best historical value, and do not interpolate at unobserved times.
| Elapsed-time budget | SBC val. CE | SBC-K1 val. CE | NR val. CE |
|---|---|---|---|
| 60 minutes | 2.2611 | 2.3912 | 2.2976 |
| 90 minutes | 2.0603 | 2.2297 | 2.1325 |
| 120 minutes | 1.9964 | 2.1282 | 1.9222 |
At elapsed-time budgets of 60, 90, and 120 minutes, SBC reduces validation CE relative to SBC-K1 by approximately 5.44%, 7.60%, and 6.19%, respectively, achieving better validation performance at every listed budget. At 60 and 90 minutes, SBC also achieves the lowest validation CE among all three methods. These results complement the matched-update quality comparison with a recorded elapsed-time perspective, further supporting the practical value of adaptive coupled-block correction over fixed one-dimensional correction and demonstrating SBC's performance advantage at the reported shorter time budgets.
Thank you for the additional experiments. However, the sustained benefit of block correction remains insufficiently established, so I maintain my rating of 2.
Response to Reviewer 7xp9
Thank you for your comments. Below, we clarify our contribution and present additional experiments and analyses.
1. Shared initialization, numerical stability, and controlled comparisons (Q1)
To address the shortfall in successful updates caused by overflows in the original BERT Muon run, we added a controlled Muon-NS5 experiment with matrix learning rate 0.02, weight decay 0.1, momentum 0.9, no Nesterov momentum, and five Newton–Schulz iterations. This run completed all 2,035 successful updates with zero overflows and zero skipped updates.
The new Muon run is no longer affected by overflows or a shortfall in successful updates. Under the reported configurations, the original SBC/RSP validation endpoints remain lower. The SBC/RSP entries are the original runs; SBC's initialization difference remains documented in Appendix E.2. We will clearly distinguish newly added records from historical ones.
The new cross-width experiments also provide a more direct controlled comparison with shared initialization. At widths 256, 512, and 1024, SBC, SSO, and controlled Muon-NS5 load the same original initialization file within each width, use BF16, and complete the same 1,000 successful updates without overflow. In these settings, SBC has lower final validation MLM loss than Muon-NS5 at all three widths and lower loss than SSO at widths 256 and 1024; the full results are presented in the next section. These new observations do not depend on the initialization difference or Muon's update shortfall in the original BERT main experiment.
For learning-rate selection, the original main experiments used optimizer-appropriate configurations selected from training experience. The new cross-width experiments retain rates selected at the source width without target-width retuning, while the new RSP component experiments compare all component combinations over the same three-rate grid. We will describe these selection protocols separately so that each comparison is tied to the factors it controls.
The additional experiments strengthen the evidence through shared initialization, numerical stability, matched successful-update budgets, and controlled component comparisons. In particular, SBC achieves lower final validation MLM loss than controlled Muon-NS5 at all three widths with shared initialization, providing further support for its optimization performance under these controlled settings.
2. Learning-rate transfer, source-rate selection, and NR records (Q2)
The original transfer experiment organizes candidate learning rates geometrically. For each optimizer, the rate is selected using validation MLM loss at source width 256 and then reused unchanged at widths 512, 1024, and 2048. Target-width validation results do not enter source-rate selection; they are used only to calculate transfer penalties relative to each method's target-width grid minimum. Thus, this protocol tests whether a learning rate selected solely from the source model remains effective as model width increases.
Regarding your question about near-optimal source rates, the sensitivity analysis in the original paper indeed shows that candidate rates with very similar source-width losses can yield different transfer-penalty rankings. We will explicitly qualify this ranking by the reported grids and source-selection rule, distinguishing the practical effectiveness of a fixed source rate from an invariant ranking across all near-optimal source rates.
To further assess fixed source rates over longer training, we extended the cross-width evaluation to 1,000 successful updates. SBC, SSO, and Muon-NS5 use source learning rates of approximately 0.01009076, 0.00848528, and 0.012, respectively, with each method retaining its rate across all four widths.
All 12 main conditions use 24-layer BERT, BF16, batch size 384, sequence length 512, a fixed validation set, and seed 830101. Every condition completes 1,000 successful updates without overflow. Final validation MLM losses are: