acceptodds
Withdrawn from ICLR 2027

Does Muon Need Fine-Grained Spectral Shaping?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

31 views

Abstract

Muon combines current and past gradients into matrix momentum. For , the idealized polar update gives every singular direction the same weight. We refer to this as the flat profile. Several recent optimizers replace this flat profile with fine-grained spectral maps that give each direction its own gain. We ask how much of this spectral detail a Muon update needs. Our spectral diagnostics show that approximately – percent of measured singular modes lie below an estimated noise edge, yet collectively align positively with a reference gradient. We introduce BulkBoost, a two-band spectral reweighting framework with fixed-rank and noise-calibrated variants. The latter uses split-minibatch gradient differences to calibrate a Marchenko–Pastur reference edge for Muon's Nesterov input, separating the bulk below the edge from the spikes above it. Both variants increase the bulk's relative weight through one shared gain while preserving the Frobenius norm of each matrix's unreweighted direction. For a fixed partition, our theory gives the first-order condition under which moving weight toward the bulk lowers the loss. It also quantifies the fraction of the maximal first-order improvement rate, over all per-mode reallocations, that two bands can capture. A polar-plus-low-rank representation enables an approximation without a full SVD, tracking only the spike subspace. Across continued-pretraining settings spanning Pythia-M to M and six corpora, two-band reweighting is competitive with the fine-grained power-law profile of Freon and outperforms Spectra. Measured against Muon's flat profile, Freon reduces final loss by of the pre-adaptation loss on average, whereas the two-band variants achieve reductions of –. These observations suggest that useful departures from the flat profile are surprisingly low-dimensional: a single bulk-to-spike gain captures at least as much benefit as the fine-grained spectral profiles.

Primary area:
optimization
Keywords:
MUON optimizer, spectral optimization, spectral reweighting

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ZgbsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper asks whether Muon needs fine-grained spectral shaping or whether a two-band reweighting is enough. It proposes BulkBoost, which splits the singular directions of Muon’s momentum into a few leading spikes and a large bulk, using either a fixed fraction or a noise edge, and reweights the two bands with one shared gain. In continued pretraining of Pythia models from 14M to 410M parameters on six corpora, it reports small gains over Muon and over the power-law profile of Freon.

Critical strengths

None

Critical weaknesses
  1. The central claim is not supported. Neither BulkBoost nor Freon improves on Muon by more than 0.15% of the initial loss on average, and BulkBoost’s gain of about 0.0025 in loss comes with steps that are 20% to 50% slower, so even where it wins it reaches Muon’s final loss later in wall-clock time. The comparison also favors BulkBoost. It uses Polar Express while Muon uses Newton–Schulz, Muon with Polar Express is missing, and the shared learning rate at 70M is above Muon’s own optimum, which is where most of the gain comes from. At 160M, BulkBoost is worse than Muon on four of six corpora. The key quantity, how much of the benefit two bands can capture, is defined but never measured.
  2. The writing and structure do not meet basic standards. The abstract runs to two paragraphs, although the ICLR template limits it to one. There is no conclusion section, and the conclusion and limitations appear as short paragraphs inside Section 5.1, a subsection on comparing methods. Notation is inconsistent. For example, ∆ is defined as the change in loss during continued pretraining but used for the gap to Muon in Table 2, and q denotes both the number of block iterations and the spike fraction. The main claims are stated as win counts, where a lower three-seed mean at the printed precision of 0.001 counts as a win even when the gap is within the seed standard deviation.
Questions for authors

Does the gain over Muon survive a fair comparison? Please add Muon with Polar Express and tune the learning rate separately for each method, and identify how many of the wins exceed seed noise?

Reviewer_eMeEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether fine-grained spectral shaping is necessary for Muon and proposes BULKBOOST, a simple two-band spectral reweighting method. Specifically, it assigns a baseline weight of 1 to Spike directions and a relative weight of ρ\rho to Bulk directions, followed by Frobenius norm matching.

While the method is straightforward and the paper is clearly written, I have concerns about its novelty, hyperparameter sensitivity, practical effectiveness, and experimental validation.

Critical strengths
  1. Clear presentation. The paper is well organized and easy to follow. The motivation, methodology, and experimental results are clearly presented.

  2. Simple and intuitive idea. The proposed two-band reweighting strategy is straightforward and avoids fine-grained spectral adjustments.

Critical weaknesses

1. Limited novelty.

The idea of assigning different weights to different singular directions has been extensively explored in previous works, including Muon^p [1], Delving into Muon and Beyond [2], and HTMuon [3]. While BULKBOOST adopts a simpler two-band strategy, the underlying idea is quite similar to these existing approaches. Therefore, I find the methodological novelty somewhat limited.

2. Hyperparameter sensitivity and general applicability.

The method relies on ρ\rho and the threshold τ\tau to determine spectral reweighting. It is unclear whether these parameters generalize across different network architectures and model scales or require extensive tuning.

Such flexibility may allow modest improvements under carefully tuned settings, but it raises concerns about the robustness and practical applicability of the method. I would like to see stronger evidence that the method works reliably without architecture-specific hyperparameter tuning.

3. Uncertainty about practical effectiveness.

Several related spectral reweighting methods have been proposed, but such approaches do not always translate into consistent improvements in practical training. Given this context, I remain skeptical about whether BULKBOOST can provide robust benefits beyond the specific experimental settings considered in this paper.

4. Limited experimental scale.

The experiments are conducted on relatively small Pythia models, with the largest containing 410M parameters. I believe experiments on substantially larger models, particularly Mixture-of-Experts (MoE) architectures, would strengthen the paper and provide more convincing evidence of general applicability.

5. Modest improvements over Muon.

According to Table 1, the proposed method does not demonstrate a substantial or consistent advantage over standard Muon in many settings. The improvements in final validation loss are often relatively small. Given the additional spectral reweighting mechanism and hyperparameters, I am not convinced that the reported gains sufficiently justify the added complexity.

Overall, the paper is clearly written and presents a simple approach. However, the idea is closely related to existing spectral reweighting methods, and the empirical improvements over Muon appear relatively modest. The sensitivity to hyperparameter choices and the lack of large-scale experiments further limit my confidence in the method's practical effectiveness and generalizability.

References

[1] Dong, Y., & Sawin, W. (2026). Muon^p: Muon with Fractional Spectral Powers. arXiv:2606.13867.

[2] Qi, X., et al. (2026). Delving into Muon and Beyond: Deep Analysis and Extensions. arXiv:2602.04669.

[3] Pang, T., et al. (2026). HTMuon: Improving Muon via Heavy-Tailed Spectral Correction. Findings of ACL 2026.

Questions for authors

see above

Reviewer_e3rcOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes BulkBoost, a norm-preserving two-band spectral modification of Muon that increases the bulk’s relative weight against leading singular modes. It develops fixed-rank and noise-calibrated partitions, characterizes their local improvement potential, and implements them through a polar-plus-low-rank representation. Experiments cover 30 Pythia continued-pretraining settings.

Critical strengths

The question being studied here is interesting, testing whether coarse spectral allocation matches more elaborate spectral shaping is scientifically valuable, and Frobenius norm matching improves interpretability.

Critical weaknesses

My biggest concern is that a large computational overhead is needed for a relatively small gain. The per step cost is notably more expensive, and the loss gain, e.g. shown in Table 1, is often comparable to variation across seeds. There is also inconsistency in the comparison with baselines. BulkBoost uses Polar Express, but the Muon baseline uses Newton Schulz. I also believe that the algorithmic novelty is limited, compared to prior works such as Spectra or COSMOS.

Questions for authors

Please refer to the weaknesses above. I am in particular concerned about limited novelty and empirical validation, regarding the breadth of methods and the scale of evaluation, e.g. evaluating only on models up to 410M.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.