acceptodds
Under review as a conference paper at ICLR 2027

BMuon: Budgeting Muon's Spectral Geometry for Structured Compressibility

10 views

Abstract

Muon has emerged as a strong optimizer for neural network training and admits an interpretation as steepest descent under a spectral-norm geometry. Yet we find that Muon-trained Transformers can degrade sharply when their MLP hidden channels are pruned after training. To reconcile Muon's optimization performance with structured compressibility, we introduce Budget Muon (BMuon), which modifies Muon's spectral geometry so that hidden channels across blocks compete for a shared allocation budget. The resulting oracle is exactly normalized steepest descent under a family of norms that interpolate between Muon's geometry and structured sparse geometries. Our principal sparsity control is an interpretable global channel-allocation budget, rather than a setting-dependent sparsity-penalty coefficient. For linear factorization models, we show that Muon admits minimum-cost exact representations with redundant hidden width, whereas under the BMuon oracle geometry and a sufficiently scarce budget, every minimum-cost representation is width-minimal. For practical training, we derive a minorization-maximization surrogate for the exact allocation problem. On ImageNet-1K with ViT-S/16, BMuon improves over Muon under gradual structured MLP pruning, and its learned allocations provide a more effective pruning criterion than weight magnitude. On a 1B-parameter LLaMA-style model trained on FineWeb, BMuon improves over Muon under post-training MLP hidden-channel pruning.

Primary area:
optimization
Keywords:
Muon, steepest descent, spectral norm, structured pruning, structured sparsity, optimization geometry, Transformers

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_45p8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The current submission proposes Budget Muon (BMuon). it's an optimizer that introduces a global allocation budget over MLP hidden channels into Muon's spectral update geometry. Each allocation couples the incoming and outgoing weights of a channel. The authors characterize the resulting norm family and prove that, under a budget-scarcity condition, every minimum-cost factorization of fixed linear predictors has minimal hidden width. A minorization-maximization update makes the allocation practical.

Critical strengths
  1. I think the method has a clear motivation and a meaningful structural distinction from existing compression-aware optimizers. The shared budget operates over coupled hidden channels rather than individual matrices or singular directions.

  2. The experimental controls support an effect of the training method itself. BMuon improves over Muon when both use the same magnitude-based pruning criterion, and the language experiment includes a shared activation-aware criterion.

Critical weaknesses
  1. The advantage over directly training a model from scratch with the final MLP capacity remains unclear, particularly for language modeling. For example, the language experiment uses K≈0.2NK \approx 0.2N throughout training, and unpruned perplexity is worse than Muon's: 12.843 versus 10.921. Preserving this quality after 80% MLP pruning establishes compressibility, but does not establish whether the resulting model is preferable to one trained directly at the corresponding size.

  2. The comparison with NuMuon is less conclusive than the comparison with standard Muon. In Appendix E.2, NuMuon modifies both attention and MLP matrices, whereas BMuon modifies only MLPs, and only MLPs are compressed at evaluation. This setup is a bit unfair.

Questions for authors

Could the authors compare the language result with a model trained directly using Muon at the corresponding final MLP capacity and the same token budget, including its training time?

Reviewer_jWrhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Budget Muon, an optimizer designed to make Transformer MLPs easier to compress through hidden-channel pruning. It modifies Muon’s updates by assigning channels a shared global budget, allowing different channels and layers to receive different allocations. The paper characterizes the resulting optimization geometry and shows that its representation cost favors minimal hidden width in linear factorization models under a sufficiently small budget. A practical approximation makes the allocation update tractable. Experiments on CIFAR-100, ImageNet-1K, and a 1B-parameter language model show improved pruning robustness compared with Muon.

Critical strengths

1. Modifying Muon’s update geometry to encourage structured compressibility is well motivated. The shared allocation couples the weights belonging to each hidden channel, and the theoretical analysis provides a useful connection between optimization geometry and representation width.

2. The experiments show substantial improvements in pruning robustness on both vision and language models. BMuon outperforms Muon even when both use the same magnitude-based pruning criterion, which supports a benefit from the training method itself. Using the learned allocations for pruning further improves performance, suggesting that they also provide useful information about channel importance.

Critical weaknesses

1. The contribution of adaptive allocation is not sufficiently isolated. Reducing the budget also changes the update magnitude, but the experiments lack a uniform-allocation control under the same budget schedule. The interaction between the budget and reweighting exponent α\alpha is also not empirically investigated. It is therefore unclear how much of the improvement comes specifically from learning which channels to prioritize.

2. The practical advantage at language-model scale remains unclear. BMuon improves pruning robustness but worsens unpruned perplexity from 10.92 to 12.84, without comparison to a model trained directly at the final reduced widths. Moreover, Muon+GL and NuMuon are evaluated only on CIFAR-100. These missing comparisons make it difficult to establish whether BMuon offers a better quality–compute tradeoff than simpler alternatives.

3. The presentation could be improved. Some passages are dense with notation, and the writing would benefit from clearer explanations of the intuition behind the mathematical formulations.

Questions for authors

1. Can you compare adaptive allocation with a uniform allocation $\mu_j=K/N$ under the same budget schedule? How much of the compressibility improvement comes from channel-specific allocation rather than the change in update magnitude?

2. Given the higher unpruned perplexity of BMuon, how does the pruned language model compare with a model trained from scratch using the same final per-layer widths? Does BMuon retain an advantage under a matched training-compute budget?

3. Muon+GL and NuMuon are included on CIFAR-100 but missing from the larger-scale experiments. Could you include these comparisons on ImageNet or the language model under comparable compression budgets?

4. How do the budget ratio K/NK/N and reweighting exponent α\alpha affect the tradeoff between unpruned quality and pruning robustness? Does using a smaller budget with stronger reweighting suppress useful channels too early, making them difficult to recover later?

Reviewer_a4FPOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Budget Muon (BMuon), a modification of Muon that encourages structured compressibility by making MLP hidden channels across layers compete for a shared update budget. This budget reweights Muon’s spectral-norm update geometry, coupling the input and output weights of each channel without adding forward-pass gates. The authors characterize the resulting norm family and prove that, in linear factorization models under a sufficiently restrictive budget, minimum-cost representations use the minimum possible number of active channels. They derive a practical minorization–maximization approximation and evaluate it on CIFAR-100, ImageNet-1K, and a roughly 1B-parameter language model trained on FineWeb. BMuon improves robustness to structured MLP pruning, while its learned allocations provide useful pruning scores and identify nonuniform layer widths; however, these compression benefits can come at the expense of unpruned model quality, particularly in the language-model experiment.

Critical strengths
  • The paper provides a principled formulation of compression-aware optimization. BMuon incorporates a global channel-allocation budget into Muon’s spectral update geometry, coupling the matrix slices that constitute each prunable MLP channel and allowing capacity allocation across layers without introducing forward-pass gates. The resulting norm family connects Muon to structured sparse geometries, and the representation-cost analysis establishes conditions under which minimum-cost linear factorizations are width-minimal. This provides a concrete theoretical connection between optimization geometry and structured compressibility, even though it does not establish the implicit bias of practical Transformer training.
Critical weaknesses

The experimental evaluation does not sufficiently establish BMuon’s practical benefits. On ImageNet, the approximately 22M-parameter ViT-S baseline reaches only 75.23% accuracy, whereas the similarly sized, non-distilled DeiT-S achieves 79.8% [1]. Although the training recipes and budgets differ, this gap leaves open whether BMuon’s pruning robustness persists under stronger training recipes and at more competitive accuracy levels.

For language modeling, the evaluation is limited to a single approximately 1B-parameter model trained at roughly 13 tokens per parameter, below the commonly cited Chinchilla guideline of approximately 20 [2]. This alone does not establish undertraining, but it leaves longer training regimes unexplored—particularly those relevant to deployment, where models such as the smaller Llama 3 variants are deliberately trained beyond the compute-optimal budget to improve quality at a fixed inference cost [3]. BMuon already incurs a notable quality gap relative to Muon in the reported setting (PPL 12.84 vs. 10.92), and it remains unclear how this gap evolves with additional training. A more convincing evaluation would compare both optimizers across multiple model sizes and token budgets, including smaller models trained for substantially longer, and report quality–compute trade-offs before and after pruning. Smaller models trained directly with Muon under matched training compute and final inference cost would also provide an essential baseline.

[1] Touvron et al. Training Data-Efficient Image Transformers & Distillation through Attention. ICML 2021, Table 5.

[2] Hoffmann et al. Training Compute-Optimal Large Language Models. NeurIPS 2022.

[3] Dubey et al. The Llama 3 Herd of Models. 2024, Section 1.

Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.