- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The current submission proposes Budget Muon (BMuon). it's an optimizer that introduces a global allocation budget over MLP hidden channels into Muon's spectral update geometry. Each allocation couples the incoming and outgoing weights of a channel. The authors characterize the resulting norm family and prove that, under a budget-scarcity condition, every minimum-cost factorization of fixed linear predictors has minimal hidden width. A minorization-maximization update makes the allocation practical.
Critical strengths
-
I think the method has a clear motivation and a meaningful structural distinction from existing compression-aware optimizers. The shared budget operates over coupled hidden channels rather than individual matrices or singular directions.
-
The experimental controls support an effect of the training method itself. BMuon improves over Muon when both use the same magnitude-based pruning criterion, and the language experiment includes a shared activation-aware criterion.
Critical weaknesses
-
The advantage over directly training a model from scratch with the final MLP capacity remains unclear, particularly for language modeling. For example, the language experiment uses throughout training, and unpruned perplexity is worse than Muon's: 12.843 versus 10.921. Preserving this quality after 80% MLP pruning establishes compressibility, but does not establish whether the resulting model is preferable to one trained directly at the corresponding size.
-
The comparison with NuMuon is less conclusive than the comparison with standard Muon. In Appendix E.2, NuMuon modifies both attention and MLP matrices, whereas BMuon modifies only MLPs, and only MLPs are compressed at evaluation. This setup is a bit unfair.
Questions for authors
Could the authors compare the language result with a model trained directly using Muon at the corresponding final MLP capacity and the same token budget, including its training time?