The Illusion of Speed: Uncovering the Trade-Offs of Muon’s Spectral Flattening

Muon has become a promising optimizer for accelerating neural network training by flattening the momentum update spectrum (momentum orthogonalization). Because it often improves model capabilities, the literature has focused on its theoretical understanding and applications while largely overlooking the underlying costs and drawbacks of the acceleration mechanism behind Muon. Our analysis reveals a fundamental tradeoff behind Muon’s speed: this mechanism alters implicit biases that help neural networks generalize in non-synthetic, noisy data settings, degrading performance in real-world environments. In practice, by equalizing the singular values of the update step, momentum orthogonalization assigns equal weight to update directions driven by the "clean" data distribution and those driven by rare, weak, or purely noisy data examples. We develop a theoretical account of this tradeoff, showing that spectral equalization encourages high-rank learning (while gradient-based methods are usually known as low-rank learners), removes the natural separation between signal and noise during training, and can move optimization away from simple solutions. We further identify asymmetric label corruption as a realistic failure mode: corrupted examples create coherent but weak directions that Muon promotes relative to Stochastic Gradient Descent (SGD). Experiments with Vision Transformers on CIFAR-100 support this prediction. Muon matches AdamW on clean data but underperforms SGD under label corruption, with a larger gap under asymmetric than symmetric noise. These findings motivate a new direction for optimizer design that retains the speed of spectral equalization while preserving the robustness of amplitude-sensitive learning.