MultiNormStep: Leaving the Corner of Steepest Descent

First-order optimizers are the workhorse of large language model training, and after a decade in which AdamW was the $\text{de facto}$ default, Muon has recently become the first optimizer to $\text{consistently}$ beat a well-tuned AdamW across model and data scales. Yet Muon acts only on the 2D matrix weights; the remaining parameters (the normalization scales, biases, and most consequentially, the token embedding and LM head) are left as a compromise to a default AdamW. This split looks like two unrelated optimizers bolted together, but through the lens of steepest descent the pieces are more related than they appear: signSGD and Muon are the $\text{same}$ $\infty$-norm corner read in two gauges, the ${\ell }_{\infty }$ box on coordinates and the Schatten-$\infty$ ball on singular values, and Adam is in turn a $\text{conservative}$ signSGD that shrinks every coordinate strictly inside that box. SignSGD sits aggressively at the corner and Adam timidly within it, yet neither leaves it. Rather than commit to a single corner, we constrain a $\text{weighted family}$ of ${\ell }_p$ norms through a Young function, obtaining $\text{MultiNormStep}$ (MN-step): a family with a closed-form coordinatewise update that both $\text{includes}$ the single-norm methods (signSGD, PowerStep, normalized SGD) and $\text{generates}$ new interior maps, a bounded $\text{sigmoid}$ and an unbounded $\text{log}$. The identical construction lifts from coordinates to the spectrum, where it recovers Muon, so these maps and Muon are two faces of $\text{one}$ steepest-descent principle. Under full-parameter training our bounded sigmoid step stays on par with a carefully tuned AdamW across GPT-2 124M/350M and Qwen3 0.6B/4B; its decisive role is the group Muon leaves to AdamW, where, in a Muon-hybrid that fixes Newton-Schulz Muon on the 2D weights and swaps only that branch, the sigmoid step beats the AdamW branch on validation at $\text{both}$ Qwen3 0.6B and 4B and again outperforms AdamW at two scales of a proprietary mixture-of-experts model (Model-X). These consistent gains show that widening the geometry beyond a single norm genuinely helps, precisely on the parameters Muon leaves to AdamW.