- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper extends Muon’s functional view of update geometry to softmax modules, using Hilbert distance instead of RMS to constrain changes in output distributions. Since Hilbert distance under softmax equals the variation seminorm of the preactivation perturbation, probability constraints become parameter constraints. For the LM head, a common-ball approximation of the resulting row-diameter constraint recovers RowNorm, which retains at least of the optimal decrease. For attention, it recovers the QK steepest descent problem of Compositional Muon, unifying both methods.
Critical strengths
- A single principle unifies two independently developed methods. One Hilbert-distance constraint on softmax outputs recovers RowNorm for the LM head and the QK steepest descent problem of Compositional Muon, explaining why the two seemingly unrelated designs work.
- Well-justified choice of metric. Hilbert distance matches the shift invariance of softmax and reduces to the variation seminorm of the preactivation, so probability constraints translate directly into parameter constraints. It also simultaneously bounds pairwise odds, total variation, and cross-entropy change.
- Rigorous and sharp analysis. The paper derives the exact dual of the LM-head problem, a sharp Jung-based bound showing that RowNorm retains at least of the optimal decrease, and a proof that row-centering alone leaves Muon unchanged.
Critical weaknesses
- Lack of empirical support for the motivation. The mismatch between RMS and Hilbert distance is illustrated only through a synthetic example (Figure 1). Since standard Muon recipes already optimize the LM head with AdamW, it remains unclear whether this mismatch is a practical concern in actual training.
- Limited novelty. The framework re-derives existing methods, RowNorm and the QK problem of Compositional Muon, rather than producing new ones, and projected RowNorm was already derived from symmetry by Lau & Su (2026). For attention, it only recovers the optimization problem without offering a solution.
- Restrictive assumptions in the attention analysis. The analysis excludes RoPE, QK-norm, the second-order term, and the value pathway, all of which are standard in modern Transformers.
Questions for authors
-
Suboptimality gap in practice. The Jung-based bound is sharp for regular-simplex gradients, but how large is the actual gap between RowNorm and the diameter-constrained optimum for gradients observed in real LM heads?
-
Tied embeddings. The analysis assumes an untied LM head. How would the framework handle tied input and output embeddings, where the same matrix also defines the input representation?
-
Extension to other non-linearities. Could the same principle be applied to modules with other saturating outputs, such as sigmoid attention or top- routing, where the output is not a full softmax distribution?
We thank all three reviewers for their thoughtful feedback. A concern shared across the reviews was the lack of empirical evidence. We have uploaded a revised manuscript that adds an experimental evaluation in language-model pretraining (Appendix A). Below, we summarize our new additions.
The reviewers also raised interesting questions about the design consequences of the framework, training speed, the assumptions of the attention analysis, and the gap to the optimal update in practice. We have addressed them in the individual comments, and we summarize our responses below.
We hope these additions make the empirical support for our framework easier to assess. We would be glad to answer any further questions.