acceptodds
Under review as a conference paper at ICLR 2027

Taming Softmax at the Limits of Muon: A Unified Approach through Hilbert Geometry

Abstract

Muon controls weight updates through the RMS movement of a linear layer's output. This implicitly treats the linear output as a proxy for the full module, including the nonlinearity that follows. This proxy can be misleading for softmax, where the RMS shift in activations does not faithfully measure the change in the output distribution. We propose Hilbert distance as a natural metric for bounding changes in these output distributions. Hilbert distance bounds changes in cross-entropy loss and limits probability redistribution. For softmax, it takes the form of the variation seminorm of the preactivation perturbation, allowing probability constraints to be expressed as parameter constraints. Formulating the steepest descent problem with a Hilbert distance constraint brings two recently proposed methods under a unified mathematical framework. For the LM head, solving a row-wise approximation to the resulting diameter-constrained problem leads us to RowNorm pethick2025trainingdeeplearningmodels. For scaled dot-product attention, the same principle yields the first-order query–key steepest descent problem underlying Compositional Muon keigwin2026compositional. These connections motivate using Hilbert distance as a guiding principle for optimizer design in softmax modules.

Primary area:
optimization
Keywords:
Softmax, Muon, Hilbert distance, probabilities, steepest descent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rGtvOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends Muon’s functional view of update geometry to softmax modules, using Hilbert distance instead of RMS to constrain changes in output distributions. Since Hilbert distance under softmax equals the variation seminorm of the preactivation perturbation, probability constraints become parameter constraints. For the LM head, a common-ball approximation of the resulting row-diameter constraint recovers RowNorm, which retains at least 1/21/\sqrt{2} of the optimal decrease. For attention, it recovers the QK steepest descent problem of Compositional Muon, unifying both methods.

Critical strengths
  • A single principle unifies two independently developed methods. One Hilbert-distance constraint on softmax outputs recovers RowNorm for the LM head and the QK steepest descent problem of Compositional Muon, explaining why the two seemingly unrelated designs work.
  • Well-justified choice of metric. Hilbert distance matches the shift invariance of softmax and reduces to the variation seminorm of the preactivation, so probability constraints translate directly into parameter constraints. It also simultaneously bounds pairwise odds, total variation, and cross-entropy change.
  • Rigorous and sharp analysis. The paper derives the exact dual of the LM-head problem, a sharp Jung-based bound showing that RowNorm retains at least 1/21/\sqrt{2} of the optimal decrease, and a proof that row-centering alone leaves Muon unchanged.
Critical weaknesses
  • Lack of empirical support for the motivation. The mismatch between RMS and Hilbert distance is illustrated only through a synthetic example (Figure 1). Since standard Muon recipes already optimize the LM head with AdamW, it remains unclear whether this mismatch is a practical concern in actual training.
  • Limited novelty. The framework re-derives existing methods, RowNorm and the QK problem of Compositional Muon, rather than producing new ones, and projected RowNorm was already derived from symmetry by Lau & Su (2026). For attention, it only recovers the optimization problem without offering a solution.
  • Restrictive assumptions in the attention analysis. The analysis excludes RoPE, QK-norm, the second-order term, and the value pathway, all of which are standard in modern Transformers.
Questions for authors
  • Suboptimality gap in practice. The Jung-based bound is sharp for regular-simplex gradients, but how large is the actual gap between RowNorm and the diameter-constrained optimum for gradients observed in real LM heads?

  • Tied embeddings. The analysis assumes an untied LM head. How would the framework handle tied input and output embeddings, where the same matrix also defines the input representation?

  • Extension to other non-linearities. Could the same principle be applied to modules with other saturating outputs, such as sigmoid attention or top-kk routing, where the output is not a full softmax distribution?

Reviewer_164EOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed to formulate the steepest descent algorithm using Hilbert distance. This unifies the recent studies in the same framework. For the LM head, a tighter common-ball constraint leads to RowNorm in closed form. For the attention, it recovers the steepest descent problem posed in Compositional Muon.

Critical strengths
  1. Very well written and logic flow is very smooth. Used an example to illustrate the motivation of appropriate perturbation metric for softmax.
  2. Intriguing and informative theoretical analysis. It shows Hilbert distance bounds pairwise odds ratio, and probability redistribution. This address the limitation of Muon’s RMS control which conceals probability redistribution.
Critical weaknesses

Though the work is very interesting, empirical study on the training speed and quality impact would be very useful to provide a holistic picture of the impact of the proposed method.

Questions for authors

Is there any empirical results to verify the effectiveness of the proposed method?

Reviewer_NBE5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper uses Hilbert distance to constrain changes in softmax output distributions. Its variation-seminorm representation induces a row-diameter constraint for an untied LM head and a spectral constraint on the QK product for attention with fixed inputs. A common-ball restriction recovers RowNorm, with a sharp rank-dependent approximation guarantee from Jung's theorem. Linearizing the QK product recovers the existing Compositional Muon optimization problem. The appendix also develops module-level forward and backward perturbation bounds.

Critical strengths
  1. The induced geometries are derived clearly, with the assumptions and distinctions between exact and linearized statements mostly explicit. The discussion of centered perturbations usefully goes beyond the common-shift example, which the authors correctly acknowledge is not itself a practical failure of Muon.

  2. The RowNorm approximation guarantee in Eq. (9) is a concrete theoretical contribution. The exact diameter dual and sharp rank-dependent bound quantify what is lost by replacing the coupled diameter constraint with a tractable common-ball problem.

Critical weaknesses
  1. The main contribution is a new interpretation of existing optimizer constructions, and its additional significance remains limited. RowNorm is already available in Scion, and Eq. (14) is explicitly identified with the existing Compositional Muon problem. The geometric connection and Eq. (9) are useful, but the paper does not yet show a new optimizer-design decision, training guarantee, or testable consequence that follows from choosing this geometry. Section 6 leaves practical utility to future work. The issue is the amount of new insight established by the framework, rather than an expectation that every theoretical paper must include large-scale experiments.

  2. The attention derivation leaves an important gap between the first-order problem and finite probability control. Equation (14) constrains only Delta W_Q W_K^T + W_Q Delta W_K^T. For example, Delta W_Q = a W_Q and Delta W_K = -a W_K make this quantity zero, while the true product change is -a^2 W_Q W_K^T. Thus the linearized constraint alone cannot control a finite Hilbert displacement. The paper explicitly states that the quadratic term is omitted, and Appendix B requires the full product change, so this is a limitation of the resulting update prescription rather than a contradiction of the first-order calculation. A rule for selecting and scaling an update representative would make the proposed stability interpretation substantially more actionable.

Questions for authors
  1. What concrete optimizer-design consequence or prediction does the Hilbert formulation provide beyond recovering RowNorm, the existing QK problem, and the approximation bound in Eq. (9)? A focused analytical example would help establish the additional significance.

  2. How should a representative QK update and step size be selected so that the quadratic product term is controlled and the finite Hilbert-distance bound follows?

  3. In which regime is worst-case relative-odds control preferable to less restrictive probability geometries, given the sensitivity to rare outcomes discussed in Section 6? Can this choice be justified by a small controlled comparison or a specific analytical criterion?

Comments on the paper

1 comment

We thank all three reviewers for their thoughtful feedback. A concern shared across the reviews was the lack of empirical evidence. We have uploaded a revised manuscript that adds an experimental evaluation in language-model pretraining (Appendix A). Below, we summarize our new additions.

  • Experimental setup (Appendix A.1). We train 32-layer Transformers with 190M, 380M, and 640M parameters at 20 and 100 tokens per parameter (TPP). Hyperparameters are tuned on the 190M model and carried over to the larger models without retuning. Table 2 lists the result of every comparison.
  • LM head (Appendix A.2 and A.3). We compare RowNorm against AdamW, tuning AdamW over 240 runs and RowNorm over 60 runs. RowNorm matches the tuned AdamW head at 20 TPP at all three model sizes, and overtakes it at 100 TPP, by 0.0039 at 190M and by 0.0050 at 380M (Figures 2 to 5).
  • Query and key projections (Appendix A.4 and A.5). We compare Compositional Muon against Muon over 60 configurations each. Compositional Muon lowers the validation loss by 0.0101, 0.0090, and 0.0081 at 190M, 380M, and 640M. It is also far less sensitive to hyperparameter choice: 31 of its 60 configurations outperform the best Muon configuration (Figures 6 and 7).
  • 640M model trained on 64B tokens (Appendix A.6). RowNorm and Compositional Muon together improve the validation loss of the default Muon recipe from 2.7708 to 2.7522 (Figure 8).
  • Hilbert distance during training (Appendix A.7). We measure the Hilbert distance by which each LM-head update moves the output distribution. In these runs, it is much smaller for RowNorm than for AdamW at every evaluation, while RowNorm reaches the same or a lower loss (Figure 9).

The reviewers also raised interesting questions about the design consequences of the framework, training speed, the assumptions of the attention analysis, and the gap to the optimal update in practice. We have addressed them in the individual comments, and we summarize our responses below.

  • Design consequences. The framework determines the optimizer for each softmax module: RowNorm for the LM head and Compositional Muon for the query and key projections. It also gives the rule we use to scale the RowNorm learning rate with model width (Appendix A.1).
  • Training speed. In our implementation, RowNorm and Compositional Muon train at the same speed as their baselines, within about 2%.
  • Assumptions of the attention analysis. We omit the second-order term following Compositional Muon (Keigwin et al., 2026). Every run uses RoPE, and Compositional Muon still improves on Muon at every model size. Our query–key experiments and our final run do not use QK-norm, which matches the setting of the analysis.
  • Gap to the optimal update in practice. Equation 9 guarantees that RowNorm retains a constant fraction of the optimal first-order decrease. In training, the measured diameter of the RowNorm step matches its bound, with a median ratio of 0.997 (Appendix A.7).

We hope these additions make the empirical support for our framework easier to assess. We would be glad to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.