- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes AF-Muon, an AdamW-free optimizer that uses Muon for hidden matrices, a finite-cap LMO for tied embeddings, and RMS normalization for vector parameters. Experiments across nine settings show improved validation loss and approximately 20% lower optimizer-state memory.
Critical strengths
-
Novel optimization formulation. The paper proposes a principled finite-cap LMO that interpolates between Sign and Row-RMS for tied embeddings.
-
Extensive experiments. The method is evaluated across multiple architectures and modalities, with useful ablation studies and memory measurements.
Critical weaknesses
1. Insufficient theoretical motivation for the proposed geometry. Although tied embeddings receive both sparse input and dense output gradients, it remains unclear why their optimization geometry should be the intersection of row-wise L2 and coordinate-wise infinity-norm constraints. The proposed geometry appears heuristic rather than derived from the functional properties of tied embeddings.
2. Unclear effectiveness of the proposed optimization geometry. Under matched update scales, finite-cap LMO achieves nearly identical performance to Sign and Row-RMS, while the update scale has a much larger impact. This raises doubts about whether the proposed geometry provides meaningful improvements beyond appropriate update normalization and learning-rate tuning.
Questions for authors
-
Why not normalize the update RMS across different parameter groups to achieve consistent update scales, rather than introducing additional learning-rate scaling factors?
-
Could the authors compare AF-Muon with the symmetry-compatible embedding and LM-head optimizers proposed in Symmetry-Compatible Matrix-Gradient Methods: Equivariant Updates, Spectral Operators, and Convergence, both theoretically and empirically [1]?
3 comments
I thank the authors for the detailed rebuttal and the additional experiments.
The clarification on W1 is helpful: I agree that the intersection construction follows once the role norms are chosen, though the role norms and the cap remain modeling choices. On Q2, thank you for pointing to the existing discussion of Lau & Su; the theoretical comparison is clear.
However, my main concern (W2) remains. In the new table, scale-matched Sign already recovers almost all of AF-Muon's improvement over Hybrid Muon (3.7831 vs. 3.7815), and that remaining gap is smaller than the seed standard deviation. This suggests that the gain comes mainly from the update scale, not from the finite-cap geometry, which is presented as the paper's central contribution.
I therefore maintain my original score.
Clarifying the Statistical Difference
Thank you for the careful follow-up. We are glad that the clarification resolves W1 and that the theoretical comparison requested in Q2 is now clear. We address the remaining concern regarding W2 directly.
The relevant evidence extends beyond the single NanoGPT comparison to the cross-setting matched-scale analysis and the requested RowNormM controls. The table below combines the paired analyses underlying revised Tables 38, 40, and 41. Gain is baseline validation loss minus AF-Muon validation loss, so positive values favor AF-Muon. Within every comparison, the tied-update scale is held at , with the same architecture, data, initialization seeds, minibatch order, hidden-matrix optimizer, vector branch, and training budget.
| Source | Matched comparison | Statistical unit | Mean gain | Two-sided paired |
|---|---|---|---|---|
| Table 40 | Matched Sign vs. AF-Muon | 5 settings | ||
| Table 40 | Row-RMS vs. AF-Muon | 5 settings | ||
| Table 41 | Embedding RowNormM vs. AF-Muon | 3 seeds | ||
| Table 41 | Centered LM-head adaptation vs. AF-Muon | 3 seeds | ||
| Table 41 | RMS-matched adaptation vs. AF-Muon | 3 seeds | ||
| Table 38 | NanoGPT: matched Sign vs. AF-Muon | 3 seeds | ||
| Table 38 | NanoGPT: row-RMS vs. AF-Muon | 3 seeds |
For the three requested Table 41 comparisons, the Bonferroni-adjusted threshold is ; both embedding RowNormM and the explicitly RMS-matched adaptation remain significant. Thus, restoring global update RMS does not close the gap.
We agree that the three-seed NanoGPT comparison alone does not distinguish AF-Muon significantly from either endpoint, and we do not present it as doing so. The broader result is that the same small advantage recurs across all five matched settings and is not reproduced by the requested RMS-matched controls.
AF-Muon's contribution does not depend on showing that Sign is universally poor. Sign and row-RMS are principled exact LMOs for their endpoint geometries, but each captures only one side of the tied-table problem. AF-Muon derives the exact interior LMO for , separates directional geometry () from update magnitude (), and completes an AdamW-free optimizer for every parameter role.
The empirical evidence supports both the complete optimizer and its finite-cap geometry. AF-Muon achieves the best mean result over the original baselines across all nine main settings. More importantly for W2, at matched , AF-Muon outperforms both Sign and row-RMS in all five Table 40 settings, with significant paired setting-level differences ( and , respectively). In Table 38, gives the lowest mean among all seven tested caps at fixed scale. In Table 41, AF-Muon is lower than all three requested RowNormM variants in every paired seed; the embedding and explicitly RMS-matched comparisons remain significant after Bonferroni correction. Together, these results confirm that scale provides the dominant improvement, while the finite-cap geometry contributes a systematic effect that cannot be explained by scale or global RMS matching alone.
We hope this complete paired analysis clarifies the finite-cap contribution beyond scale and RMS matching, and respectfully ask whether it resolves the remaining concern regarding W2.
Response to the reviewer 8GGr
We thank the reviewer for recognizing our novelty and experimental breadth. We clarify the geometry, scale, and requested comparison below
Weakness 1: Why the intersection geometry?
We agree that tying alone does not determine the geometry uniquely (§2.2, §4); our goal is to control one shared perturbation’s functional effect on both roles, not gradient sparsity alone.
Each constraint is the induced norm of one role. For lookup, Nin(U)=maxiRMS(ui) bounds every token-embedding displacement; for classification, Nout(U)=maxi,j∣Uij∣ bounds worst-case logit change. Together, maxi∣(Uh)i∣≤min�0�. The cap is tighter for concentrated hidden states, and neither constraint is redundant for 1<c<d (Appendix D.3).
One stored perturbation changes both roles, so admissible updates lie in the intersection of their role balls (Theorem 1, App. C.2). Appendix E gives the exact support-aware LMO; for homogeneous shared roles, the same principle returns ordinary Muon (§2.2, App K.9).
What is derived and what is chosen. Given the role norms, the intersection, logit bound, and exact LMO are derived; the role norms and relative cap c remain modeling choices. They follow SCION’s row-normalized one-hot-input and Sign-output geometries, but we do not claim that tying determines them uniquely or that they are optimal among role-aware alternatives.
Weakness 2 and Question 1: Does geometry help beyond scale normalization?
We agree that scale has the larger empirical effect (App. k.7). Because each branch is already normalized in its role-specific geometry, s is the tied branch’s relative step size rather than an additional normalization correction. Equal entrywise RMS across branches would not equalize their functional effects because they use different operator norms.
The complete matched-scale results are reported in revised Tables 38(42 runs), 40(15 runs), and 41(18 runs) (totally 75 runs). Reducing s from 1 to 0.5 has the larger effect, But at fixed s=0.5, AF-Muon has the lowest mean validation loss in all settings. We therefore recommend tuning s first and then selecting c from a small moderate grid.
Question 2: Comparison with symmetry-compatible RowNormM
Thank you for pointing us to Symmetry-Compatible Matrix-Gradient Methods [1]. Our paper already cites this work as Lau & Su (2026) and discusses it in Appendices A, C.4, D.2, G, and H. We see the two works as complementary: [1] considers separate embedding and LM-head matrices, whereas AF-Muon treats one tensor hard-tied across both roles.
Containment. For c≥d, our oracle reduces to RowNormM up to the constant d and recovers its full left-permutation/right-orthogonal equivariance. The right-spectral and hybrid rules in [1] couple rows and therefore lie outside our row-wise family (Theorem 7).
Where we differ. For 1≤c<d, AF-Muon retains vocabulary-permutation and signed-feature-permutation equivariance, but not arbitrary rotational equivariance (Theorem 6). In return, it guarantees PAPR≤c2 on dense rows (Proposition 7; Appendix G).
Centering. The centered LM-head rule in [1] relies on logit-shift invariance. Under hard tying, the same shift also changes every input embedding, so it is not redundant and centering can remove lookup signal (Propositions 3–4). AF-Muon therefore uses the uncentered merged momentum.
Convergence. Both analyses are deterministic and conditional. Appendix H analyzes exact LMOs in the product norm, and Theorem 8 additionally accounts for momentum, Newton–Schulz, and bisection error; neither analysis covers practical stochastic training.
We implemented the released RowNormM rule under the identical NanoGPT protocol, testing uncentered, centered LM-head, and RMS-matched centered tied-table variants. At step 750 (393.2M tokens; seeds 43–45), AF-Muon is lower than every variant in every seed
We thank the reviewer again for the constructive suggestions. In response, the revised paper now include the complete cap/scale sweep, detailed diagnostics, RowNormM implementations, matched configurations, and per-seed results. We hope these additions address the reviewer’s concerns and would be grateful if the reviewer would reconsider the score.