acceptodds
Under review as a conference paper at ICLR 2027

AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models

Abstract

Muon improves large-scale training by applying a spectral-norm steepest-descent update to matrix parameters, but practical models also contain parameter blocks that do not fit dense-matrix geometry. One important case is the tied vocabulary table, which appears in language models and other token generators and can receive multiple structurally different gradient sources, from sparse input lookups to dense output-classifier updates. In the reference recipe these blocks are handed to an auxiliary AdamW optimizer, which restores second-moment state and updates the aliased table as a generic tensor. We propose AF-Muon, an AdamW-free extension of Muon that keeps the Muon matrix update for hidden weight matrices while using a support-aware finite-cap linear minimization oracle for tied vocabulary tables and an RMS-normalized update for one-dimensional auxiliary parameters. AF-Muon therefore trains every parameter class with a single first-moment buffer and no second-moment state, saving around 20% optimizer-state memory relative to Hybrid Muon in our benchmark. Across nine tied-token settings—decoder-only language models from 124M to 1B parameters, a fully shared T5-style encoder–decoder, and ImageGPT-style image-token, protein, and sparse-MoE variants, spanning text, image, and protein-sequence data—AF-Muon improves mean validation loss and perplexity over both Hybrid Muon and a SCION-style Sign endpoint. Long-horizon runs and hyperparameter sensitivity studies confirm the gain is robust, and identical-momentum diagnostics attribute it to the finite cap, which preserves more within-row magnitude than Sign while bounding the coordinate concentration of row-RMS. These results identify tied vocabulary tables as a distinct optimizer geometry and yield a robust AdamW-free Muon variant across models, modalities, and architectures, with about 1% step-time overhead in matched training.

Primary area:
optimization
Keywords:
AdamW-free optimization, Muon, tied embeddings, optimizer geometry, linear minimization oracles

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8GGrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AF-Muon, an AdamW-free optimizer that uses Muon for hidden matrices, a finite-cap LMO for tied embeddings, and RMS normalization for vector parameters. Experiments across nine settings show improved validation loss and approximately 20% lower optimizer-state memory.

Critical strengths
  1. Novel optimization formulation. The paper proposes a principled finite-cap LMO that interpolates between Sign and Row-RMS for tied embeddings.

  2. Extensive experiments. The method is evaluated across multiple architectures and modalities, with useful ablation studies and memory measurements.

Critical weaknesses

1. Insufficient theoretical motivation for the proposed geometry. Although tied embeddings receive both sparse input and dense output gradients, it remains unclear why their optimization geometry should be the intersection of row-wise L2 and coordinate-wise infinity-norm constraints. The proposed geometry appears heuristic rather than derived from the functional properties of tied embeddings.

2. Unclear effectiveness of the proposed optimization geometry. Under matched update scales, finite-cap LMO achieves nearly identical performance to Sign and Row-RMS, while the update scale has a much larger impact. This raises doubts about whether the proposed geometry provides meaningful improvements beyond appropriate update normalization and learning-rate tuning.

Questions for authors
  1. Why not normalize the update RMS across different parameter groups to achieve consistent update scales, rather than introducing additional learning-rate scaling factors?

  2. Could the authors compare AF-Muon with the symmetry-compatible embedding and LM-head optimizers proposed in Symmetry-Compatible Matrix-Gradient Methods: Equivariant Updates, Spectral Operators, and Convergence, both theoretically and empirically [1]?

[1] https://arxiv.org/abs/2605.18106

3 comments
AuthorsOpenReview ↗

Response to the reviewer 8GGr

We thank the reviewer for recognizing our novelty and experimental breadth. We clarify the geometry, scale, and requested comparison below

Weakness 1: Why the intersection geometry?

We agree that tying alone does not determine the geometry uniquely (§2.2, §4); our goal is to control one shared perturbation’s functional effect on both roles, not gradient sparsity alone.

Each constraint is the induced norm of one role. For lookup, Nin(U)=max⁡iRMS⁡(ui)N_{\rm in}(U)=\max_i\operatorname{RMS}(u_i) bounds every token-embedding displacement; for classification, Nout(U)=max⁡i,j∣Uij∣N_{\rm out}(U)=\max_{i,j}\lvert U_{ij}\rvert bounds worst-case logit change. Together, max⁡i∣(Uh)i∣≤min⁡�0�\max_i\lvert(Uh)_i\rvert\leq\min�0�. The cap is tighter for concentrated hidden states, and neither constraint is redundant for 1<c<d1<c<\sqrt{d} (Appendix D.3).

One stored perturbation changes both roles, so admissible updates lie in the intersection of their role balls (Theorem 1, App. C.2). Appendix E gives the exact support-aware LMO; for homogeneous shared roles, the same principle returns ordinary Muon (§2.2, App K.9).

What is derived and what is chosen. Given the role norms, the intersection, logit bound, and exact LMO are derived; the role norms and relative cap cc remain modeling choices. They follow SCION’s row-normalized one-hot-input and Sign-output geometries, but we do not claim that tying determines them uniquely or that they are optimal among role-aware alternatives.

Weakness 2 and Question 1: Does geometry help beyond scale normalization?

We agree that scale has the larger empirical effect (App. k.7). Because each branch is already normalized in its role-specific geometry, ss is the tied branch’s relative step size rather than an additional normalization correction. Equal entrywise RMS across branches would not equalize their functional effects because they use different operator norms.

The complete matched-scale results are reported in revised Tables 38(42 runs), 40(15 runs), and 41(18 runs) (totally 75 runs). Reducing ss from 11 to 0.50.5 has the larger effect, But at fixed s=0.5s=0.5, AF-Muon has the lowest mean validation loss in all settings. We therefore recommend tuning ss first and then selecting cc from a small moderate grid.

Question 2: Comparison with symmetry-compatible RowNormM

Thank you for pointing us to Symmetry-Compatible Matrix-Gradient Methods [1]. Our paper already cites this work as Lau & Su (2026) and discusses it in Appendices A, C.4, D.2, G, and H. We see the two works as complementary: [1] considers separate embedding and LM-head matrices, whereas AF-Muon treats one tensor hard-tied across both roles.

Containment. For c≥dc\ge\sqrt d, our oracle reduces to RowNormM up to the constant d\sqrt d and recovers its full left-permutation/right-orthogonal equivariance. The right-spectral and hybrid rules in [1] couple rows and therefore lie outside our row-wise family (Theorem 7).

Where we differ. For 1≤c<d1\le c<\sqrt d, AF-Muon retains vocabulary-permutation and signed-feature-permutation equivariance, but not arbitrary rotational equivariance (Theorem 6). In return, it guarantees PAPR≤c2\mathrm{PAPR}\le c^2 on dense rows (Proposition 7; Appendix G).

Centering. The centered LM-head rule in [1] relies on logit-shift invariance. Under hard tying, the same shift also changes every input embedding, so it is not redundant and centering can remove lookup signal (Propositions 3–4). AF-Muon therefore uses the uncentered merged momentum.

Convergence. Both analyses are deterministic and conditional. Appendix H analyzes exact LMOs in the product norm, and Theorem 8 additionally accounts for momentum, Newton–Schulz, and bisection error; neither analysis covers practical stochastic training.

We implemented the released RowNormM rule under the identical NanoGPT protocol, testing uncentered, centered LM-head, and RMS-matched centered tied-table variants. At step 750 (393.2M tokens; seeds 43–45), AF-Muon is lower than every variant in every seed

Method / tied-table ruleCap ccTied-update scalingVal. loss ↓
Hybrid Muon--AdamW, LR 3×10−43\times10^{-4}3.87411±.00079
Scale-matched Sign1AF-Muon scale s=0.5s=0.53.7831±.0033
AF-Muon3s=0.5s=0.53.7815±.0026
Embedding RowNormM--LR matched to AF-Muon s=0.5s=0.53.78572±.00175
LM-head RowNormM adaptation--Same LR conversion after centering3.79096±.00473
RMS-matched LM-head adaptation--Final RMS matched to s=0.5s=0.5 target3.79595±.00100

We thank the reviewer again for the constructive suggestions. In response, the revised paper now include the complete cap/scale sweep, detailed diagnostics, RowNormM implementations, matched configurations, and per-seed results. We hope these additions address the reviewer’s concerns and would be grateful if the reviewer would reconsider the score.

I thank the authors for the detailed rebuttal and the additional experiments.

The clarification on W1 is helpful: I agree that the intersection construction follows once the role norms are chosen, though the role norms and the cap remain modeling choices. On Q2, thank you for pointing to the existing discussion of Lau & Su; the theoretical comparison is clear.

However, my main concern (W2) remains. In the new table, scale-matched Sign already recovers almost all of AF-Muon's improvement over Hybrid Muon (3.7831 vs. 3.7815), and that remaining gap is smaller than the seed standard deviation. This suggests that the gain comes mainly from the update scale, not from the finite-cap geometry, which is presented as the paper's central contribution.

I therefore maintain my original score.

AuthorsOpenReview ↗

Clarifying the Statistical Difference

Thank you for the careful follow-up. We are glad that the clarification resolves W1 and that the theoretical comparison requested in Q2 is now clear. We address the remaining concern regarding W2 directly.

The relevant evidence extends beyond the single NanoGPT comparison to the cross-setting matched-scale analysis and the requested RowNormM controls. The table below combines the paired analyses underlying revised Tables 38, 40, and 41. Gain is baseline validation loss minus AF-Muon validation loss, so positive values favor AF-Muon. Within every comparison, the tied-update scale is held at s=0.5s=0.5, with the same architecture, data, initialization seeds, minibatch order, hidden-matrix optimizer, vector branch, and training budget.

SourceMatched comparisonStatistical unitMean gainTwo-sided paired pp
Table 40Matched Sign vs. AF-Muon5 settings0.0016440.0016440.01130.0113
Table 40Row-RMS vs. AF-Muon5 settings0.0037590.0037590.03400.0340
Table 41Embedding RowNormM vs. AF-Muon3 seeds0.0042060.0042060.01420.0142
Table 41Centered LM-head adaptation vs. AF-Muon3 seeds0.0094420.0094420.15480.1548
Table 41RMS-matched adaptation vs. AF-Muon3 seeds0.0144360.0144360.008580.00858
Table 38NanoGPT: matched Sign vs. AF-Muon3 seeds0.0016320.0016320.42700.4270
Table 38NanoGPT: row-RMS vs. AF-Muon3 seeds0.0029130.0029130.28540.2854

For the three requested Table 41 comparisons, the Bonferroni-adjusted threshold is 0.05/3=0.01670.05/3=0.0167; both embedding RowNormM and the explicitly RMS-matched adaptation remain significant. Thus, restoring global update RMS does not close the gap.

We agree that the three-seed NanoGPT comparison alone does not distinguish AF-Muon significantly from either endpoint, and we do not present it as doing so. The broader result is that the same small advantage recurs across all five matched settings and is not reproduced by the requested RMS-matched controls.

AF-Muon's contribution does not depend on showing that Sign is universally poor. Sign and row-RMS are principled exact LMOs for their endpoint geometries, but each captures only one side of the tied-table problem. AF-Muon derives the exact interior LMO for 1<c<d1<c<\sqrt{d}, separates directional geometry (cc) from update magnitude (ss), and completes an AdamW-free optimizer for every parameter role.

The empirical evidence supports both the complete optimizer and its finite-cap geometry. AF-Muon achieves the best mean result over the original baselines across all nine main settings. More importantly for W2, at matched s=0.5s=0.5, AF-Muon outperforms both Sign and row-RMS in all five Table 40 settings, with significant paired setting-level differences (p=0.0113p=0.0113 and p=0.0340p=0.0340, respectively). In Table 38, c=3c=3 gives the lowest mean among all seven tested caps at fixed scale. In Table 41, AF-Muon is lower than all three requested RowNormM variants in every paired seed; the embedding and explicitly RMS-matched comparisons remain significant after Bonferroni correction. Together, these results confirm that scale provides the dominant improvement, while the finite-cap geometry contributes a systematic effect that cannot be explained by scale or global RMS matching alone.

We hope this complete paired analysis clarifies the finite-cap contribution beyond scale and RMS matching, and respectfully ask whether it resolves the remaining concern regarding W2.

Reviewer_A4BSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks an interesting and timely question on how to make the hybrid-Muon training where the model is trained with a mix of AdamW and Muon into AdamW free version where the entire model is trained without second moment. They introduce AF-Muon (AdamW free Muon) where hidden matrices are still trained with Muon but tied token parameters are trained via their proposed method support-aware finite cap LMO saving memory by 20% when compared against hybrid-Muon. This method is shown to work for tied-token settings. The authors do a broad set of experiments on nine tied token settings including multiple model architectures and from model sizes ranging from 124M to 1B models over three seeds each.

Critical strengths

I like the idea of making the entire model train by requiring single first-moment buffer and not requiring AdamW at all. Even though SCION works in a similar line, the empirical evidence is much broader in this current study in terms of coverage over various model architectures and over three seeds to see the gains are beyond noise. Taking hybrid-Muon and SCION-style Sign methods as baseline, the proposed method works consistently gains across all setups.

Critical weaknesses

The main weaknesses I see is in evaluation and clarity

  • The evaluation is restricted to loss and perplexity, do you see if the gain hold on downstream tasks too. To see this, you can take 1B model and measure downstream performance on some standard evaluation tasks. Measure baselines and proposed method. This will make the claim more reliable.
  • The models specially 0.5B and greater than than are trained on far less than chinchilla token budget. The larger training horizon run is on Lla,a-3.2-1B and on 9.98B token budget but this is under chinchilla token budget by a larger margin. So the conclusion drawn on gains of the proposed method at these scale may be limited.
  • The method gains is restricted to tied-token settings. For the current study it is scoped well but this limits the scope of the improved optimizer as compared against baselines.

Clarity

  • The validation perplexity graphs in the main section is not adding much to the results, this can be removed and rather add the related work section in the main section of the paper.
  • Its harder to follow the experimental settings for each setup. I suggest to add tables with model size, token budgets, hyperparameters used in the main section for more clarity.
  • Line 084 - start new para
Questions for authors

None

3 comments
AuthorsOpenReview ↗

Response to reviewer A4BS

We thank the reviewer for recognizing the breadth and value of our experiments. We agree that downstream evaluation, training horizon, scope, and presentation needed clearer discussion.

Additional FP32 downstream evaluation

We evaluated the matched seed-43 Llama-3.2-1B checkpoints on four standard tasks using a pinned lm-evaluation-harness checkout. All use the complete labeled split, zero-shot likelihood scoring, FP32 inference, identical prompts and tokenization, and no downstream or method-specific tuning.

Task and metricAreaHybrid MuonSCION-style SignAF-Muon
LAMBADA accuracy ↑Contextual language modeling32.7833.7135.80
LAMBADA perplexity ↓35.7235.8633.18
SST-2 accuracy ↑Sentiment classification49.5452.9862.04
SST-2 balanced accuracy ↑50.4553.8162.38
SciQ accuracy ↑Scientific QA73.2073.8073.40
SciQ length-normalized accuracy ↑63.2065.9064.30
COPA accuracy ↑Causal commonsense reasoning69.0070.0072.00

On 5,153 LAMBADA examples, AF-Muon improves final-word accuracy by 3.03 points over Hybrid Muon and 2.10 over Sign, while reducing perplexity by 2.54 and 2.68. On 872 SST-2 examples, it improves accuracy by 12.50 and 9.06 points. AF-Muon also has the best balanced accuracy, although this partly reflects improved zero-shot label calibration. On SciQ, AF-Muon lies between the baselines. On the smaller 100-example COPA split, AF-Muon obtains 72%, versus 70% for Sign and 69% for Hybrid Muon.

These results provide direct evidence that the validation-loss gain transfers to contextual prediction and sentiment classification, with a smaller positive result on causal reasoning. SciQ also makes clear that downstream ordering is task-dependent rather than uniformly favorable. Only seed-43 Llama checkpoints were retained, so these downstream results do not estimate training-seed variability and do not replace our three-seed validation-loss comparisons. We added the complete protocol, prediction audits, results, and this limitation to the revised appendix J.4.2. page 42.

Training-token budget

We agree that the 1B-parameter experiment at 9.98B tokens, or ~10 tokens per parameter, is below the commonly cited Chinchilla regime. Our claim concerns optimizer efficiency under a matched resource budget, not full-horizon convergence. Such comparisons are standard in scaling-law analyses of available compute and compute-optimal allocation (Kaplan et al., 2020; Hoffmann et al., 2022).

All compared methods use the same architecture, initialization protocol, data stream, batch size, token budget, and number of updates. Under this controlled budget, AF-Muon achieves lower validation loss than Hybrid Muon while using approximately 20% less optimizer-state memory and about 1% measured step-time overhead. Moreover, two smaller-model experiments are near the commonly used 20-token-per-parameter heuristic:

  • NanoGPT: 2.7B tokens for 124M parameters, approximately 21.8 tokens per parameter.
  • SmolLM2-135M: 2.5B tokens, approximately 18.5 tokens per parameter.

AF-Muon's advantage persists in both settings, so the ordering is not restricted to the most undertrained models. We nevertheless now state explicitly that the 1B experiment does not establish full-horizon convergence.

Scope of tied-token settings

We agree that AF-Muon's new oracle specifically addresses physically tied token tables and is not claimed as a general replacement for optimizers of arbitrary untied parameters. Within this deliberate scope, the evidence covers nine settings across decoder-only language models, a fully shared encoder-decoder, sparse MoE, ImageGPT-style image modeling, and protein language modeling. Thus, the method is specialized, but the tied-table geometry transfers across architectures and modalities. We now make this scope and limitation explicit.

Clarity revisions

We added a compact main-text table (Table 1) summarizing architecture, domain, model size, training budget, and seeds across all nine settings and the two long-horizon continuations; Appendix J and the supplementary README.md retain full hyperparameters. We retain the validation-perplexity curves because they show more than final values: when gaps emerge and whether they persist or close during training. We clarified this purpose, strengthened the main-text related-work discussion, and began a new paragraph at the former line 84.

We appreciate the reviewer's thoughtful suggestions. We hope the new downstream evidence, explicit training-budget limitation, clarified scope, and presentation revisions address the concerns and invite reconsideration of the score.

Thank you for adding downstream results and revising the scope and limitations of the method. Compared to baselines, the proposed method is limited to a tied-token setting and has been shown to work on a small scale. Multiple setting evauation is a strength, but many of the models are undertrained. The scope of the method is limited, and therefore I retain my score.

AuthorsOpenReview ↗

Clarification on Scope and Training Scale

Thank you for reviewing our revisions and clarifying your remaining concerns. We appreciate your time and the opportunity to address the scope and training scale directly.

Scope: We address the scope concern directly. Weight tying is used in several modern compact model families, including the Qwen2.5, Llama-3.2, SmolLM2, and T5Gemma variants evaluated here. By sharing one vocabulary matrix between lookup and prediction, tying reduces model parameters [4]. In our representative decoder-only settings, this table accounts for approximately 21--31% of all parameters and is often the largest single tensor. Because Hybrid Muon leaves it on AdamW with both first- and second-moment states, it is a major source of the remaining optimizer-state memory. AF-Muon's measured approximately 20% reduction therefore targets a central parameter block, not a peripheral group.

AF-Muon retains the identical Muon update for hidden matrices and replaces only the auxiliary AdamW treatment of the tied table and one-dimensional parameters with first-moment, role-specific updates. It therefore extends Muon's AdamW-free coverage rather than restricting its application. Recent work continues to study the gradient imbalance, optimization geometry, and parameter-efficiency trade-offs created by weight tying [1,2,3], but, to our knowledge, no prior method provides an AdamW-free optimizer for the merged lookup/classifier gradient. AF-Muon fills this gap and is validated across nine tied-token settings spanning decoder-only, encoder–decoder, dense, MoE, text, image, and protein models.

[1] Weight Tying Biases Token Embeddings Towards the Output Space, ACL Findings 2026.
[2] Attention with Markov: A Curious Case of Single-Layer Transformers, ICLR 2025.
[3] SLlama: Parameter-Efficient Language Model Architecture for Enhanced Linguistic Competence Under Strict Data Constraints, EMNLP 2025.
[4] Using the Output Embedding to Improve Language Models, EACL 2017.

Training Scale: We agree that the ~125M--1B-parameter runs at ~2.7B--10B tokens are below the Chinchilla compute-optimal estimate for some models such as Llama, and we do not claim compute-optimal convergence. However, controlled experiments at comparable scales are common in accepted optimizer research: Sophia [5] reports experiments on models from 125M up to 1.5B parameters under matched steps and compute; GaLore [6] studies memory-efficient pretraining on 1B and 7B models with up to 19.7B tokens; Beyond Outliers [7] compares AdamW, Muon, SCION, Shampoo, SOAP, and PSGD on 50M--1.5B models at the Chinchilla-optimal token ratio; and SCALE [8] evaluates memory-efficient optimization on 60M--1B and 7B models. Moreover, our NanoGPT run uses 2.67B tokens for 124M parameters (21.6 tokens per parameter), and SmolLM2 uses 2.5B tokens for 135M parameters (18.5 tokens per parameter), both near the approximately 20-token-per-parameter Chinchilla regime, with the same optimizer ordering. These precedents do not replace longer training; they support our claim about optimizer efficiency under identical, controlled budgets.

[5] Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training, ICLR 2024.
[6] GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection, ICML 2024.
[7] Beyond Outliers: A Study of Optimizers Under Quantization, ICLR 2026.
[8] Memory-Efficient LLM Pretraining via Minimalist Optimizer Design, ICML 2026.

Our conclusion is therefore precisely scoped and directly supported by the experiments: under identical architecture, data order, initialization seeds, batch size, update count, and token budget, AF-Muon improves validation loss across diverse tied-token settings while using less optimizer-state memory. The present evidence establishes matched-budget optimizer efficiency for tied-token models; compute-optimal convergence and untied architectures remain outside this study.

We hope the requested downstream evaluations and the literature-grounded evidence on scope and training scale address your concerns. Please let us know if any questions remain; we would be happy to clarify them during the remaining discussion period.

Reviewer_SdjHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work prsents AF-Muon, an extension to Muon to train tied-embedding models that eliminate the need of AdamW completely. The main idea is to extend SCION's extreme sign based normalized score update rule to a smoother update rule that behaves proportionally to the signal (the so called capped row-normalized update) for the shared vocabulary table. Experimental study shows that the new update rule is effective with improved mean performance.

Critical strengths
  1. The motivation of the work is clearly stated.

  2. Evaluation looks comprehensive and substantial.

Critical weaknesses
  1. The work doesn't say clearly how one shall determine the geometric cap c that defines the normalized shape of the update. Experiments used a tuned config with c = 3.

  2. The performance of the updating rule is affected by both the cap c and the scale s. It would be better if we can separate the contributions of c and s.

  3. The work picked (c, s) of AF-Muon via extensive hyperparameter tuning, while Hybrid Muon used a fixed auxiliary learning rate of 3 x 10^{-4}; it is not sure whether tuning Hybrid Muon's setup would close the gap given that the gap is not that big.

  4. The paper is not very well written, with acronyms and cryptic terms cluttered especially in the introduction (without later sections, it's not really possible to digest Fig 1 accurately). It is a bit hard to read and thus to appreciate the contribution for readers not working on the same problem.

Questions for authors

Q1. How should one decide the cap c and scale s when using AF-Muon?

Q2. How was Hybrid Muon's config determined?

Q3. Which contributes more to AF-Muon's performance, c or s?

1 comment
AuthorsOpenReview ↗

Response to reviewer SdjH

We thank the reviewer for the positive assessment and constructive feedback on presentation and tuning. We address each concern below.

Regarding the fourth weakness, we added one paragraph to the introduction to make the problem and Figure 1 clearer:

Figure 1 turns this mixed-role problem into an optimizer design. Panel 1 shows how the lookup and classifier gradients merge on the same tied table. Panel 2 shows AF-Muon's role-specific assignment: Muon for hidden matrices, a finite-cap linear minimization oracle (LMO) for the tied table, and RMS-normalized updates for one-dimensional parameters. Panel 3 illustrates how the cap cc changes the normalized tied-table geometry: c=1c=1 gives the Sign endpoint, intermediate values retain within-row magnitude information while limiting coordinate concentration, and c≥dc\geq\sqrt{d} gives the row-RMS endpoint. The separate multiplier ss changes the overall update magnitude without changing this geometry.

Q1 and Q3: How should cc and ss be selected, and which matters more?

The two parameters have distinct roles. The scale ss controls the overall tied-update magnitude, whereas cc controls coordinate concentration after row normalization. We recommend tuning ss first and then choosing cc from a small moderate range such as {2,3,4}\{2,3,4\}. When no tuning budget is available, we use (c,s)=(3,0.5)(c,s)=(3,0.5). This setting was selected once and transferred unchanged across all original model families, modalities, and architectures, showing that it is a robust default rather than a value retuned for every experiment.

To make this point clear, we expanded Appendix K.7 with a three-seed sweep over c∈{1,2,3,4,6,10,∞}c\in\{1,2,3,4,6,10,\infty\} and s∈{0.5,1}s\in\{0.5,1\}: 14 settings and 42 runs (Table 38-39). We use the same model, data, initialization seeds, minibatch order, optimizer branches, and training budget across this sweep. We also added a matched-scale comparison across five different architectures, again using three seeds and identical configurations and budgets within each row (Table 40-41) totally 75 runs and in all of them AF-Muon has better performance. Table 41 summary is shown below:

Method / tied-table ruleCap ccTied-update scalingVal. loss ↓
Hybrid Muon--AdamW, LR 3×10−43\times10^{-4}3.87411±.00079
Scale-matched Sign1AF-Muon scale s=0.5s=0.53.7831±.0033
AF-Muon3s=0.5s=0.53.7815±.0026
Embedding RowNormM--LR matched to AF-Muon s=0.5s=0.53.78572±.00175
LM-head RowNormM adaptation--Same LR conversion after centering3.79096±.00473
RMS-matched LM-head adaptation--Final RMS matched to s=0.5s=0.5 target3.79595±.00100

These results separate the contributions of scale and geometry. Reducing ss provides the largest gain, but after matching this scale, AF-Muon with c=3c=3 still achieves the lowest mean validation loss below scale-matched Sign and below the strongest RowNormM variant. AF-Muon is also lower than every tested RowNormM variant in every paired seed. Thus, ss sets the overall update magnitude, while finite cc provides an additional concentration-control benefit; (c,s)=(3,0.5)(c,s)=(3,0.5) is the best tested combined configuration and we used it as default for all our experiments.

Q2: How was Hybrid Muon configured?

Full model, data, and hyperparameter settings, including those for Hybrid Muon, are provided in Appendix J and the supplementary README.md. In summary, Hybrid Muon uses Muon for hidden matrices and AdamW for the tied table and one-dimensional parameters. All methods use matrix LR 0.020.02; Hybrid Muon uses auxiliary AdamW LR 3×10−43\times10^{-4}.

Beyond this fixed comparison, Section 3.4 compares AF-Muon and Hybrid Muon across changes in batch size, auxiliary learning rate, vocabulary size, and hidden width. AF-Muon is more robust across these settings; the separate scale-sensitivity results examine the effect of ss.

To test whether this fixed auxiliary learning rate disadvantaged Hybrid Muon, we added a seed-43 sensitivity sweep in Appendix K.6 from 7.5×10−57.5\times10^{-5} to 9.6×10−39.6\times10^{-3}. Increasing the auxiliary learning rate substantially narrows the gap. At Hybrid Muon's best tested value, 2.4×10−32.4\times10^{-3}, its validation loss is 3.72333.7233, compared with 3.72083.7208 for AF-Muon at the corresponding setting. Thus, tuning Hybrid Muon nearly closes the loss gap in this diagnostic, but does not reverse the best observed ordering. We now state this limitation directly in the revised paper.

We appreciate the reviewer's time and thoughtful feedback. We have uploaded a revised manuscript containing the clearer introduction, the complete cap-and-scale analysis, the cross-model matched-scale results, and the expanded Hybrid Muon sensitivity experiment. We hope these changes resolve the concerns and kindly ask the reviewer to reconsider the score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We thank the reviewers for their constructive feedback. We have uploaded a revised manuscript with the following main changes:

  • clarified the tied-table motivation and Figure 1 in the introduction (Page 2 line 81);
  • added a compact main-text overview of all nine experimental settings and two long-horizon continuations (Table 1);
  • expanded Appendix K.7 with matched cap/scale comparisons, diagnostics, and cross-model results (Tables 38-40);
  • added theoretical and empirical comparisons with the symmetry-compatible RowNormM rules (Table 41);
  • added FP32 downstream evaluation of the matched Llama-3.2-1B checkpoints (Subsection J.4.2 Page 42); and
  • clarified the matched-budget scope, training-horizon limitations, and experimental protocols.

All new comparisons use the same frozen architectures, data, optimizer branches, seeds, and matched training protocols as the original experiments. We did not modify the original supplementary material. Detailed responses and relevant section and table references are provided in our replies to each reviewer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.