acceptodds
Under review as a conference paper at ICLR 2027

TMuon: Tangent Momentum improves Muon for LLM Training

11 views

Abstract

Geometry-aware optimization has recently advanced large-scale model pretraining by more effectively leveraging gradient structure and spectral information. Muon, as a representative example, uses Newton–Schulz orthogonalization to construct update directions from momentum matrices. However, Muon’s standard momentum signal can contain substantial components geometrically aligned with the current model weights. In this work, we propose TMuon, a simple yet effective optimizer that incorporates tangent-projected momentum as input to Newton–Schulz orthogonalization within the Muon framework. Tangent filtering suppresses locally redundant components in the update signal, yielding a more informative momentum matrix for spectral normalization. We further show that norm-dependent tangent filtering leverages local geometry along the optimization trajectory and improves the input to Newton–Schulz orthogonalization. Across pretraining experiments on LLaMA and Qwen3 models, TMuon consistently improves optimization performance over Muon and recent variants at negligible additional cost. In particular, on Qwen3-1.7B trained with a B-token budget, TMuon improves average benchmark accuracy over Muon by percentage points, suggesting a promising direction for LLM optimizers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Pretraining, Optimizer

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_94mrOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes TMuon, a variant of Muon motivated by the intuition that part of its momentum is redundant. In particular, the authors consider the momentum component aligned with the current weights to be redundant, which mainly changes the size of the weights rather than their direction. TMuon removes this component from each row or column of the momentum before the Newton-Schulz iteration. The experiments cover dense and MoE models up to 1.7B parameters, including both compute-optimal training and long-horizon training up to 150B tokens. TMuon shows consistent lower loss than Muon, AdamW and its closely related previous work Mano, on both test loss and downstream evaluations.

Critical strengths
  1. The propose method is simple and computationally cheap. It adds only row/column-wise operations on the momentum, and can be easily plugged into existing Muon implementations.

  2. The paper contains comprehensive pre-training experiments that are considered computationally costly under academic budget. It covers a wide range of architecture, scales, and training tokens, with properly swept hyperparameters for moderate scales. The results and the gains of the proposed method are significant and consistent across all variants.

Critical weaknesses
  1. The motivation of the method is not well established. The paper hypothesizes that the momentum component aligned with the current weights mainly changes the size of the weights and is therefore redundant, but there is no direct evidence for it. The paper neither measures how large this component is in Muon’s momentum, nor shows how much or whether it contributes to the loss decrease. There is also no theory behind it. Viewing each row or column as a point on a sphere is what TMuon borrows from Mano, but its weights are never constrained to the sphere as if they were in manifold optimization. Therefore, changing the norm of a row or column is by design a valid way to decrease the loss, and the sphere view cannot justify why removing it would help or not.

  2. As the motivation is not well supported by theory or empirical evidence, it is not clear whether removing the redundant component is what brings the gain, or the alternating row and column rescaling that each TMuon step does. The rescaling alone already changes the matrix fed into the Newton-Schulz iteration by balancing the scales of its rows or columns, and could be the source of some or most of the gain, which is conceptually similar to Muon^2-F [1]. An ablation that only does the rescaling but not the removal would help clarify it, and to establish that TMuon is not reinventing what the community already has.

  3. The comparison with related work is somewhat limited. Besides AdamW, the paper only compares with Muon and Mano. Its own related work lists several Muon variants that change the update in similar ways, such as NorMuon [2], AdaMuon [3] and Muon+ [4]. A comparison with these methods would further strengthen the paper.

  4. Several downstream tasks remain around or even below random guessing in Table 4, such as MMLU, ARC-Challenge, OpenBookQA, CommonsenseQA, LogiQA and MathQA. Differences on these tasks carry little signal, yet some of the largest gains come from them. I suggest evaluating these tasks after fine-tuning on their training data, so that their numbers actually mean something and can separate the quality of the models trained by different optimizers.

[1] Liu, Ziyue, et al. "Muon 2^ 2: Boosting Muon via Adaptive Second-Moment Preconditioning." arXiv preprint arXiv:2604.09967 (2026).

[2] Li, Zichong, et al. "Normuon: Making muon more efficient and scalable." arXiv preprint arXiv:2510.05491 (2025).

[3] Si, Chongjie, Debing Zhang, and Wei Shen. "Adamuon: Adaptive muon optimizer." arXiv preprint arXiv:2507.11005 (2025).

[4] Zhang, Ruijie, et al. "MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-training." arXiv preprint arXiv:2602.21545 (2026).

Questions for authors
  1. What is the purpose of the blockwise generalization? None of the block sizes performs better than the alternating version, and the blockwise variants are not used in any later experiment. It would help to clarify what this generalization adds, either to the method or to the understanding of why it works. If it adds little value, I suggest the authors removing it from the draft to avoid confusion.

  2. Some implementation details are unclear or inconsistent. In the equations, the rescaling divides by the squared norm of each row or column, while the text describes it as a normalization, these two can mean very different things. Could the authors clarify which one is implemented, and also report the number of Newton-Schulz steps, the tokenizer, the sequence length, and which parameters are trained with AdamW?

2 comments

We thank the reviewer for a thorough and well-organised review, and for identifying the baselines, ablations and implementation details that would make our claims verifiable; each is addressed below with new experiments.

W1: Motivation of the method not well established.

A1: TMuon is motivated by making Muon’s Newton–Schulz input parameter-aware: before global spectral normalization, it filters momentum aligned with the current weight slices and emphasizes components that explore new local directions. The product-of-hyperspheres view supplies this local decomposition rather than constraining the parameters to spheres. Empirically, normalization alone matches Muon within 0.0005, whereas filtering without normalization recovers approximately 96% of TMuon’s improvement, supporting this filtering principle as the primary source of the gain.

W2: Lack of ablation that only does the row and col rescaling but not the removal.

A2: We appricate the reviewer for pointing out that limitation. Applying the slice-wise rescaling without the tangent removal attains 2.7556 at LLaMA-350M, reproducing Muon's tuned 2.7551 to within 0.0005. The tangent removal without the rescaling attains 2.6927, recovering about 96% of TMuon's improvement over Muon. The gain is therefore not attributable to rebalancing the Newton--Schulz input, and TMuon is not reinventing rescaling-based variants such as Muon$^2$-F. Learning-rate sweeps for both controls are given in our response to reviewer Hnqy (W2/Q2).

W3: Comparison with related work is limited, like NorMuon, AdaMuon, and Muon+.

A3: We have added NorMuon, AdaMuon and Muon+ to our LLaMA-350M setting, each tuned with its own learning-rate sweep under a matched token budget, architecture and schedule, so that every method is compared at its own optimum. TMuon remains the strongest, improving on the best NorMuon baseline by 0.0554 and on Muon by 0.0654, a margin far beyond the seed spread reported in our response to reviewer Hnqy. We also note that AdaMuon does not outperform the Muon baseline in our experiment setup, attaining 2.7665 against Muon's 2.7551.

Optimizer6e-48e-41e-31.5e-32e-33e-3
Muon–2.77342.76472.75512.75522.7778
AdaMuon–2.78542.77732.76652.77172.7837
Muon+––2.75642.75512.75412.7768
NorMuon–2.76922.75962.74882.74512.7599
Mano2.74232.73872.74432.7813––
TMuon2.72252.69912.68972.7061––

W4: Several downstream tasks remain around or even below random guessing.

A4: We could not perform supervised fine-tuning within the rebuttal period because of computational-resource constraints. However, the near-chance tasks do not drive TMuon’s aggregate advantage: after removing MMLU, ARC-Challenge, OpenBookQA, CommonsenseQA, LogiQA, and MathQA, TMuon’s average lead over Muon increases from 1.24 to 1.84 percentage points for Qwen3-0.6B and from 2.29 to 2.86 points for Qwen3-1.7B. Thus, excluding these benchmarks strengthens rather than weakens the reported improvement.

Q1: Purpose of the block-wise generalization.

A5: The block-wise generalization serves both the method and the explanation. In our study, TMuon extends Mano's Oblique-manifold-constrained update to norm-dependent momentum filtering and, through the block-wise form, to a general family of product manifolds whose granularity is a design choice — per coordinate, per block, per row/column, or the whole matrix. Block-wise filtering is not merely a generalization that preserves the gain but is itself effective: at LLaMA-350M every block size from 8 to 32 outperforms Muon's tuned 2.7551, with block-32 reaching 2.7100 for a margin of 0.0451. The ladder also turns at block-64, so granularity has an interior optimum and is a genuine design axis rather than a free parameter. The same holds at LLaMA-130M, where multiple block sizes improved on Muon's tuned best of 2.9277.

Block size8163264Muon
Val. loss2.74122.71412.71002.83102.7551

Q2: Implementation details are unclear.

A6: We clarify the implementation details here. The tangent filtering uses the unnormalized weight slice, m←m−α⟨m,w⟩wm \leftarrow m - \alpha \langle m, w \rangle w, so unlike a true tangent projection it does not divide by ∥w∥2\|w\|^2, as discussed in Section 3.3. The subsequent rescaling divides each filtered slice by its L2 norm to the first power. All reported runs use 5 Newton--Schulz iterations, the T5 tokenizer with a 32,000-token vocabulary, sequence length 512 for LLaMA-130M/350M and 1024 for LLaMA-780M/1.3B and Qwen3, and route the embeddings, output head and all one-dimensional parameters to the auxiliary AdamW, while the matrix optimizer updates only the two-dimensional block weights.

We thank the reviewer again for these constructive comments.

I thank the authors for their detailed rebuttal. My concerns are mostly addressed, except the downstream tasks that remain at random guessing. I strongly recommend the authors to add fine-tuned version to their camera ready version if it get accepted, or, to the next round of submission if not. Nevertheless, I believe this paper now meets the bar for ICLR and will improve my score to 3.

Reviewer_CvW5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TMuon, which applies weight-norm-dependent row- or column-wise momentum filtering before Muon's Newton–Schulz orthogonalisation, alternating between the two across optimisation steps. Experiments on LLaMA, Qwen3, and MoE models show lower pretraining loss than AdamW, Muon, and Mano. TMuon also improves average downstream accuracy in two long-run Qwen3 experiments.

Critical strengths
  1. TMuon modifies the momentum signal before Newton–Schulz orthogonalisation using a simple, computationally inexpensive operation. This approach could complement other Muon variants without substantially increasing computational overhead.

  2. The experiments cover different architectures and scales, including dense models from 130M to 1.7B parameters, an MoE model, and long-run pretraining with up to 150B tokens. TMuon achieves the lowest loss in all five settings in Table 3.

  3. The motivation is interesting. Weight-aligned updates mainly change weight magnitudes rather than directions, which may be less useful in networks with normalisation or scale invariance. Combining this local geometric consideration with Muon's global spectral normalisation provides a reasonable direction for optimiser design.

Critical weaknesses
  1. The experiments do not adequately validate the proposed mechanism. The paper does not measure the alignment between momentum and weights, the loss effects of radial and tangential components, or the distribution of s=∥w∥2s=\lVert w\rVert^2 during training. The TMuonS ablation shows that completely removing the radial component is less effective, but no experiment isolates how much of TMuon’s improvement over Muon comes from slice-wise normalisation, effective learning-rate changes, or implicit regularisation. A control that applies the same slice-wise normalisation without subtracting ⟨g,w⟩w\langle g,w\rangle w would help distinguish these effects.

  2. The strength of the radial modification is not systematically studied. TMuon uses g−⟨g,w⟩w=g⊥+(1−s)g∥g-\langle g,w\rangle w=g_\perp+(1-s)g_\parallel, where s=∥w∥2s=\lVert w\rVert^2. The radial component therefore reverses when s>1s>1, not only when s≥2s\geq2 as stated in Section 3.3. For s>2s>2, its reversed magnitude exceeds the original. The paper neither reports the distribution of ss nor examines the general transformation g−γ⟨g,w⟩wg-\gamma\langle g,w\rangle w where γ\gamma is the coefficient. The choice γ=1\gamma=1 requires stronger justification.

  3. The long-run Qwen3 experiments use the same learning rate for Muon and TMuon, although Figure 4 shows that they generally prefer different learning rates. Table 4 therefore demonstrates an advantage under a shared configuration, not necessarily over independently tuned Muon. This is particularly important for the 1.7B, 150B-token result highlighted in the abstract.

Questions for authors
  1. Could the authors report the distribution of s=∥w∥2s=\lVert w\rVert^2 and the momentum-alignment ratio ∥g∥∥2/∥g∥2\lVert g_\parallel\rVert^2/\lVert g\rVert^2 across layers and training stages? What fractions of slices fall into s<1s<1, 1≤s≤21\leq s \leq2, and s>2s>2?

  2. Have the authors tested g−γ⟨g,w⟩wg-\gamma\langle g,w\rangle w with different values of γ\gamma? What motivates γ=1\gamma=1? How does TMuon compare with a control using the same slice-wise normalisation without radial subtraction?

  3. Given the different learning-rate preferences in Figure 4, could the authors test nearby learning rates for Muon and TMuon separately in the Qwen3-0.6B and Qwen3-1.7B experiments?

I would raise my score if the authors address these questions properly.

2 comments

We thank the reviewer for a detailed and technically precise review, and in particular for the observation on the reversal threshold, which identified a genuine imprecision in our Section 3.3 and prompted the distributional analysis reported below.

W1: Experiments do not validate the proposed mechanism.

A1: Applying the same slice-wise normalization without subtracting the radial component attains 2.7460 at LLaMA-350M under its own learning-rate sweep, only 0.0091 better than Muon's tuned 2.7551, whereas subtracting the radial component without the normalization attains 2.6927, recovering about 96% of TMuon's improvement; the gain therefore comes from the radial removal rather than from slice-wise normalization. Effective update scale is also not the explanation: the filtered momentum is passed through Newton--Schulz orthogonalization and the update magnitude is then fixed by Muon's shape-based factor 0.2max⁡(m,n),η0.2\sqrt{\max(m,n)},\eta, so a change in pre-orthogonalization scale is largely absorbed, yet the two controls differ by 0.053 in final loss under identical post-orthogonalization scaling.

W2/Q1: Strength of the radial modifications is not sysmematically studied.

A2: Writing s=α∥w∥2s = \alpha\|w\|^2, the filtered momentum is m⊥+(1−s)m∥m_\perp + (1-s)m_\parallel. Measuring ss over all 168 matrix parameters and 254,208 slices, the filtering is genuinely norm-dependent and evolves during training: the median rises from 0.390 at step 500 to 0.738 at step 27,000, the sign-reversed fraction from 9.0% to 39.4%, while the reversed-larger fraction stays below 0.6%. The bare momentum--weight cosine is modest (median 0.042), but the norm-weighted component that TMuon removes is substantial (median 0.218 of the momentum norm, 0.80 at the 90th percentile), which is precisely why the norm-dependent form matters. On the choice of α\alpha, the sweep in A3 improves monotonically from α=0\alpha=0 to α=1\alpha=1 and is flat to α=1.25\alpha=1.25, placing the default at the optimum.

W3/Q3: Qwen3 long-run use the same learning rate.

A3: Following Moonlight AI’s Muon recipe, Muon and TMuon use the same shape-based update rescaling to match their nominal update RMS and share the same learning-rate schedule. Under this controlled configuration, TMuon reaches lower final loss at both the 100B- and 150B-token scales. This trend is also consistent with our Chinchilla-optimal Qwen3-0.6B and 1.7B experiments, where TMuon improves over Muon, showing that the advantage is not confined to one long-run configuration. Explicit optimizer-specific sweeps at the target scales would require multiple additional 100B–150B-token runs and were infeasible within the rebuttal period. We will therefore qualify the long-run claims as matched-configuration results and attempt to provide independent target-scale tuning in a subsequent revision.

Q2: Ablation studies on different values of \gamma.

A4: The coefficient γ\gamma in m−γ⟨m,w⟩wm - \gamma \langle m, w \rangle w is the filtering strength we denote α\alpha. Sweeping it at LLaMA-350M with the learning rate held at TMuon's optimum gives an interior minimum at γ=1\gamma = 1, with both under-filtering and over-filtering worse, so the default is the empirical optimum rather than an arbitrary choice.

γ\gamma00.250.50.751.01.251.52.0
Val. loss2.75562.71662.69872.69282.68972.69022.69302.7074

We thank the reviewer again for their reading. We would like to emphasize what we believe this study contributes: an optimizer design perspective in which the update is shaped at two geometry levels, a local stage that performs weight-dependent momentum clipping at different granularities of product manifolds, and the global spectral stage performed by plain Muon. While certain limitation applies, we hope this separation is contributive to the community beyond the particular optimizer we propose.

Thank you for the detailed response and additional experiments. The new experiments and analyses have addressed my concerns regarding W1, W2, Q1, and Q2.

However, I do not believe W3/Q3 has been fully resolved. The two methods should be compared after each has been adequately tuned, since the shared configuration may favour one method, and practitioners would typically tune whichever method they use.

Overall, the authors have addressed most of my concerns, so I will raise my score.

Reviewer_HnqyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TMuon, a geometry-aware optimizer for large language model pretraining that incorporates tangent momentum filtering into the Muon framework. Specifically, TMuon applies row-wise and column-wise momentum filtering before Newton–Schulz orthogonalization to suppress weight-aligned components. Unlike standard tangent projection, the proposed norm-dependent filtering retains information about local parameter magnitudes. The paper also presents a product-manifold interpretation that generalizes the filtering operation to different geometric granularities.

The main contribution is a lightweight modification to Muon's momentum processing that aims to improve optimization performance with minimal additional computational overhead. The authors evaluate TMuon on LLaMA and Qwen3 models across multiple scales, including long-run pretraining with up to 150B tokens. Experimental results show consistent reductions in pretraining loss compared with Muon, AdamW, and Mano, along with improvements in average downstream benchmark accuracy. Ablation studies investigate different projection geometries and the effectiveness of norm-dependent filtering, while computational analyses suggest that the additional optimizer overhead is small.

Critical strengths

TMuon achieves lower pretraining loss than Muon, AdamW, and Mano across multiple LLaMA and Qwen3 configurations, including dense and mixture-of-experts models. The improvements also extend to long-run pretraining with up to 150B tokens, where TMuon improves average downstream benchmark accuracy by 2.29 percentage points over Muon on Qwen3-1.7B. These results provide meaningful evidence of the method's practical effectiveness and scalability, representing the strongest argument in favor of acceptance.

TMuon introduces a lightweight momentum filtering operation without modifying Muon's Newton–Schulz orthogonalization procedure. The reported optimizer-level runtime overhead is only approximately 0.4%–3.5%, suggesting that the method can be incorporated into existing Muon-based training pipelines with relatively little additional cost. This favorable trade-off between implementation complexity and empirical performance is particularly valuable for large-scale language model training.

Critical weaknesses
  1. Limited methodological novelty and insufficient distinction from prior work. TMuon primarily combines the tangent projection idea from Mano with the Newton–Schulz orthogonalization used in Muon. Its main methodological difference is the use of unnormalized weight vectors in the projection, resulting in norm-dependent filtering. While this modification is empirically effective, the paper does not sufficiently establish why it constitutes a substantial methodological advance rather than a relatively simple modification of existing techniques.

  2. Insufficient evidence supporting the proposed optimization mechanism. The paper argues that weight-aligned momentum components are less informative and that norm-dependent filtering improves the input to Newton–Schulz orthogonalization. However, these claims are not directly validated. In particular, it remains unclear whether the improvements arise from removing redundant momentum components, implicitly controlling parameter norms, or simply changing the effective update directions and scales.

  3. Incomplete ablation studies and hyperparameter analysis. Although the paper compares different projection geometries and standard versus norm-dependent tangent filtering, several important factors remain unexplored. For example, there is no systematic analysis of the filtering strength or its interaction with momentum, weight decay, and update normalization. Moreover, the large-scale Qwen3 experiments use shared learning rates rather than independently tuned configurations, making it difficult to assess the robustness of the reported improvements.

  4. Lack of theoretical guarantees and incomplete efficiency evaluation. The paper provides a geometric interpretation and FLOP complexity analysis but does not establish convergence guarantees, descent properties, or a theoretical explanation of how tangent filtering interacts with Newton–Schulz orthogonalization. Furthermore, the reported runtime analysis focuses on individual optimizer operations rather than end-to-end training efficiency, leaving the practical computational benefits insufficiently characterized.

Questions for authors
  1. Could the authors clarify the key methodological novelty of TMuon relative to Mano and Muon? Since the main difference from standard tangent projection is the removal of weight-norm normalization, it would be helpful to explain why this modification represents a distinct optimization principle rather than a heuristic adjustment to existing methods.

  2. What evidence supports the proposed explanation for the effectiveness of norm-dependent tangent filtering? Could the authors investigate whether the improvements primarily result from suppressing weight-aligned momentum, controlling parameter norms, or changing the effective update scale? An ablation with a tunable filtering coefficient, together with an analysis of the resulting update directions, would help clarify the underlying mechanism.

  3. How sensitive is TMuon to its hyperparameters and implementation choices? In particular, could the authors examine the effects of filtering strength, momentum coefficient, and weight decay? Since the large-scale Qwen3 experiments use shared learning rates, it would also be useful to establish whether the improvements persist when Muon and TMuon are independently tuned.

  4. How robust are the empirical improvements compared with other recent Muon variants? Could the authors include comparisons with representative methods such as NorMuon, Muon+, or AdaMuon under matched training budgets? Additional random seeds and statistical analysis, particularly for the smaller models, would help establish the reliability of the improvements.

  5. Can the authors provide stronger theoretical or empirical justification for the optimization and efficiency claims? In particular, is it possible to characterize the descent properties or convergence behavior of the proposed update, especially given its interaction with Newton–Schulz orthogonalization?

2 comments

We sincerely thank the reviewer for a careful and demanding review, and for pressing on novelty, mechanism and efficiency together; we address each below with new measurements.

W1/Q1: Limited methodological novelty and insufficient distinction from prior work.

A1: TMuon’s distinct principle is norm-dependent filtering of weight-aligned momentum before Newton--Schulz orthogonalization, rather than Mano’s exact tangent projection of normalized slices. Its blockwise formulation extends this principle across geometric granularities, locating where in the weight structure the redundant momentum resides.

W2/Q2: Insufficient evidence of the proposed optimization mechanism.

A2: We isolate the two operations that tangent filtering performs, each with its own learning-rate sweep at LLaMA-350M: slice-wise rescaling without the tangent removal attains 2.7460, improving on Muon's tuned 2.7551 by 0.0091 but remaining 0.0563 behind TMuon, while the tangent removal without rescaling attains 2.6927, recovering about 96% of TMuon's improvement over Muon. The gain comes from suppressing the weight-aligned momentum component, not from rebalancing the Newton-Schulz input.

Variant8e-41e-31.5e-32e-33e-3
Muon2.77342.76472.75512.75522.7778
Normalization-only TMuon2.76462.75562.74812.74602.7478
Projection-only TMuon2.70162.69272.7020––
TMuon2.69912.68972.7061––

W3: Incomplete ablation studies of filtering strength and hyperparameter analysis.

A3: We sweep the filtering strength α at LLaMA-350M with the learning rate held at TMuon's optimum. Performance improves monotonically from α = 0, which applies no removal, to α = 1, and is flat between 1 and 1.25, so the default α = 1 lies at the optimum rather than being an arbitrary choice.

α\alpha00.250.50.751.01.251.52.0
Val. loss2.75562.71662.69872.69282.68972.69022.69302.7074

W4/Q3: Lack of theoretical guarantees.

A4: We characterize the local stability of TMuon directly. Decomposing a momentum slice as m=m⊥+m∥m=m_\perp+m_\parallel relative to ww gives Fα(m,w)=m⊥+(1−α∥w∥2)m∥\mathcal F_\alpha(m,w)=m_\perp+(1-\alpha\|w\|^2)m_\parallel. Thus, TMuon preserves the orthogonal component exactly and scales only the weight-aligned component according to the local parameter norm. For fixed ww, this is a linear operator with Lipschitz constant max⁡{1,∣1−α∥w∥2∣}\max\{1,|1-\alpha\|w\|^2|\}, providing an exact local stability characterization of the filtering step. Convergence analyses of modern LLM optimizers generally do not explain their empirical advantages and are often secondary to experimentally validated design principles. Our theoretical claim is therefore limited to this local characterization; it does not imply global descent after stochastic momentum and finite-step Newton--Schulz orthogonalization. Establishing an end-to-end guarantee for their composition remains future work.

W5/Q5: Incomplete efficiency evaluation.

A5: We report the per-update wall time of the complete training step rather than isolated optimizer operations, measured as the median across tuned learning-rate runs of each run's trimmed-mean update time. Beyond the smallest scale, TMuon's overhead over Muon is within run-to-run variation (+0.1% to +0.9%), and both remain 5--11% above AdamW; at LLaMA-350M this is the same ≈10.5{\approx}10.5 h per 27,000-step run for Muon and TMuon.

ModelAdamWMuonTMuonTMuon vs. Muon
LLaMA-130M0.330 s0.359 s0.365 s+1.7%
LLaMA-350M0.918 s1.021 s1.022 s+0.1%
LLaMA-780M6.335 s6.596 s6.658 s+0.9%
LLaMA-1.3B8.415 s9.253 s9.270 s+0.2%

Q4: Lack of comparisons with other recent Muon variants and statistical analysis for the smaller models.

A6: We compare against two recent Muon variants at LLaMA-350M, each with its own learning-rate sweep. TMuon attains the lowest validation loss, improving on the strongest baseline of NorMuon by 0.0554 and on Muon by 0.0654. We sincerely guide the reviewer to A3 of our response to reviewer 94mr for the full comparison, which additionally includes AdaMuon. On statistical reliability, we rerun both TMuon and Muon at their tuned learning rates across three independent seeds that vary initialization and data order. The resulting 0.0668 margin is much smaller than TMuon's margin over the strongest baseline 0.0565 of NorMuon, so the improvement is far outside run-to-run variation.

Optimizer (LR)seed 0seed 1seed 2mean ± 95% CI
Muon (1.5e-3)2.75512.75422.75692.7554 ± 0.0034
TMuon (1e-3)2.68972.68742.68882.6886 ± 0.0029

We are grateful to the reviewer for the care taken with our ablations, which directed us toward the filtering-strength sweep and the component controls reported above.

Dear reviewer, we sincerely appreciate your time and effort in reviewing our work; we have sought to address your concerns during the rebuttal period and would be grateful if you could reconsider your score. If you have any remaining concern, we would like to further address it before the end of the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.