- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes TMuon, a variant of Muon motivated by the intuition that part of its momentum is redundant. In particular, the authors consider the momentum component aligned with the current weights to be redundant, which mainly changes the size of the weights rather than their direction. TMuon removes this component from each row or column of the momentum before the Newton-Schulz iteration. The experiments cover dense and MoE models up to 1.7B parameters, including both compute-optimal training and long-horizon training up to 150B tokens. TMuon shows consistent lower loss than Muon, AdamW and its closely related previous work Mano, on both test loss and downstream evaluations.
Critical strengths
-
The propose method is simple and computationally cheap. It adds only row/column-wise operations on the momentum, and can be easily plugged into existing Muon implementations.
-
The paper contains comprehensive pre-training experiments that are considered computationally costly under academic budget. It covers a wide range of architecture, scales, and training tokens, with properly swept hyperparameters for moderate scales. The results and the gains of the proposed method are significant and consistent across all variants.
Critical weaknesses
-
The motivation of the method is not well established. The paper hypothesizes that the momentum component aligned with the current weights mainly changes the size of the weights and is therefore redundant, but there is no direct evidence for it. The paper neither measures how large this component is in Muon’s momentum, nor shows how much or whether it contributes to the loss decrease. There is also no theory behind it. Viewing each row or column as a point on a sphere is what TMuon borrows from Mano, but its weights are never constrained to the sphere as if they were in manifold optimization. Therefore, changing the norm of a row or column is by design a valid way to decrease the loss, and the sphere view cannot justify why removing it would help or not.
-
As the motivation is not well supported by theory or empirical evidence, it is not clear whether removing the redundant component is what brings the gain, or the alternating row and column rescaling that each TMuon step does. The rescaling alone already changes the matrix fed into the Newton-Schulz iteration by balancing the scales of its rows or columns, and could be the source of some or most of the gain, which is conceptually similar to Muon^2-F [1]. An ablation that only does the rescaling but not the removal would help clarify it, and to establish that TMuon is not reinventing what the community already has.
-
The comparison with related work is somewhat limited. Besides AdamW, the paper only compares with Muon and Mano. Its own related work lists several Muon variants that change the update in similar ways, such as NorMuon [2], AdaMuon [3] and Muon+ [4]. A comparison with these methods would further strengthen the paper.
-
Several downstream tasks remain around or even below random guessing in Table 4, such as MMLU, ARC-Challenge, OpenBookQA, CommonsenseQA, LogiQA and MathQA. Differences on these tasks carry little signal, yet some of the largest gains come from them. I suggest evaluating these tasks after fine-tuning on their training data, so that their numbers actually mean something and can separate the quality of the models trained by different optimizers.
[1] Liu, Ziyue, et al. "Muon : Boosting Muon via Adaptive Second-Moment Preconditioning." arXiv preprint arXiv:2604.09967 (2026).
[2] Li, Zichong, et al. "Normuon: Making muon more efficient and scalable." arXiv preprint arXiv:2510.05491 (2025).
[3] Si, Chongjie, Debing Zhang, and Wei Shen. "Adamuon: Adaptive muon optimizer." arXiv preprint arXiv:2507.11005 (2025).
[4] Zhang, Ruijie, et al. "MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-training." arXiv preprint arXiv:2602.21545 (2026).
Questions for authors
-
What is the purpose of the blockwise generalization? None of the block sizes performs better than the alternating version, and the blockwise variants are not used in any later experiment. It would help to clarify what this generalization adds, either to the method or to the understanding of why it works. If it adds little value, I suggest the authors removing it from the draft to avoid confusion.
-
Some implementation details are unclear or inconsistent. In the equations, the rescaling divides by the squared norm of each row or column, while the text describes it as a normalization, these two can mean very different things. Could the authors clarify which one is implemented, and also report the number of Newton-Schulz steps, the tokenizer, the sequence length, and which parameters are trained with AdamW?
2 comments
I thank the authors for their detailed rebuttal. My concerns are mostly addressed, except the downstream tasks that remain at random guessing. I strongly recommend the authors to add fine-tuned version to their camera ready version if it get accepted, or, to the next round of submission if not. Nevertheless, I believe this paper now meets the bar for ICLR and will improve my score to 3.
We thank the reviewer for a thorough and well-organised review, and for identifying the baselines, ablations and implementation details that would make our claims verifiable; each is addressed below with new experiments.
W1: Motivation of the method not well established.
A1: TMuon is motivated by making Muon’s Newton–Schulz input parameter-aware: before global spectral normalization, it filters momentum aligned with the current weight slices and emphasizes components that explore new local directions. The product-of-hyperspheres view supplies this local decomposition rather than constraining the parameters to spheres. Empirically, normalization alone matches Muon within 0.0005, whereas filtering without normalization recovers approximately 96% of TMuon’s improvement, supporting this filtering principle as the primary source of the gain.
W2: Lack of ablation that only does the row and col rescaling but not the removal.
A2: We appricate the reviewer for pointing out that limitation. Applying the slice-wise rescaling without the tangent removal attains 2.7556 at LLaMA-350M, reproducing Muon's tuned 2.7551 to within 0.0005. The tangent removal without the rescaling attains 2.6927, recovering about 96% of TMuon's improvement over Muon. The gain is therefore not attributable to rebalancing the Newton--Schulz input, and TMuon is not reinventing rescaling-based variants such as Muon$^2$-F. Learning-rate sweeps for both controls are given in our response to reviewer Hnqy (W2/Q2).
W3: Comparison with related work is limited, like NorMuon, AdaMuon, and Muon+.
A3: We have added NorMuon, AdaMuon and Muon+ to our LLaMA-350M setting, each tuned with its own learning-rate sweep under a matched token budget, architecture and schedule, so that every method is compared at its own optimum. TMuon remains the strongest, improving on the best NorMuon baseline by 0.0554 and on Muon by 0.0654, a margin far beyond the seed spread reported in our response to reviewer Hnqy. We also note that AdaMuon does not outperform the Muon baseline in our experiment setup, attaining 2.7665 against Muon's 2.7551.
W4: Several downstream tasks remain around or even below random guessing.
A4: We could not perform supervised fine-tuning within the rebuttal period because of computational-resource constraints. However, the near-chance tasks do not drive TMuon’s aggregate advantage: after removing MMLU, ARC-Challenge, OpenBookQA, CommonsenseQA, LogiQA, and MathQA, TMuon’s average lead over Muon increases from 1.24 to 1.84 percentage points for Qwen3-0.6B and from 2.29 to 2.86 points for Qwen3-1.7B. Thus, excluding these benchmarks strengthens rather than weakens the reported improvement.
Q1: Purpose of the block-wise generalization.
A5: The block-wise generalization serves both the method and the explanation. In our study, TMuon extends Mano's Oblique-manifold-constrained update to norm-dependent momentum filtering and, through the block-wise form, to a general family of product manifolds whose granularity is a design choice — per coordinate, per block, per row/column, or the whole matrix. Block-wise filtering is not merely a generalization that preserves the gain but is itself effective: at LLaMA-350M every block size from 8 to 32 outperforms Muon's tuned 2.7551, with block-32 reaching 2.7100 for a margin of 0.0451. The ladder also turns at block-64, so granularity has an interior optimum and is a genuine design axis rather than a free parameter. The same holds at LLaMA-130M, where multiple block sizes improved on Muon's tuned best of 2.9277.
Q2: Implementation details are unclear.
A6: We clarify the implementation details here. The tangent filtering uses the unnormalized weight slice, m←m−α⟨m,w⟩w, so unlike a true tangent projection it does not divide by ∥w∥2, as discussed in Section 3.3. The subsequent rescaling divides each filtered slice by its L2 norm to the first power. All reported runs use 5 Newton--Schulz iterations, the T5 tokenizer with a 32,000-token vocabulary, sequence length 512 for LLaMA-130M/350M and 1024 for LLaMA-780M/1.3B and Qwen3, and route the embeddings, output head and all one-dimensional parameters to the auxiliary AdamW, while the matrix optimizer updates only the two-dimensional block weights.
We thank the reviewer again for these constructive comments.