- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Mano constructs updates by projecting momentum onto weight-dependent tangent spaces and alternating row-wise and column-wise normalization. It offers a computationally inexpensive alternative to Muon, with modest but consistent improvements in the reported pretraining experiments.
Critical strengths
- The optimizer uses a single momentum state and avoids matrix multiplication, making the implementation and computational cost attractive.
- End-to-end timing supports the efficiency claim: training steps remain close to AdamW while being faster than Muon in the tested settings.
Critical weaknesses
- Several recent optimizers already improve on Muon in small-to-medium-scale pretraining, but Mano is compared only with AdamW and Muon. Comparisons with NorMuon (Li et al., 2025) and AdaMuon (Si et al., 2025) would help establish its relative advantage. Matched comparisons of loss and training cost are needed to establish Mano’s advantage over these methods. Besides, the ablations do not cleanly establish the contribution of the proposed geometry.
- Experiments reach only 1.7B parameters, with the largest model trained for 10B tokens. Since optimizer behavior can depend on model size and training duration, these results provide limited evidence that the gains would persist at larger scales, such as 7B, 10B+, and other LLM training stages, such as SFT and RL.
Questions for authors
See Weakness.
2 comments
Thank you for the additional experiments. The comparisons with NorMuon, AdaMuon, and Muon+, together with the component ablations and longer training run, address a substantial part of my concerns.
A few points remain. First, the timing results are reported against Muon, whereas my concern also involves the accuracy–cost trade-off against the newly added baselines. Could you provide their step times and time to a common validation-loss target under the same setup? Please also clarify the memory comparison: standard Muon already maintains a single momentum buffer, so any advantage from temporary workspace should be distinguished from persistent optimizer-state savings.
Second, please provide the best validation losses of both ablated variants after their respective learning-rate sweeps. Their results at Mano’s learning rate demonstrate sensitivity, but the tuned results are more informative for assessing each component’s contribution.
Finally, the longer 350M run provides useful evidence about training duration, but the concern about larger model sizes remains. I therefore consider my concerns partially, but meaningfully, addressed, and will take the additional evidence into account when reconsidering my rating.
We thank the reviewer for the review, and for identifying the comparisons and cost accounting that would make our claims verifiable; each is addressed below with new experiments.
Q1: Comparisons with NorMuon and AdaMuon would help establish its relative advantage. A1: We thank the reviewer for this suggestion and have added NorMuon, AdaMuon, and Muon+ to our LLaMA-350M setting, each with its own learning-rate sweep so that all methods are compared at their own optimum. Mano attains the lowest validation loss of the five; the sweep and the per-optimizer optima are shown below.
Q2: Matched comparisons of loss and training cost are needed. A2: Mano reaches Muon's final validation loss using 14% fewer tokens and 22% less wall-clock time, while also costing less per step (0.99 s against 1.08 s) because it needs neither Newton–Schulz iterations nor a second optimizer state.
We note that we did not optimize Muon's system behaviour in this academic setting, and a distributed implementation that shards the Newton–Schulz iteration across ranks would reduce the measured per-step overhead. The arithmetic comparison, however, is independent of engineering effort: Mano performs no matrix multiplications and maintains a single momentum buffer, so its FLOP and memory advantages persist under any implementation, and because its update is element-wise it requires no cross-rank communication beyond the gradient reduction that both optimizers already perform.
Q3: The ablations do not cleanly establish the contribution of the proposed geometry. A3: To isolate the contribution of the proposed geometry, we ablate Mano into its two constituent operations — the tangent projection onto the Oblique manifold and the row/column update normalization — and tune each with its own learning-rate sweep. Removing either component costs more than the full method gains over Muon, and neither single-component variant tolerates Mano's learning rate: projection-only degrades to 3.3137 and normalization-only to 3.0240 at 1e-3. This ablation indicates that the projection and the normalization stabilise each other rather than contributing independently.
Q4: The results may not persist at larger scales, such as 7B, 10B+, and other LLM training stages, such as SFT and RL. A4: Within the rebuttal period, we could not extend to 7B+ parameters or to SFT and RL, which we state as a limitation. We did test longer training and larger token batches at LLaMA-350M: at TPP=100 (35.4B tokens, 5× Chinchilla) with a 1M-token batch, Mano retains its advantage under per-optimizer learning-rate sweeps.
We are grateful to the reviewer for pushing on the scaling and cost questions, which sharpened how we report both. We would add that the contribution is not only empirical: the analysis identifies uniform and near-uniform amplitude changes as zero- or low-Fisher directions under normalized-block assumptions, and states the condition under which removing them raises the retained update's Fisher–Rayleigh quotient. The geometry is therefore motivated by which directions carry curvature information, not chosen by search, and we hope this framing is useful independently of the specific results reported here.