acceptodds
Under review as a conference paper at ICLR 2027

Mano: Restriking Manifold Optimization for LLM Pretraining

Abstract

While large language models (LLMs) have emerged as a significant advancement in artificial intelligence, optimizer design for LLMs has attracted much attention and costs huge resources. In this line of research, advanced optimizers, such as Adam and Muon, with complex operations are mainstream methods, while conventional manifold optimization methods have been largely overlooked due to their poor performance in large-scale model optimization. In this study, we try to restrike manifold optimization for training LLMs. We design a novel Manifold-Normalized Optimizer (Mano) that trains LLMs on a special rotational Oblique manifold implicitly. Mano's Oblique geometry actually corresponds to a product of neuron-wise hyperspheres that make training LLMs easy even with simple momentum. We also theoretically analyzed Mano in terms of convergence and Fisher information. Experimental results demonstrate that Mano outperforms AdamW and Muon across multi-scale LLaMA and Qwen3 models. Meanwhile, Mano incurs lower computational and memory costs than Muon and Adam.

Primary area:
optimization
Keywords:
Large Language Model, Optimizer

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DoFSOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Mano constructs updates by projecting momentum onto weight-dependent tangent spaces and alternating row-wise and column-wise normalization. It offers a computationally inexpensive alternative to Muon, with modest but consistent improvements in the reported pretraining experiments.

Critical strengths
  1. The optimizer uses a single momentum state and avoids matrix multiplication, making the implementation and computational cost attractive.
  2. End-to-end timing supports the efficiency claim: training steps remain close to AdamW while being faster than Muon in the tested settings.
Critical weaknesses
  1. Several recent optimizers already improve on Muon in small-to-medium-scale pretraining, but Mano is compared only with AdamW and Muon. Comparisons with NorMuon (Li et al., 2025) and AdaMuon (Si et al., 2025) would help establish its relative advantage. Matched comparisons of loss and training cost are needed to establish Mano’s advantage over these methods. Besides, the ablations do not cleanly establish the contribution of the proposed geometry.
  2. Experiments reach only 1.7B parameters, with the largest model trained for 10B tokens. Since optimizer behavior can depend on model size and training duration, these results provide limited evidence that the gains would persist at larger scales, such as 7B, 10B+, and other LLM training stages, such as SFT and RL.
Questions for authors

See Weakness.

2 comments

We thank the reviewer for the review, and for identifying the comparisons and cost accounting that would make our claims verifiable; each is addressed below with new experiments.

Q1: Comparisons with NorMuon and AdaMuon would help establish its relative advantage. A1: We thank the reviewer for this suggestion and have added NorMuon, AdaMuon, and Muon+ to our LLaMA-350M setting, each with its own learning-rate sweep so that all methods are compared at their own optimum. Mano attains the lowest validation loss of the five; the sweep and the per-optimizer optima are shown below.

Optimizer6e-48e-41e-31.5e-32e-33e-3
Muon–2.77342.76472.75512.75522.7778
Mano2.74232.73872.7443–––
NorMuon–2.76922.75962.74882.74512.7599
AdaMuon–2.78542.77732.76652.77172.7837
Muon+––2.75642.75512.75412.7769

Q2: Matched comparisons of loss and training cost are needed. A2: Mano reaches Muon's final validation loss using 14% fewer tokens and 22% less wall-clock time, while also costing less per step (0.99 s against 1.08 s) because it needs neither Newton–Schulz iterations nor a second optimizer state.

Optimizers/stepTokens to 2.7551Time to 2.7551
Muon1.087.08B8.1 h
Mano0.996.06B6.4 h

We note that we did not optimize Muon's system behaviour in this academic setting, and a distributed implementation that shards the Newton–Schulz iteration across ranks would reduce the measured per-step overhead. The arithmetic comparison, however, is independent of engineering effort: Mano performs no matrix multiplications and maintains a single momentum buffer, so its FLOP and memory advantages persist under any implementation, and because its update is element-wise it requires no cross-rank communication beyond the gradient reduction that both optimizers already perform.

Q3: The ablations do not cleanly establish the contribution of the proposed geometry. A3: To isolate the contribution of the proposed geometry, we ablate Mano into its two constituent operations — the tangent projection onto the Oblique manifold and the row/column update normalization — and tune each with its own learning-rate sweep. Removing either component costs more than the full method gains over Muon, and neither single-component variant tolerates Mano's learning rate: projection-only degrades to 3.3137 and normalization-only to 3.0240 at 1e-3. This ablation indicates that the projection and the normalization stabilise each other rather than contributing independently.

Variant6e-48e-41e-3
Projection-only Mano2.79582.78683.3137
Normalization-only Mano2.84382.88773.0240
Mano2.74572.73872.7449

Q4: The results may not persist at larger scales, such as 7B, 10B+, and other LLM training stages, such as SFT and RL. A4: Within the rebuttal period, we could not extend to 7B+ parameters or to SFT and RL, which we state as a limitation. We did test longer training and larger token batches at LLaMA-350M: at TPP=100 (35.4B tokens, 5× Chinchilla) with a 1M-token batch, Mano retains its advantage under per-optimizer learning-rate sweeps.

Tokens/stepSeq lenGlobal batch (samples)Optimizer6e-48e-41e-31.5e-32e-33e-34e-3
1M10241024Muon––2.2792–2.26962.2752–
1M10241024Mano2.2595–2.24412.2706–––

We are grateful to the reviewer for pushing on the scaling and cost questions, which sharpened how we report both. We would add that the contribution is not only empirical: the analysis identifies uniform and near-uniform amplitude changes as zero- or low-Fisher directions under normalized-block assumptions, and states the condition under which removing them raises the retained update's Fisher–Rayleigh quotient. The geometry is therefore motivated by which directions carry curvature information, not chosen by search, and we hope this framing is useful independently of the specific results reported here.

Thank you for the additional experiments. The comparisons with NorMuon, AdaMuon, and Muon+, together with the component ablations and longer training run, address a substantial part of my concerns.

A few points remain. First, the timing results are reported against Muon, whereas my concern also involves the accuracy–cost trade-off against the newly added baselines. Could you provide their step times and time to a common validation-loss target under the same setup? Please also clarify the memory comparison: standard Muon already maintains a single momentum buffer, so any advantage from temporary workspace should be distinguished from persistent optimizer-state savings.

Second, please provide the best validation losses of both ablated variants after their respective learning-rate sweeps. Their results at Mano’s learning rate demonstrate sensitivity, but the tuned results are more informative for assessing each component’s contribution.

Finally, the longer 350M run provides useful evidence about training duration, but the concern about larger model sizes remains. I therefore consider my concerns partially, but meaningfully, addressed, and will take the additional evidence into account when reconsidering my rating.

Reviewer_51FdOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes the Mano for LLM pretraining inspired from manifold optimization. The main steps of the Mano algorithm include projecting the momentum onto the tangent space of the - alternating row/column wise - normalized weights, then alternating row/column wise normalization of the projected momentum and finally using a rescaled version of it as the update (along with decoupled weight decay). The paper's pretraining experiment include LLaMA and Qwen3 comparisons with AdamW and Muon on C4 and Pile, suggesting improved performance.

Critical strengths

The topic of the paper is important as the potential benefits of a faster optimizer for pretraining are substantial (pretraining is the most costly phase of the LLM development pipeline). The proposed optimizer appears simple and does not contain compute-heavy operations.

Critical weaknesses

Issues with experimental comparison/settings.

The experimental setup and settings for the optimizer comparisons raise several question about the reliability and practical value of the results. I list my main concerns below:

  1. C4 sequence length. The main pretraining dataset used is C4, which has an average of 427 tokens per sample ([1] table 1; divide number of tokens by number of documents). Yet the paper mentions using sequence length of up to 1024. This suggests that a large number of the tokens per sequence are padding, which effectively reduces the (token) batch size (number of non-padding tokens) and the total number of tokens used for training. If some packing technique was used instead this needs to be disclosed.

  2. Token Batch sizes. Connected to the above issue, the token batch sizes (sequence length x batch size) used are quite low, as low as 0.26M tokens, and assuming more than half of those are padding the actual token batch size can be as low as 0.1M. While small batch size training can be useful for memory-constrained scenarios, the current writing of the paper does not constrain itself to memory-efficient applications, instead it claims generic optimizer performance improvements in pretraining.

    Practical pretraining setups use orders of magnitude larger token batch sizes (e.g., 16M-33M tokens as in [3]). As Muon (and generally optimizers that perform orthogonalization/NS ) is also known to perform best for larger batchsizes, this raises the question of whether Mano can handle such batchsizes, or at least of comparable order (e.g. 4M tokens). Can the authors provide an ablation showing whether Mano can sustain larger batchsizes as compared to Muon?

  3. Qwen learning rates. It appears that the Qwen learning rates are not tuned separately for the different optimizers ("using shared per-corpus learning rates of 1x10−3 and 3x10−4, respectively"). If this is the case the Qwen comparisons are not useful since it has not been established if those learning rates are best/suitable for each optimizer. The LLaMA results indicate that the best learning rates differ for each optimizer. I ask the authors to clarify about the Qwen learning rates.

  4. No weight decay tuning. The experiments use weight decay 0.1 for all experiments and optimizers. Optimal weight decay can differ between optimizers and settings (e.g., as indicated in Fig. 1 of [2]). Also from the LLaMA experiments Table 4 it is shown that the Mano LR is up to 2.5x smaller than the Muon LR. This means that with the same weight decay coefficient Muon receives up to 2.5x decay (lr*wd) compared to Mano, which might not be optimal.

  5. Auxiliary Adam settings. The paper does not report the settings for the auxiliary Adam optimizer used to train the non-matrix parameters in the Muon/Mano experiments. Is the auxiliary Adam learning rate tuned separately or the main optimizer learning rates are reused for that purpose too? If the later is the case this is another confounding factor; it is possible that the lower Mano LRs are more suitable to be reused as the aux Adam LRs.

  6. Muon time overhead. The paper reports up to 10% wall-clock time overhead per step for Muon compared to Adam in a 8 and 16 GPU setup. This is quite high and possibly indicates that the Muon optimizer was implemented improperly from a systems perspective. It is standard practice in multi-gpu setups to distribute the NS calculation across the GPUs so that each GPU handles a different set of the network's layers. Large scale studies (e.g., Moonshot AI [3]) report negligible latency of distributed Muon compared to Adam.

    The paper does not report any implementation details of Muon nor provides training code, so I cannot determine what Muon scheme was used; I ask the authors to clarify.

  7. Lack of competitive benchmarking results. The paper performs experiments under its own benchmarking setup and lacks results under competitive benchmarking settings. It is common for pretraining optimizer papers test in the Modded-NanoGPT Optimization Benchmark [4]. As the paper claims improved performance over Muon, it would be much more convincing if such a reproducible benchmarking result was included. Showing that Mano can outperform Muon and its recent variants that have been independently tuned by the optimization community would be much stronger evidence of its success.

    I encourage the authors to report such a result along with all the necessary details for reproduction. If the advantage over Muon remains I will reconsider my score.

The second weakness is that, as written in the limitations section:

"we lack a theoretical explanation for why this subspace or manifold is effective."

Although I understand that having a theoretical explanation for the performance of an optimizer is very challenging (e.g., even Muon still lacks a rigorous theoretical understanding of its success), this raises the bar for the experimental evidence, which I believe are currently not convincing enough on their own (based on the issues I mentioned above).

[1] Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus. Dodge et al., 2021.

[2] Fantastic Pretraining Optimizers and Where to Find Them, Wen et al., 2025.

[3] MUON IS SCALABLE FOR LLM TRAINING. Liu et al., 2025

[4] Modded-NanoGPT Optimization Benchmark. https://github.com/KellerJordan/modded-nanogpt/tree/master/records/track_3_optimization. Jordan 2026.

Questions for authors

Asked in the weaknesses section.

2 comments

We thank the reviewer for a detailed and technically precise review, and especially for the attention paid to our experimental configuration, which prompted several clarifications and additional measurements below.

Q1: C4 sequence length.

A1: We use sequence packing in all our experiment setups, so every sequence is filled to the full context length with concatenated documents and contains no padding; the reported batch sizes in tokens are therefore the true number of optimized tokens per step.

Q2: Token Batch Sizes.

A2: The LLaMA-350M pretraining protocol introduced by GaLore [2], and adopted unchanged by subsequent optimizer work such as APOLLO [3], uses exactly our architecture with a 131K-token batch at sequence length 256, so our setting is twice the batch and four times the context of the established academic recipe. We additionally test a 1M-token batch, where Mano continues to outperform Muon with each optimizer tuned over its own learning-rate sweep.

Tokens/stepSeq lenGlobal batch (samples)Optimizer6e-48e-41e-31.5e-32e-33e-3
1M10241024Muon––2.2792–2.26962.2752
1M10241024Mano2.2595–2.24412.2706––

Q3: Qwen learning rates.

A3: The submitted Qwen comparisons use shared learning rates and therefore establish performance under a matched configuration, not independently tuned optima. Independent sweeps were computationally infeasible within our budget, so we adopted the learning-rate transfer strategy of Liu et al. [1], where the update RMS is controlled equal.

Q4: Weight decay tuning.

A4: We thank the reviewer for noting that a shared weight decay gives Muon up to 2.5× the effective learning rate (lr × wd) of Mano. We therefore sweep Muon's weight decay over {0.1, 0.05, 0.025} across its learning rate grid, and its attainable loss if flat within 0.001, and none beat Mano, so the effective learning rate assymetry does not account for the gap between the two optimizers.

Muon wd1e-31.5e-32e-33e-3
0.12.76472.75512.75522.7778
0.052.76332.75642.75602.7794
0.0252.76322.75582.75582.7781

Q5: Auxiliary Adam settings.

A5: In our setup the auxiliary AdamW learning rate defaults to the matrix-optimizer learning rate, as the motivation proposed by Moonlight that the update RMS of muon is rescaled to the range of 0.2x in coordinance with adamw [1]. Nevertheless, we confound the reviewer's hypothesis by swapping the two AdamW learning rate between Mano and Muon, where Mano over Muon grows further with AdamW learning rate raise, and Muon gains nothing in reverse. Therefore, Mano's advantage cannot be attributed to the AdamW setting.

OptimizerMatrix LRAdamW LR 8e-4AdamW LR 1.5e-3
Muon1.5e-32.76682.7551
Mano8e-42.73872.7309

Q6: Muon time overhead.

A6: Our timing uses five unsharded Newton–Schulz iterations on every DDP rank and therefore is not a lower bound on optimized distributed Muon latency. However, on the arithmetic perspective, Mano performs no matrix multiplications and keeps a single momentum buffer, so its FLOP and memory advantage holds regardless of engineering effort, and Mano requires strictly less cross-rank communication than a distributed Muon, since its update is element-wise. The data-efficiency comparison is likewise implementation-independent: Mano reaches Muon's final validation loss in 14% fewer tokens.

Q7: Benchmarking results.

A7: We thank the reviewer for this suggestion. Within the rebuttal period, we compare Mano against three recent Muon variants: NorMuon, AdaMuon and Muon+, in an identical setting with a per-optimizer learning-rate sweep, where Mano attains the lowest validation loss. We sincerely guide the reviewer to A1 of our response to reviewer 5hu2 for the full comparison.

Q8: Lack of theoretical explanation.

A8: Mano provides a conditional theoretical mechanism, although not an end-to-end performance guarantee. Under normalized-block assumptions, the analysis identifies uniform and near-uniform amplitude changes as zero- or low-Fisher directions and characterizes when tangent projection increases the retained update’s Fisher–Rayleigh quotient. We thank the reviewer again for these constructive comments. We like to emphasize what we believe this study contributes: a new optimizer design perspective with theoretical insights and empirical validation. While limitation exists, we hope the perspective itself is contributive to the community beyond the particular optimizer we propose.

References:

[1] Liu et al. Muon is Scalable for LLM Training. 2025.

[2] Zhao et al. GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. ICML 2024.

[3] Zhu et al. APOLLO: SGD-like Memory, AdamW-level Performance. MLSys 2025.

Thank you for your response and for your efforts. Below is my follow up:

I am disappointed that the rebuttal does not address the request for results on the competitive Modded-NanoGPT Optimization Benchmark, despite emphasizing that such results would be much more convincing and that I will reconsider my score if such results were provided.

Since the paper suggests that a non-orthogonalization based optimizer outperforms Muon, which I consider to be a claim carrying considerable weight, testing on a competitive and open evaluation setting is very important. The particular benchmark is a standard choice among optimizer design papers, and incudes several base optimizer as well as newly proposed ones, for example AdamW, Muon, AdamH, MuonH, Muon^2, NorMuon, SOAP, Newton-Muon, Aurora, Shampoo, EMA-Nesterov, Muown, DynMuon. Instead, all the current results are based on internal benchmarking on the paper's own protocol, without provided code for reproducibility.

I would like the authors to clarify: Has any testing been done on the requested competitive benchmark? Even preliminary results would be acceptable. If not, could the authors justify their choice of not providing any such results? The current response does not address this at all, since the results it references are still based on the paper's internal training setup. In addition, I am not convinced that not including the requested results was due to time constraints (as implied by the response "Within the rebuttal period"); the rebuttal provide numerous additional training runs and sweeps most for a larger model (350M) and some even for extended training horizons (5x chinchilla). Instead, the requested benchmark uses a 160M model and should take considerably less time to train compared to the newly added experiments.

Also, some other comments about the rest of the rebuttal:

About the batch size: The GaLore benchmark is intended for testing memory efficient optimizers, where smaller batchsize is justified. This is evident as it also casts the model weights to BF16 precision (does not use mixed-precision), which is preferred for memory-efficient applications but is not standard in general large-scale pretraining. This raises the question if the authors also used BF16 only, including for optimizer states. Could the authors please clarify? Also, the newly provided results on 1M batchsize appear to be for a 5x Chinchilla setting, while the paper uses the Chinchilla-optimal budget. Also the budget is not provided in the response to my review, but provided to the other 2 responses. Could the authors please clarify about the training budget?

About the Qwen comparison, thanks for clarifying, but I would not consider such comparison using common learning rates informative, as different optimizers tend to have different optimal learning rate values (as also suggested by the paper's own results).

About the Muon time overhead, thanks for clarifying, however I believe it is important benchmarking to be done using the typical systems implementation of the baseline algorithms to ensure fairness. It is standard for NS calculation of Muon to be distributed across the GPUs. The paper does not consider this, so the current 10% wall-clock time overhead of Muon compared to Mano is not representative.

Finally, thanks for clarifying that the paper uses sequence packing and for the extra tuning results for LR and WD. However, the auxiliary Adam learning rate is tied to the matrix-optimizer learning rate in the main experiments. The reported ablation addresses the concern only at 350M and does not establish whether it affects the other main comparisons.

Reviewer_5hu2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Mano, a manifold-inspired optimizer for large language model pretraining. Unlike conventional manifold optimization methods that explicitly constrain model parameters and apply retractions to maintain manifold feasibility, Mano performs updates in the ambient Euclidean parameter space while projecting momentum onto an Oblique tangent space defined by the current weight matrix. It then normalizes the projected update along either the row or column dimension, alternating between the two orientations across optimization steps. Theoretically, the paper establishes a convergence result for a simplified version of Mano and analyzes the potential effect of tangent-space projection through the Fisher-Rayleigh quotient. Empirically, the authors evaluate Mano on LLaMA and Qwen3 models and report lower final validation losses than AdamW and Muon in the evaluated settings, while incurring substantially less per-step overhead than Muon.

Critical strengths
  • Mano adds dimension-wise reductions, tangent-space projection, and element-wise normalization to standard momentum updates. It avoids the Newton–Schulz matrix multiplications used by Muon and requires only one momentum state.

  • The evaluation includes LLaMA and Qwen models at several parameter scales, using C4 and The Pile. Mano achieves the lowest reported final validation loss in all settings where the relevant baselines were evaluated. These results provide encouraging preliminary evidence for the method.

Critical weaknesses
  • The main comparisons focus on AdamW and Muon. Given the normalization- and geometry-based nature of Mano, comparisons with additional relevant optimizers, such as suitable normalized or spherical optimization methods, would strengthen the evaluation. The paper also applies Mano only to internal matrix parameters while retaining AdamW for embeddings, the output head, and one-dimensional parameters; the implications of this hybrid strategy should be clarified in the baseline comparisons. Furthermore, optimizer-specific learning-rate tuning makes it difficult to determine whether the gains arise primarily from tangent projection, update normalization, or a particular combination of update scale and learning rate.

  • The reported improvements over Muon are approximately 0.014–0.0215 in final validation loss. Although potentially meaningful, the paper does not adequately report results across independent random seeds, confidence intervals, or other measures of uncertainty. This is particularly important for the axis ablations, where the difference between alternating Mano and row-wise Mano is small.

  • The mechanism behind the core design choices remains unclear. The paper reports that alternating row-wise and column-wise operations outperform certain dual-axis variants, but does not sufficiently explain why alternating orientations is beneficial while applying both projections or sequentially applying both normalizations performs worse. The authors also acknowledge that the effectiveness of neuron-wise hyperspherical geometry in LLM pretraining remains theoretically unexplained.

Questions for authors
  • Could the authors provide ablations that separately remove tangent-space projection, normalization of the projected update, and alternating orientations? Would these results clarify which component is primarily responsible for the observed improvements?

  • Could the authors report results over multiple random seeds and compare the methods under matched training FLOPs or wall-clock budgets, including the number of tokens and time required to reach a target validation loss?

  • Could the authors quantify the relationship between the reported gradient SNR and validation-loss improvement, and establish whether the metric reflects more informative gradient signals rather than changes induced by normalization?

  • Can the authors validate Mano at more high-quality dataset (such as FineWeb), larger model scales or under larger TPP?

3 comments

We sincerely thank the reviewer for recognizing both the strengths and weaknesses of our study. We attempt to address these concerns with the following statements and supplementary experiment results.

W1: Lack of comparisons with additional relevant optimizers.

A1: Under independently tuned learning rates, Mano reaches 2.7387, compared with 2.7451 for NorMuon, 2.7541 for Muon+, 2.7551 for Muon, and 2.7665 for AdaMuon. Its margin over the strongest added baseline, NorMuon, is 0.0064.

Optimizer6e-48e-41e-31.5e-32e-33e-3
Muon–2.77342.76472.75512.75522.7778
Mano2.74232.73872.7443–––
NorMuon–2.76922.75962.74882.74512.7599
AdaMuon–2.78542.77732.76652.77172.7837
Muon+––2.75642.75512.75412.7769

W2: AdamW for the embeddings, the output head, and 1D parameters should be clarified.

A2: The AdamW partition is identical for Mano and Muon: both use AdamW for embeddings, the output head, and one-dimensional parameters, while differing only on internal matrices. Therefore, the hybrid partition cannot explain Mano’s relative improvement. This partition and the matrix-update rescaling follow Liu et al. [1], approximately aligning the update RMS of the matrix optimizer and AdamW across the model.

W3/Q1: Difficult to determine whether the gains arise primarily from tangent projection, update normalization, or a particular combination of update scale and learning rate.

A3: Mano’s gain requires projection and normalization together. Their independently tuned best losses are 2.7868 and 2.8438, respectively—both worse than Muon’s 2.7551—whereas full Mano reaches 2.7387. We evaluate projection-only Mano and normalization-only row/column-norm SGD on LLaMA-350M with a separate learning-rate sweep for each variant, ensuring that the comparison uses each variant’s best rate.

Variant4e-46e-48e-41e-3
Projection-only Mano2.83842.79582.78683.3137
Normalization-only Mano2.85942.84382.88773.0240
Mano–2.74232.73872.7443

W4/Q2: Does not report results across independent random seeds or with uncertainty measures.

A4: Across three independent seeds that vary initialization and data order, Mano obtains 2.7419 ± 0.0070, compared with 2.7554 ± 0.0034 for Muon. The non-overlapping 95% confidence intervals show that the observed gap exceeds the measured run-to-run variation.

OptimizerSeed 0Seed 1Seed 2Mean ± 95% CI
Muon2.75512.75422.75692.7554 ± 0.0034
Mano2.73872.74312.74402.7419 ± 0.0070

W5: The mechanism behind the core design choices remains unclear.

A5: Mano combines two complementary operations on each selected row or column: tangent projection removes the momentum component aligned with the weight slice, while update normalization equalizes the step magnitude across slices. Their interaction is essential: after independent learning-rate tuning, projection-only and normalization-only obtain losses of 2.7868 and 2.8438, respectively, whereas alternating Mano reaches 2.7387 (Muon: 2.7551). Thus, neither component alone explains Mano’s improvement.

Row- and column-wise operations capture complementary output- and input-side structures. Applying both within one step imposes their constraints simultaneously, reducing the update’s available degrees of freedom; sequential normalization can also disturb the constraint imposed along the first axis. Alternation instead applies one simple constraint per step while covering both structures across consecutive steps. We present this as a design intuition rather than a theoretical guarantee.

Q2: Could the authors compare methods under matched training FLOPs or wall-clock budget, including the number of tokens and time to reach a target loss?

A6: Under matched budgets, Mano reaches Muon's final validation loss (2.7551) after 6.06B tokens and 6.4 h versus Muon's 7.08B tokens and 8.1 h, which is 14% fewer tokens and 22% less wall-clock. Mano is also cheaper per step while requiring neither Newton–Schulz iterations from Muon nor a second optimizer state from AdamW.

Optimizers/stepTokens to 2.7551Time to 2.7551
Muon1.087.08B8.1 h
Mano0.996.06B6.4 h

Q3: Quantify the relationship between the reported gradient SNR and validation-loss improvement.

A7: We quantify the reported gradient SNR against validation loss, where Mano attains a higher SNR than Muon at every scale, and the advantage widens with scale at 350M, 780M and 1.3B. We therefore report SNR as a consistent structural difference induced by the tangent projection, not as a quantitative predictor of the gain. Prior work interprets higher parameter-wise GSNR as stronger agreement among sample gradients and relates it to improved generalization [2]. We use this result only to motivate SNR as a diagnostic, not as evidence that SNR quantitatively predicts validation loss.

ModelMuon SNRMano SNRMuon lossMano loss
LLaMA-350M4.879.122.75512.7387
LLaMA-780M10.4716.642.51532.4960
LLaMA-1.3B7.8318.952.28782.2738

Q4: Can the authors validate Mano at more high-quality dataset, larger model scales or under larger TPP?

A8: Due to computational resource constraints and the rebuttal time limit, we are not able to ablate Mano across more diverse scenarios such as FineWeb, larger model scales, or post-training stages. We did, however, run an overtrained setup at TPP=100 (35.4B tokens, 5× Chinchilla) with the token batch increased to 1M at LLaMA-350M, where under a per-optimizer learning rate sweep Mano still outperforms Muon, suggesting that the advantage persists under data scaling.

TPPTokens/stepSeq lenGlobal batch (samples)Optimizer6e-48e-41e-31.5e-32e-33e-3
1001M10241024Muon––2.2792–2.26962.2752
1001M10241024Mano2.2595–2.24412.2706––

We are grateful to the reviewer for the care taken with our ablations in particular, which directed us toward the component and seed studies reported above and left the paper stronger than it was.

References:

[1] Liu et al. Muon is Scalable for LLM Training. 2025.

[2] Liu et al. Understanding Why Neural Networks Generalize Well Through GSNR of Parameters. ICLR 2020.

I would like to thank the authors for their rebuttal. The responses have addressed my concerns. I will increase my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.