acceptodds
Under review as a conference paper at ICLR 2027

MOSES: Geometric Preconditioning and Hybrid Quantization for Memory-Efficient LLM Optimization

Abstract

Optimizer states are the largest static memory cost in large model training. We introduce MOSES, a memory-efficient optimizer that addresses this challenge through two core innovations. First, Geometry-Aware Adaptive Preconditioning integrates gradient orthogonalization with factorized second-moment estimation, leveraging the geometry of the gradient space to provide both spatial smoothing and temporal adaptivity with negligible memory overhead. Second, Distribution-Aware Hybrid 1/4/8-bit Quantization adaptively allocates precision to post-preconditioning momentum based on the spectral properties of EMA updates, enabling aggressive compression while preserving optimization quality. Together, these innovations reduce optimizer state memory by 12.3× compared to Adam. Across model scales from 125M to 6.7B parameters, Moses achieves a compelling memory–performance trade-off that existing memory-efficient methods struggle to match, particularly excelling in the small-learning-rate regime that is characteristic of large-scale production training. Comprehensive evaluations across pretraining and downstream tasks show that Moses achieves competitive optimization quality while significantly improving memory efficiency, thereby reducing the hardware requirements for large language model training.

Primary area:
optimization
Keywords:
Memory Efficient Optimizer, Large Language Model Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YkiLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper suggest a new optimizer with a smaller memory footprint by mixed precision quantization of the optimizer states, and various other modifications.

Critical strengths
  1. This is an interesting problem.
  2. The authors make some interesting observations in figure 1.
Critical weaknesses

• Table 1 should compare with 8bit muon (e.g. "Effective Quantization of Muon Optimizer States") and 4bit muon ("MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimization", COLM 2026). This would be a more fair comparison, given both MOSES and Muon use only one additional state vector and use orthogonalization.

• The experiments scale is much too small to be convincing for a paper that suggests a new optimizer (either short models or short runs).

Questions for authors

• I was confused by the explanation "large magnitude implies low frequencies implies large quantization noise" which motivated the hybrid quantization method. It seems quite hand-wavy and not supported by any empirical evidence. I think that a simpler explanation would be that, since the quantization is done at floating point, the quantization noise has magnitude proportional to the signal magnitude (in constant to the assumptions of theoretical results). Can you disprove this?

• Peak memory improvements in table 4 are pretty small, why is this? It seems very far from the optimistic 12.3× claim.

Some more minor comments/questions:

• Not clear what happens in table 1b when hybrid quantization is removed. Everything goes to 8bit?

• Figure 1: all fonts in panels are too small

• Why is moses in bold in table 4b?

• Not clear that section 4.2.1 talks on Table 2 (it needs to be referenced).

• “This corresponds to an effective compression ratio of approximately 12.3× relative to Adam’s 64-bit two-state optimizer representation.” not sure if this the SOTA comparison. e.g. 8-bit compression of Adam is quite standard (a factor of 3×).

• Why is the definition of P line 12 in the algorithm is different from definition of P in line 213?

• In eq. 3 and 4, the notation for \mathbb{E}_i is non-standard and not defined.

• “A lightweight 2-bit metadata map M records region membership per element for efficient reconstruction.” Not clear how does it work, or how is it implemented efficiently (e.g. at what resolution are these regions partitioned? for every parameter? block? layer?).

• In figure 1 the bottom panels, what is being compared exactly? the full algorithm vs vanilla momentum SGD? make this clearer. Also, how is the comparison vs adam or muon? they are mentioned in the text (line 238), but I don't see a comparison anywhere.

• Theorem 3.1 is a re-statement pretty basic and well-known results, not sure I would call it a Theorem (this name suggests that there is some novelty claimed here).

• Corollary 3.2 is not Corollary of Theorem 3.1, as might be interpreted in the main text. Actually, it is not a corollary of Theorem A.1, since it is A.1 is modified in the proof (“Assume 1-beta_2 is unity”).

• Theorem A.1 should be referenced from some source.

• hybrid quantization is mentioned (e.g. in table 1b), but not clearly defined. Is it eq. 8?

• Why R is normalized by the mean in eq. 5 and c is not?

• In Theorem 3.1, why can we assume zero bias? Is stochastic rounding used in the quantization?

• Motivation is many times unclear: – “we orthogonalize the raw gradient Gt directly to decouple spatial normalization from temporal preconditioning”; why is this decoupling important? I guess it is because we want to better compress the momentum? this should be explained more clearly. or not explained: – “We apply orthogonalization exclusively to weight matrices in self-attention and feed-forward layers” why? common practice? – “Since orthogonalization has already smoothed the singular values of Gt, the factored approximation Rt ⊗ Ct now captures the dominant structure of pure stochastic variance rather than conflated mixture of geometry and noise” why is this true?

3 comments

Thank you for the detailed comments. We address the main concerns below.

1. Quantized Muon baselines. We added the requested baselines under the same 125M/20K protocol. MuonQ uses its officially recommended learning rate of 10−310^{-3}. All compression ratios below are relative to Adam's two FP32 optimizer states.

MethodValidation lossCompression vs. Adam
8-bit Muon2.2578×8\times
MuonQ2.312Up to 14.6×14.6\times
MOSES (0.5,2)(0.5,2)2.31312.3×12.3\times
MOSES (2,3)(2,3)2.36020.3×20.3\times

MuonQ is contemporaneous work. It achieves a slightly lower loss than default MOSES; however, MOSES can reach a more aggressive compression ratio of 20.3×20.3\times versus Adam, compared with MuonQ's maximum of 14.6×14.6\times, at the cost of some additional loss.

2. Training scale. The 125M and 350M models are trained for approximately 2.5B and 7B tokens, respectively, following the Chinchilla-optimal ratio of roughly 20 tokens per parameter. The 6.7B experiment is a large-model stability probe, and we additionally doubled its training horizon:

StepsValidation loss
10K2.395
20K2.142

The loss continues to decrease without late divergence.

3. Design motivation. Our design principle is to allocate precision according to an entry's effect on the update. Large momentum entries have greater influence and generally incur larger absolute quantization error, so they receive more bits; small entries receive fewer bits. The MOSES pipeline first shapes the momentum distribution, making this aggressive allocation less damaging.

PipelineNo quantizationHybrid quantizerGap
MOSES2.1382.142+0.2%
Muon2.1502.215+3.0%

EMA being a low-pass filter provides intuition, but does not imply a strict equivalence between magnitude and frequency. We will clarify this wording.

4. Peak GPU memory. The 12.3×12.3\times figure refers to optimizer-state compression, while peak memory also includes parameters, gradients, activations, and temporary buffers. Therefore, total peak memory cannot decrease by 12.3×12.3\times.

All GPU-memory comparisons in the paper and this response are apples-to-apples: within each comparison, all optimizers use the same model, batch size, sequence length, numerical precision, activation-checkpointing configuration.

Under the same GPT-3 13B configuration with approximately 2M tokens per update:

OptimizerPeak memory / GPU
Adam54.10 GiB
MOSES33.32 GiB

MOSES reduces actual peak GPU memory by 38.4%.

I thank the authors for the new experiments. However, I still vote for rejection, as my concerns remain:

  1. MuonQ appeared in May on Arxiv, so it can be considered prior work, according to ICLR guidelines, and the provided comparison does not convince that MOSES is significantly better, especially given the next issue.
  2. The 125M and 350M models are too small for a new optimizer paper, and the 6.7B model is still trained for a very short time.
  3. The authors admit that their theoretical motivation for the suggested method is not well connected with practical reality.

Thank you for the follow-up. We would like to clarify three points.

1. MuonQ and our contribution. You are correct that MuonQ predates our submission and should be treated as prior work. We overlooked it in the original submission, have now added the requested comparison, and will correct its positioning.

Our objective is not to establish that MOSES universally outperforms MuonQ, but to achieve a favorable memory–quality trade-off through aggressive optimizer-state compression. As reported in our previous response, default MOSES and MuonQ achieve nearly identical validation losses, while MOSES also offers a more compressed configuration at an additional quality cost. We present this as a trade-off, not as uniformly superior optimization quality.

2. Evaluation scale and training budget. We would distinguish model size from training sufficiency. Our 125M and 350M models are trained on approximately 2.5B and 7B tokens, respectively, following the Chinchilla compute-optimal allocation of approximately 20 tokens per parameter. These are completed pretraining runs at the chosen compute-optimal budgets, not short training probes.

For context, MuonQ evaluates GPT-2 Medium/Large and LLaMA 350M/1.1B. Our 125M/350M pretraining experiments and 1.3B ablations cover a broadly comparable model-scale range. We additionally provide a 6.7B pretraining probe, extended to 20K steps in our previous response, and LLaMA-3.1 8B instruction tuning in BF16 with seven downstream benchmarks.

We acknowledge that the 6.7B probe and 8B fine-tuning do not replace full-budget large-scale pretraining. Nevertheless, we believe the evaluation should be assessed as a whole: compute-optimal pretraining at smaller scales, controlled ablations, and complementary larger-model validation support the proposed memory–quality trade-off within the evaluated settings.

3. Connection between motivation and practice. Our clarification was that momentum magnitude and temporal frequency are not strictly equivalent—not that our method lacks a practical basis. We agree that EMA’s low-pass property alone does not prove the optimality of magnitude-based precision allocation. The central design premise, however, is that the MOSES pipeline shapes momentum into a distribution more amenable to aggressive quantization.

This premise is supported by the measured distribution changes in Figure 1 and the matched quantization ablation in our previous response: the same hybrid quantizer produces substantially less degradation with MOSES than with Muon. Thus, qualifying the frequency-based intuition does not negate the empirical support for the optimizer–quantizer co-design. We will revise the explanation to distinguish the theoretical intuition, its assumptions, and the empirical evidence more clearly.

We hope these clarifications help address your remaining concerns.

Reviewer_SeRrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposed MOSES, a memory-efficient optimizer for LLM training, which combines Newton-Schulz gradient orthogonalization with factorized second-moment estimation for improving the gradient updates. It applies a hybrid 1/4/8-bit quantization scheme to momentum states based on their magnitudes. Lots of experiments on GPT models and LLaMA models show that MOSES achieves competitive performance while reducing memory by 12.3x compared to Adam.

Critical strengths
  • (S1) The paper combines gradient orthogonalization, factorized second-moment estimation, and hybrid-precision quantization into a unified optimizer. The overall design is reasonable and offers a practical approach to reducing optimizer-state memory.

  • (S2) Extensive experiments across model scales, pre-training and fine-tuning tasks show MOSES achieves significant memory savings and competitive performance against strong baselines. Evaluations are based on real GPU memory and training overhead, with only small extra compute cost.

Critical weaknesses
  • (W1) Some relevant baselines, like Adam-mni, SWAN, and Apollo-mini, are omitted. The 6.7B pre-training experiment ran for only 10K steps, with a training dataset of approximately 0.66B tokens, which is insufficient to demonstrate convergence stability during long-term training. And MOSES underperforms Muon in this setting. It remains unknown whether quantization error accumulates over extended training.

  • (W2) Theorem 3.1 assumes that quantization errors have a mean of zero and are independent of one another; however, the paper does not prove that the 1/4/8-bit hybrid quantization method actually used satisfies these assumptions. And because MOSES was based on the low-pass filtering properties of EMA, the authors assume that higher precision should be allocated to large-amplitude momentum; however, the relationship between momentum amplitude and low-frequency information remains unverified.

Questions for authors
  • (Q1) Compared with strong related works. The authors compared MOSES with Adam-8bit/4bit, Adafactor, and Muon. However, they should also compare with related works such as Adam-mni, SWAN, and Apollo-mini. Could the authors add more experiments with those baselines.

  • (Q2) Could the authors provide a statistical analysis of the actual quantization error, as I mentioned in W2?

  • (Q3) It's unclear that the relationship between momentum amplitude and low-frequency information remains unverified. Could the authors add some experiments to show the relationship between momentum amplitude and frequency characteristics?

  • (Q4) Based on W1. Could the authors scale up the training tokens and iterations to observe whether quantization error affects convergence as training progresses?

3 comments

Thank you for the constructive comments. We added matched experiments for the missing baselines, extended the 6.7B training, and measured the actual quantization error.

1. Additional baselines. We ran new 125M/20K experiments using the same model, data, seed, token budget, and evaluation protocol as Table 2. Adam-mini and APOLLO-Mini use the same five-point learning-rate search as the main paper. For SWAN, we follow the paper's global learning-rate grid {0.01,0.02,0.05}\{0.01, 0.02, 0.05\}.

OptimizerCompression vs. AdamValidation lossBest global LR
Adam-mini∼2.0×\sim 2.0\times2.28910−310^{-3}
MOSES12.3×12.3\times2.3135×10−45\times10^{-4}
SWANState-free2.32310−210^{-2}
APOLLO-Mini∼768×\sim 768\times2.33910−210^{-2}

These results clarify the trade-off rather than establishing universal dominance. Adam-mini obtains a slightly lower loss with substantially more optimizer state than MOSES. SWAN and APOLLO-Mini use less matrix-state memory, while MOSES obtains lower validation loss under the matched protocol.

2. Longer training. We extended the same 6.7B MOSES run from 10K to 20K steps.

StepsValidation loss
10K2.395
20K2.142

The loss continues to decrease without late divergence. This addresses whether quantization error causes instability over the doubled horizon. We agree that this is still not Chinchilla-scale training due to limited computational budget, and will describe it as a stability test.

3.Quantization error. We test with a full 125M/20K run and sampled four representative momentum matrices throughout training. The normalized mean error has a magnitude of only 1.8×10−51.8\times10^{-5}, the cosine similarity between the original and reconstructed momentum is 0.99420.9942, and the error correlation between consecutive audit points is −9.1×10−5-9.1\times10^{-5}.

These results show negligible aggregate bias, strong preservation of the momentum direction, and no persistent temporal correlation at the sampled interval. Nevertheless, because our quantizer uses deterministic rounding, we agree that zero-mean independent errors are not guaranteed theoretically. We will therefore present our theorem as a conditional error-dynamics result rather than as an unconditional property of the quantizer.

4. Magnitude and frequency. EMA is a low-pass filter, so, for components with comparable input amplitudes, persistent low-frequency components are preferentially retained and tend to contribute more strongly to the momentum magnitude. So magnitude is a practical proxy rather than a frequency classifier.

Magnitude-based method also directly controls distortion: larger momentum entries have a greater effect on the update and therefore receive more bits. The reconstructed momentum preserves the original direction well, with a cosine similarity of 0.99420.9942. We will clarify this in the revision.

Thanks for the author's detailed responses. My concerns have been address, and I rise my score to weak aeecpted.

Thank you for your feedback and updated assessment. We sincerely appreciate your time and consideration.

Reviewer_Mb2uOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on memory-efficient LLM training, and in particular optimizer state compression. It proposes a new optimizer called MOSES relying on Newton Schulz orthogonalization of the gradient to reduce spectral ill-conditioning and a factorized second-momentum estimation similar to Adafactor, as well as a 1/4/8-bit hybrid quantization scheme for the momentum. The main experiments of the paper include LLM pretraining (GPT-3 125M and 350M trained at Chinchilla tokens, and a short run at 6.7B model size) and finetuning, suggesting competitive performance with high state compression rates compared to Adam.

Critical strengths

Importance of the topic and accurate experimental comparisons. I find the topic of memory-efficient LLM training that the paper is addressing being of high importance, as the increasing sizes of LLMs and the reliance on memory-heavy optimizers impose a significant memory cost. Also, I find the paper's main pretraining comparisions reliable, as the authors performed learning rate tuning for all baselines and train up to Chinchilla optimal number of tokens (at least for the 125M and 350M model sizes). The slightly higher loss reported with MOSES compared to Adam/Muon is expected given the compression and I appreciate the authors for reporting those results. Unfortunately this should not be taken for granted, as there are several similar works that report unrealistic results including much lower losses or 2-3x speedups (in terms of tokens needed to achieve the same loss) compared to full rank Adam.

Critical weaknesses

Limited peak GPU memory reduction.

While the paper emphasizes the 12x reduction in optimizer state memory compared to Adam, in practice peak GPU determines the minimum required memory for training. As the paper reports in Table 4, that reduction is much more limited, being around 20% compared to Adam and 10% compared to Adafactor/Muon for the 1.3B model using the (more practical for LLM pretraining) BS 64. I expect the memory saving to be even smaller for larger token batchsizes, which are common in pretraining.

Also, I find wording in the conclusion "without compromising convergence" not entirely accurate, as based on Table 2 MOSES achieves higher loss at the same token count as the full state baselines, suggesting a convergence slowdown, and even small differences in loss can result in non negligible token-equivalent slowdowns [1]. Especially the 6.7B model results, which tests performance at a more practical model size, give a loss 2.28 for Muon and 2.39 for MOSES, being a significant loss difference. Therefore, the (10-20%) memory savings need to be weighted against the potential performance degradation.

Novelty concern. The paper mostly reuses known techniques and composes them together. The Newton Schulz operation is already introduced in Muon, the factorized second order states and the RMS based clipping are from Adafactor, the learning rate scaling is from the Moonshot version of Muon and the quantization of momentum is a commonly used in relevant literature (e.g., [2-4]). As a result, the proposed optimizer seems like an ensemble of multiple known techniques, rather that an optimizer with based on some central novel idea. This itself is not a reason for rejection but somehow reduces the novelty of the paper in my opinion.

[1] Fantastic Pretraining Optimizers and Where to Find Them, Wen et al., 2025.

[2] Memory Efficient Optimizers with 4-bit States, Li et al., 2023.

[3] 8-bit Optimizers via Block-wise Quantization, Dettmers et al., 2021.

[4] Effective Quantization of Muon Optimizer States, Gupta et al., 2025.

Questions for authors

Table 4 (a) reports peak memory for up to BS 64, however as I understand the main results are with BS 128. Could the authors report the peak memory usage for BS 128 as well? Also the runtime comparison is for a 13B model with global batch size 2M tokens, could the authors report the peak memory usage for that case too? Also, is activation checkpointing enabled for the memory estimates in Table 4, and is so what would be the results without it?

Appendix D mentions that training is conducted in fp16. Could the authors provide more details about the precision scheme used? Does this mean that the Adam/Muon optimizer states also use fp16, or is a mixed-precision scheme followed? Also more commonly pretraining is performed in BF16 mixed-precision (for example using torch automatic mixed precision), could the authors elaborate why they choose fp16 training and if possible to include results with BF16 mixed-precision?

9 comments

Thank you for the comments. We clarify three points below.

1. Peak GPU memory. The 12.3×12.3\times number refers to optimizer-state memory, while Table 4 reports the peak GPU memory of the actual implementation. Peak memory also includes parameters, gradients, activations, and temporary buffers:

Mpeak=Mcommon+Moptimizer.M_{\mathrm{peak}} = M_{\mathrm{common}} + M_{\mathrm{optimizer}}.

Therefore, the percentage reduction in peak memory naturally changes with batch size. This does not contradict the 12.3×12.3\times optimizer-state compression. All methods in Table 4 are measured under the same configuration, so the comparison is apples-to-apples.

There is also a notation ambiguity: BS in Table 4 is the per-rank micro-batch size. With eight data-parallel GPUs, BS=64 corresponds to a global batch of 512 sequences, already larger than the main 125M/350M training batch.

For the requested GPT-3 13B configuration with approximately 2M tokens per update, the measured peak memory is:

OptimizerPeak memory / GPU
Adam54.10 GiB
MOSES33.32 GiB

This is a 38.4% reduction in actual peak GPU memory. Activation checkpointing is enabled for all methods. Disabling it only increases the common activation term and does not change the optimizer-state compression ratio.

2. Novelty. We do not claim that Newton--Schulz orthogonalization, factorized second moments, or momentum quantization are individually new. The contribution is the joint pipeline designed to make momentum aggressively quantizable with minimal quality loss. This interaction is discussed in the method and analysis sections.

A matched 1.3B experiment makes this point directly:

PipelineNo quantizationSame hybrid quantizerQuantization gap
MOSES2.1382.142+0.2%
Muon2.1502.215+3.0%

Thus, applying the same quantizer to Muon produces a much larger degradation. The contribution is the optimizer--quantizer co-design, rather than any individual component in isolation.

3. Precision. FP16 is used for model weights and forward/backward computation. The accumulated optimizer states of the full-precision baselines remain FP32, as in standard mixed-precision training. Adam therefore stores two FP32 moment tensors; full-precision Muon stores its momentum in FP32. MOSES stores the factorized second-moment vectors in FP32 and the matrix momentum in the packed 1/4/8-bit representation. The 12.3×12.3\times ratio is calculated against Adam's two FP32 optimizer states. Using BF16 for forward/backward computation would not change this comparison.

Thank you for your response. It resolves some of my questions but important weaknesses remain unanswered. I elaborate below.

Resolved: Thanks for clarifying about the BS in Table 4, the optimizer state precision and for providing a peak-memory comparison with Adam for the 13B model.

Unresolved:

  1. The most important remaining weakness. As I mentioned in my review, MOSES results in a non-negligible loss degradation compared to Muon when trained for the same token budget. This suggests reduced token efficiency and the additional tokens and time required to match Muon's loss remain unquantified. This combined with the extra overhead should be weighed against the potential memory savings to Muon, which are around 20% given the current single node config and activation checkpointing. The peak-memory saving might enable slightly larger batch size, but the total training time to reach the same loss target is a combination of all above factors. The paper currently provides no such comparison and the rebuttal does not address this directly.

  2. The 13B peak-memory currently only compares to Adam. I find it more suitable to compare to Muon, as it is the strongest baseline and most related to MOSES. I would estimate the reduction to drop to around 20% in that case. Also I think it is important to note that those numbers are only accurate for the current, limited-scale, 1 node GPU setup. Typical multi-node pretraining shards optimizer states across ranks, reducing the absolute per-GPU benefit of optimizer-state compression. The percentage reduction in peak memory thus is expected to be smaller in such larger-scale setups.

  3. I understand that disabling activation checkpointing increases the common activation memory. My point was that activation checkpointing is not universal among pretraining setups (e.g., [5] explicitly states disabling it) and without it the reduction in peak-memory achieved from MOSES could be smaller in percentage. A complete study should include this case.

  4. The clarification about FP32 optimizer states is helpful. However BF16 mixed precision is more representative for modern large-scale pretraining. The response does not justify the choice of FP16 over BF16 and does not establish that the current loss comparisons also hold under BF16. Unchanged optimizer-state storage does not resolve this question.

[5] The Llama 3 Herd of Models. Llama Team, 2024.

Thank you for your follow-up and for clarifying the remaining concerns. Our objective is not to outperform full-precision Muon in unconstrained optimization quality, but to achieve a favorable memory–quality trade-off through aggressive optimizer-state compression. The fixed-token loss gap should therefore be weighed against the storage savings, rather than considered in isolation.

1. Memory. We now provide the requested Muon comparison for the same 13B configuration with approximately 2M tokens per update and activation checkpointing enabled:

OptimizerPeak memory / GPU
Adam54.10 GiB
Muon42.83 GiB
MOSES33.32 GiB

MOSES reduces measured peak memory by 22.2% relative to Muon and 38.4% relative to Adam.

Importantly, the optimizer-state storage advantage follows directly from the representation. Section 3.2.3 explicitly includes the 2-bit metadata map and block scales, giving approximately 5.19 bits per quantized momentum element under the paper’s occupancy estimate, versus Muon’s 32 bits—approximately 6.2× compression of that component, with auxiliary states accounted for separately. The peak-memory experiments illustrate that our implementation translates this analytical storage advantage into actual system-level savings, now verified against both Adam and Muon.

Disabling activation checkpointing increases the common activation footprint, while uniform optimizer-state sharding reduces both methods’ per-rank state storage. These changes affect the magnitude and percentage of the total-memory benefit, but do not eliminate the underlying state-storage advantage under matched configurations. We will clarify that the reported peak-memory percentages are configuration-specific, whereas the state-storage reduction follows from explicit byte accounting.

2. Runtime and convergence. Table 4(b) already provides the relevant runtime comparison: MOSES takes 92.95 s/step versus Muon’s 92.82 s/step, approximately 0.14% additional per-step cost in the evaluated 13B configuration. Section 4.3 also analyzes how optimizer overhead is amortized as tokens per update increase. This does not establish equal or better time-to-target: the additional tokens required to match Muon’s loss remain unquantified. Our claim is a fixed-budget memory–quality trade-off, not time-to-target superiority over Muon. We will revise related wording accordingly.

3. Precision. All pretraining methods are evaluated under the same FP16 mixed-precision setup, so the reported comparisons are controlled and apples-to-apples. Our evaluation is also not limited to FP16: the LLaMA-3.1 8B instruction-tuning experiments underlying the downstream results in Table 3 use BF16, as specified in Appendix D. These provide evidence of MOSES’s effectiveness under BF16 in fine-tuning, although they do not establish identical pretraining loss comparisons across precisions. Moreover, the optimizer-state storage accounting remains applicable with BF16 forward/backward computation, provided the state representations are unchanged.

We hope that these clarifications and the additional Muon memory comparison help address your remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.