- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper suggest a new optimizer with a smaller memory footprint by mixed precision quantization of the optimizer states, and various other modifications.
Critical strengths
- This is an interesting problem.
- The authors make some interesting observations in figure 1.
Critical weaknesses
• Table 1 should compare with 8bit muon (e.g. "Effective Quantization of Muon Optimizer States") and 4bit muon ("MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimization", COLM 2026). This would be a more fair comparison, given both MOSES and Muon use only one additional state vector and use orthogonalization.
• The experiments scale is much too small to be convincing for a paper that suggests a new optimizer (either short models or short runs).
Questions for authors
• I was confused by the explanation "large magnitude implies low frequencies implies large quantization noise" which motivated the hybrid quantization method. It seems quite hand-wavy and not supported by any empirical evidence. I think that a simpler explanation would be that, since the quantization is done at floating point, the quantization noise has magnitude proportional to the signal magnitude (in constant to the assumptions of theoretical results). Can you disprove this?
• Peak memory improvements in table 4 are pretty small, why is this? It seems very far from the optimistic 12.3× claim.
Some more minor comments/questions:
• Not clear what happens in table 1b when hybrid quantization is removed. Everything goes to 8bit?
• Figure 1: all fonts in panels are too small
• Why is moses in bold in table 4b?
• Not clear that section 4.2.1 talks on Table 2 (it needs to be referenced).
• “This corresponds to an effective compression ratio of approximately 12.3× relative to Adam’s 64-bit two-state optimizer representation.” not sure if this the SOTA comparison. e.g. 8-bit compression of Adam is quite standard (a factor of 3×).
• Why is the definition of P line 12 in the algorithm is different from definition of P in line 213?
• In eq. 3 and 4, the notation for \mathbb{E}_i is non-standard and not defined.
• “A lightweight 2-bit metadata map M records region membership per element for efficient reconstruction.” Not clear how does it work, or how is it implemented efficiently (e.g. at what resolution are these regions partitioned? for every parameter? block? layer?).
• In figure 1 the bottom panels, what is being compared exactly? the full algorithm vs vanilla momentum SGD? make this clearer. Also, how is the comparison vs adam or muon? they are mentioned in the text (line 238), but I don't see a comparison anywhere.
• Theorem 3.1 is a re-statement pretty basic and well-known results, not sure I would call it a Theorem (this name suggests that there is some novelty claimed here).
• Corollary 3.2 is not Corollary of Theorem 3.1, as might be interpreted in the main text. Actually, it is not a corollary of Theorem A.1, since it is A.1 is modified in the proof (“Assume 1-beta_2 is unity”).
• Theorem A.1 should be referenced from some source.
• hybrid quantization is mentioned (e.g. in table 1b), but not clearly defined. Is it eq. 8?
• Why R is normalized by the mean in eq. 5 and c is not?
• In Theorem 3.1, why can we assume zero bias? Is stochastic rounding used in the quantization?
• Motivation is many times unclear: – “we orthogonalize the raw gradient Gt directly to decouple spatial normalization from temporal preconditioning”; why is this decoupling important? I guess it is because we want to better compress the momentum? this should be explained more clearly. or not explained: – “We apply orthogonalization exclusively to weight matrices in self-attention and feed-forward layers” why? common practice? – “Since orthogonalization has already smoothed the singular values of Gt, the factored approximation Rt ⊗ Ct now captures the dominant structure of pure stochastic variance rather than conflated mixture of geometry and noise” why is this true?
3 comments
I thank the authors for the new experiments. However, I still vote for rejection, as my concerns remain:
- MuonQ appeared in May on Arxiv, so it can be considered prior work, according to ICLR guidelines, and the provided comparison does not convince that MOSES is significantly better, especially given the next issue.
- The 125M and 350M models are too small for a new optimizer paper, and the 6.7B model is still trained for a very short time.
- The authors admit that their theoretical motivation for the suggested method is not well connected with practical reality.
Thank you for the follow-up. We would like to clarify three points.
1. MuonQ and our contribution. You are correct that MuonQ predates our submission and should be treated as prior work. We overlooked it in the original submission, have now added the requested comparison, and will correct its positioning.
Our objective is not to establish that MOSES universally outperforms MuonQ, but to achieve a favorable memory–quality trade-off through aggressive optimizer-state compression. As reported in our previous response, default MOSES and MuonQ achieve nearly identical validation losses, while MOSES also offers a more compressed configuration at an additional quality cost. We present this as a trade-off, not as uniformly superior optimization quality.
2. Evaluation scale and training budget. We would distinguish model size from training sufficiency. Our 125M and 350M models are trained on approximately 2.5B and 7B tokens, respectively, following the Chinchilla compute-optimal allocation of approximately 20 tokens per parameter. These are completed pretraining runs at the chosen compute-optimal budgets, not short training probes.
For context, MuonQ evaluates GPT-2 Medium/Large and LLaMA 350M/1.1B. Our 125M/350M pretraining experiments and 1.3B ablations cover a broadly comparable model-scale range. We additionally provide a 6.7B pretraining probe, extended to 20K steps in our previous response, and LLaMA-3.1 8B instruction tuning in BF16 with seven downstream benchmarks.
We acknowledge that the 6.7B probe and 8B fine-tuning do not replace full-budget large-scale pretraining. Nevertheless, we believe the evaluation should be assessed as a whole: compute-optimal pretraining at smaller scales, controlled ablations, and complementary larger-model validation support the proposed memory–quality trade-off within the evaluated settings.
3. Connection between motivation and practice. Our clarification was that momentum magnitude and temporal frequency are not strictly equivalent—not that our method lacks a practical basis. We agree that EMA’s low-pass property alone does not prove the optimality of magnitude-based precision allocation. The central design premise, however, is that the MOSES pipeline shapes momentum into a distribution more amenable to aggressive quantization.
This premise is supported by the measured distribution changes in Figure 1 and the matched quantization ablation in our previous response: the same hybrid quantizer produces substantially less degradation with MOSES than with Muon. Thus, qualifying the frequency-based intuition does not negate the empirical support for the optimizer–quantizer co-design. We will revise the explanation to distinguish the theoretical intuition, its assumptions, and the empirical evidence more clearly.
We hope these clarifications help address your remaining concerns.
Thank you for the detailed comments. We address the main concerns below.
1. Quantized Muon baselines. We added the requested baselines under the same 125M/20K protocol. MuonQ uses its officially recommended learning rate of 10−3. All compression ratios below are relative to Adam's two FP32 optimizer states.
MuonQ is contemporaneous work. It achieves a slightly lower loss than default MOSES; however, MOSES can reach a more aggressive compression ratio of 20.3× versus Adam, compared with MuonQ's maximum of 14.6×, at the cost of some additional loss.
2. Training scale. The 125M and 350M models are trained for approximately 2.5B and 7B tokens, respectively, following the Chinchilla-optimal ratio of roughly 20 tokens per parameter. The 6.7B experiment is a large-model stability probe, and we additionally doubled its training horizon:
The loss continues to decrease without late divergence.
3. Design motivation. Our design principle is to allocate precision according to an entry's effect on the update. Large momentum entries have greater influence and generally incur larger absolute quantization error, so they receive more bits; small entries receive fewer bits. The MOSES pipeline first shapes the momentum distribution, making this aggressive allocation less damaging.
EMA being a low-pass filter provides intuition, but does not imply a strict equivalence between magnitude and frequency. We will clarify this wording.
4. Peak GPU memory. The 12.3× figure refers to optimizer-state compression, while peak memory also includes parameters, gradients, activations, and temporary buffers. Therefore, total peak memory cannot decrease by 12.3×.
All GPU-memory comparisons in the paper and this response are apples-to-apples: within each comparison, all optimizers use the same model, batch size, sequence length, numerical precision, activation-checkpointing configuration.
Under the same GPT-3 13B configuration with approximately 2M tokens per update:
MOSES reduces actual peak GPU memory by 38.4%.