- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Quantizing Adam's states can save storage, but will introduce error propagating through momentum computation. Existing method, TorchAO, pointed out that remove 0 from the codebook of quantization can preventing excessively large update introduced by small positive second-momentum which are rounded to 0. The core idea of this paper is that the zero-point failure depends critically on the rounding rule, or optimizer-state quantizers should account for how stored states affect optimization, not solely for reconstruction error in state space. This paper defines rounding decisions in preconditioner space, reduces the probability of harmful zero.
Critical strengths
- The rounding coordinate controls the zero-point failure on a fixed zero-inclusive codebook. Through analysis, the paper proposes that optimizer-state quantizers should account for how stored states affect optimization, not solely for reconstruction error in state space. This is a novel aspect for Adam's state quantization.
Critical weaknesses
- The final-10% LM-head rule is an empirical patch whose cause leaves unresolved. In GPT-style 2.7B pretraining, NF4 round-to-nearest on the first moment departs from 32-bit AdamW during learning-rate cooldown, whereas NF4 stochastic rounding stays stable. Over the same continuations, NF4-RTN produces a persistently negative LM-head radial bias, whereas the bias under NF4-SR stays near zero. The conclusion states that this targeted stochastic rounding alleviates the observed late-training regression, but that the cause of the directional first-moment bias, and its possible relationship to learning-rate decay, remain unresolved.
Questions for authors
- Analysis on Muon may make the work complete. Muon keeps a first-order momentum state with the same shape as the weight, and quantized Muon is already cited in the related work. The paper’s central claim is that optimizer-state quantization must account for how a stored state enters the update, not only for reconstruction error, and the conclusion says the methodology should be investigated for other adaptive optimizers. It would make the paper complete if Muon is also included in both the rounding-coordinate analysis and the experiments.
3 comments
Thank you for the response. I would keep my rating at 4.
However, I find this year’s four-point scale too coarse to convey that judgment. Hearing that 2/3 of the papers this year are rated 2, I think 4 conveys somewhat more support than I intend: I would support the acceptance of this paper, but do not think that it can be highlighted as spotlight or oral at the conference. But lowering the score to 3 is not fair for the authors. I'd appreciate it if the AC would consider my opinion carefully.
Thank you for taking the time to review our manuscript and read our response. We appreciate your feedback throughout the review and your support for acceptance!
Response to Reviewer ULqc
Thank you for the thoughtful comments. We address the weaknesses and questions below.
Q1: Cause of the LM-head first-moment radial bias and its relationship to learning-rate decay
In the revised Section 4.2 (“Rounding rule for the first moment”), we expand the discussion following Figure 4(b) to explain a possible contributing mechanism. With slowly varying gradients and block scales, RTN can repeatedly discard small adjustments to the stored momentum [1]. When these adjustments would increase the momentum magnitude, discarding them produces inward bias. SR preserves these adjustments in expectation. We also discuss a possible connection to cooldown: smaller parameter changes may allow similar gradient signals to persist across successive steps.
For illustration, consider scalar momentum descent on f(w)=w2/2, with a unit preconditioner, momentum coefficient 0.9, and initialization w0=1, q0=0. The update is
w_t=w_{t-1}-\eta_t\widetilde m_t,\qquad q_t=Q(\widetilde m_t).$$ We decrease the learning rate linearly from $0.10$ to $0.01$ over 50 steps, using $\eta_t=0.10-0.09(t-1)/49$. FP32 stores the working moment directly; RTN and SR use the fixed codebook $\{-1,0,1\}$. SR results are mean (sample SD) over ten runs. Momentum denotes the stored state after rounding. | Step | LR | RTN momentum | FP32 momentum | SR momentum | RTN iterate | FP32 iterate | SR iterate | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | 1 | 0.100 | 0 | 0.10 | 0.10 (0.32) | 0.99 | 0.99 | 0.99 (0.00) | | 10 | 0.083 | 0 | 0.55 | 0.40 (0.52) | 0.91 | 0.66 | 0.58 (0.29) | | 20 | 0.065 | 0 | 0.47 | 0.50 (0.53) | 0.85 | 0.26 | 0.26 (0.50) | | 30 | 0.047 | 0 | 0.26 | 0.40 (0.52) | 0.80 | 0.06 | 0.06 (0.49) | | 50 | 0.010 | 0 | 0.03 | 0.00 (0.47) | 0.76 | -0.01 | -0.03 (0.29) | RTN stores zero throughout, so its working moment is only $0.1w_{t-1}$. Its iterates therefore follow $w_t=(1-0.1\eta_t)w_{t-1}$, repeatedly losing the gradient history that FP32 retains. SR allows small adjustments to change the stored momentum, although this coarse codebook produces substantial variation between trajectories. The mean final objective value is 0.038 under SR, compared with 0.288 under RTN. This example illustrates how momentum stalling can slow optimization during cooldown. Table 12 in Appendix C.5 provides the empirical comparison. With non-head first moments kept in FP32 and second moments using Dyn4 Update-SR, replacing only LM-head NF4-RTN with NF4-SR reduces the final validation loss from 2.579 to 2.440. Why the inward bias dominates specifically in the LM-head, and whether learning-rate decay triggers its harmful effects, remain open questions, as stated in the “Limitations and future work” paragraph of Section 6. **Q2: Extension of the rounding-coordinate analysis and experiments to Muon** We agree that Muon is a useful direction for extending this perspective. Existing work already studies how quantization interacts with its update: Gupta et al. [2] investigate 8-bit blockwise momentum quantization; 4-bit-Muon-GRASP [3] combines grid quantization with preservation of the dominant singular subspace; and MuonQ [4] uses normalization, structural decomposition, and nonuniform quantization to preserve update directions. For Muon, a possible extension would choose rounding or calibration to preserve the update after momentum orthogonalization. Under exact orthogonalization, changing positive singular values while keeping the singular vectors fixed leaves the update unchanged, whereas errors in those vectors can change its direction. This suggests accounting for the effect of quantization on the orthogonalized update when choosing how to store the momentum. The difficulty is that orthogonalization couples entries across the momentum matrix, so our scalar construction for AdamW does not directly apply. Unbiased momentum also need not produce an unbiased update after this nonlinear transformation. Designing an efficient rule would require accounting for this matrix structure and the accumulation of errors through the momentum recurrence. We leave this extension to future work. **References** [1] Kristi Topollai and Anna Ewa Choromanska. *Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resets*. EMNLP, 2026. [2] Aman Gupta et al. *On Quantizing the State of the Muon Optimizer*. SPOT Workshop at ICLR, 2026. [3] Huaijin Wu et al. *Achieving low-bit Muon through subspace preservation and grid quantization*. ICLR, 2026. [4] Yupeng Su et al. *MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimization*. COLM, 2026.