acceptodds
Under review as a conference paper at ICLR 2027

Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization

Abstract

Quantizing AdamW's optimizer states reduces persistent storage, but quantization errors propagate through the moment recurrences and perturb subsequent adaptive updates. We redesign 4-bit optimizer-state quantization for AdamW from the perspective of rounding space: the coordinate in which a quantizer chooses between adjacent reconstruction levels. For the second-moment, a local analysis of the quantization cell adjacent to zero shows that small mean state error does not ensure a small mean error in the next-step preconditioner. A one-dimensional quadratic construction further shows qualitatively different optimization dynamics under state-space and preconditioner-space rounding. These results motivate Zero-Inclusive Preconditioner-space Stochastic Rounding (ZIP-SR), which retains zero in the second-moment codebook and computes stochastic-rounding probabilities in preconditioner space. As a complementary route, Zero-Excluding EDEN calibration (ZE-EDEN) uses a zero-excluding second-moment codebook and rescales the quantized second-moment block to mitigate the preconditioner distortion caused by the positive quantization floor. Both configurations use 4-bit NormalFloat (NF4) for the first moment, with targeted stochastic rounding of the LM-head first moment during the final 10% of training. Across GPT- and Llama-style pretraining experiments ranging from 130M to 2.7B parameters, both methods reduce TorchAO 4-bit AdamW's validation-loss gap to 32-bit AdamW at every evaluated scale, with the largest reported gap reduction reaching 70%. In full-parameter supervised fine-tuning, both recipes achieve lower validation loss than TorchAO while remaining close to 32-bit AdamW on downstream tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Optimizer-state quantization, AdamW, memory-efficient optimizers, low-precision training, large language models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ULqcOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Quantizing Adam's states can save storage, but will introduce error propagating through momentum computation. Existing method, TorchAO, pointed out that remove 0 from the codebook of quantization can preventing excessively large update introduced by small positive second-momentum which are rounded to 0. The core idea of this paper is that the zero-point failure depends critically on the rounding rule, or optimizer-state quantizers should account for how stored states affect optimization, not solely for reconstruction error in state space. This paper defines rounding decisions in preconditioner space, reduces the probability of harmful zero.

Critical strengths
  1. The rounding coordinate controls the zero-point failure on a fixed zero-inclusive codebook. Through analysis, the paper proposes that optimizer-state quantizers should account for how stored states affect optimization, not solely for reconstruction error in state space. This is a novel aspect for Adam's state quantization.
Critical weaknesses
  1. The final-10% LM-head rule is an empirical patch whose cause leaves unresolved. In GPT-style 2.7B pretraining, NF4 round-to-nearest on the first moment departs from 32-bit AdamW during learning-rate cooldown, whereas NF4 stochastic rounding stays stable. Over the same continuations, NF4-RTN produces a persistently negative LM-head radial bias, whereas the bias under NF4-SR stays near zero. The conclusion states that this targeted stochastic rounding alleviates the observed late-training regression, but that the cause of the directional first-moment bias, and its possible relationship to learning-rate decay, remain unresolved.
Questions for authors
  1. Analysis on Muon may make the work complete. Muon keeps a first-order momentum state with the same shape as the weight, and quantized Muon is already cited in the related work. The paper’s central claim is that optimizer-state quantization must account for how a stored state enters the update, not only for reconstruction error, and the conclusion says the methodology should be investigated for other adaptive optimizers. It would make the paper complete if Muon is also included in both the rounding-coordinate analysis and the experiments.
3 comments
AuthorsOpenReview ↗

Response to Reviewer ULqc

Thank you for the thoughtful comments. We address the weaknesses and questions below.

Q1: Cause of the LM-head first-moment radial bias and its relationship to learning-rate decay

In the revised Section 4.2 (“Rounding rule for the first moment”), we expand the discussion following Figure 4(b) to explain a possible contributing mechanism. With slowly varying gradients and block scales, RTN can repeatedly discard small adjustments to the stored momentum [1]. When these adjustments would increase the momentum magnitude, discarding them produces inward bias. SR preserves these adjustments in expectation. We also discuss a possible connection to cooldown: smaller parameter changes may allow similar gradient signals to persist across successive steps.

For illustration, consider scalar momentum descent on f(w)=w2/2f(w)=w^2/2, with a unit preconditioner, momentum coefficient 0.90.9, and initialization w0=1, q0=0w_0=1,\ q_0=0. The update is

w_t=w_{t-1}-\eta_t\widetilde m_t,\qquad q_t=Q(\widetilde m_t).$$ We decrease the learning rate linearly from $0.10$ to $0.01$ over 50 steps, using $\eta_t=0.10-0.09(t-1)/49$. FP32 stores the working moment directly; RTN and SR use the fixed codebook $\{-1,0,1\}$. SR results are mean (sample SD) over ten runs. Momentum denotes the stored state after rounding. | Step | LR | RTN momentum | FP32 momentum | SR momentum | RTN iterate | FP32 iterate | SR iterate | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | 1 | 0.100 | 0 | 0.10 | 0.10 (0.32) | 0.99 | 0.99 | 0.99 (0.00) | | 10 | 0.083 | 0 | 0.55 | 0.40 (0.52) | 0.91 | 0.66 | 0.58 (0.29) | | 20 | 0.065 | 0 | 0.47 | 0.50 (0.53) | 0.85 | 0.26 | 0.26 (0.50) | | 30 | 0.047 | 0 | 0.26 | 0.40 (0.52) | 0.80 | 0.06 | 0.06 (0.49) | | 50 | 0.010 | 0 | 0.03 | 0.00 (0.47) | 0.76 | -0.01 | -0.03 (0.29) | RTN stores zero throughout, so its working moment is only $0.1w_{t-1}$. Its iterates therefore follow $w_t=(1-0.1\eta_t)w_{t-1}$, repeatedly losing the gradient history that FP32 retains. SR allows small adjustments to change the stored momentum, although this coarse codebook produces substantial variation between trajectories. The mean final objective value is 0.038 under SR, compared with 0.288 under RTN. This example illustrates how momentum stalling can slow optimization during cooldown. Table 12 in Appendix C.5 provides the empirical comparison. With non-head first moments kept in FP32 and second moments using Dyn4 Update-SR, replacing only LM-head NF4-RTN with NF4-SR reduces the final validation loss from 2.579 to 2.440. Why the inward bias dominates specifically in the LM-head, and whether learning-rate decay triggers its harmful effects, remain open questions, as stated in the “Limitations and future work” paragraph of Section 6. **Q2: Extension of the rounding-coordinate analysis and experiments to Muon** We agree that Muon is a useful direction for extending this perspective. Existing work already studies how quantization interacts with its update: Gupta et al. [2] investigate 8-bit blockwise momentum quantization; 4-bit-Muon-GRASP [3] combines grid quantization with preservation of the dominant singular subspace; and MuonQ [4] uses normalization, structural decomposition, and nonuniform quantization to preserve update directions. For Muon, a possible extension would choose rounding or calibration to preserve the update after momentum orthogonalization. Under exact orthogonalization, changing positive singular values while keeping the singular vectors fixed leaves the update unchanged, whereas errors in those vectors can change its direction. This suggests accounting for the effect of quantization on the orthogonalized update when choosing how to store the momentum. The difficulty is that orthogonalization couples entries across the momentum matrix, so our scalar construction for AdamW does not directly apply. Unbiased momentum also need not produce an unbiased update after this nonlinear transformation. Designing an efficient rule would require accounting for this matrix structure and the accumulation of errors through the momentum recurrence. We leave this extension to future work. **References** [1] Kristi Topollai and Anna Ewa Choromanska. *Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resets*. EMNLP, 2026. [2] Aman Gupta et al. *On Quantizing the State of the Muon Optimizer*. SPOT Workshop at ICLR, 2026. [3] Huaijin Wu et al. *Achieving low-bit Muon through subspace preservation and grid quantization*. ICLR, 2026. [4] Yupeng Su et al. *MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimization*. COLM, 2026.

Thank you for the response. I would keep my rating at 4.

However, I find this year’s four-point scale too coarse to convey that judgment. Hearing that 2/3 of the papers this year are rated 2, I think 4 conveys somewhat more support than I intend: I would support the acceptance of this paper, but do not think that it can be highlighted as spotlight or oral at the conference. But lowering the score to 3 is not fair for the authors. I'd appreciate it if the AC would consider my opinion carefully.

Thank you for taking the time to review our manuscript and read our response. We appreciate your feedback throughout the review and your support for acceptance!

Reviewer_68JEOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how rounding optimizer states affects subsequent AdamW updates. The paper proposes two 4-bit configurations. ZIP-SR retains zero in the second-moment codebook and computes stochastic-rounding probabilities in preconditioner space. ZE-EDEN excludes zero and calibrates the decoded block scale. Both configurations use 4-bit NormalFloat (NF4) for the first moment, with targeted stochastic rounding of the LM-head first moment during the final 10% of training Experiments span GPT- and Llama-style pretraining from 130M to 2.7B parameters with three paired seeds, plus full-parameter supervised fine-tuning. Both recipes achieve lower validation loss than TorchAO while remaining close to 32-bit AdamW on downstream tasks.

Critical strengths

Motivated by how the rounding coordinate affects near-zero second-moment quantization at fixed reconstruction levels, authors use Update-SR, which assigns stochastic-rounding (SR) probabilities using the preconditioners induced by the candidate reconstruction levels

Developed two 4-bit AdamW configurations. Zero-Inclusive Preconditioner-space Stochastic Rounding (ZIP-SR) combines a zero inclusive second-moment codebook with Update-SR, whereas Zero-Excluding EDEN calibration (ZE-EDEN) combines a zero-excluding codebook with EDEN block calibration

Strong experimental coverage. Multiple model sizes, two architecture families, substantial pretraining budgets, and paired seeds make the empirical findings more convincing.

Careful failure localization. Matched continuations distinguish LM-head from non-head first-moment effects and demonstrate that changing the rounding rule can prevent the observed instability.

Critical weaknesses

Practical efficiency and competitiveness are incompletely evaluated. The storage accounting is clear, but the paper does not report optimizer latency, whole-step throughput, or measured peak training memory. Additional nonlinear calculations, random-number generation, and block reductions could affect adoption despite unchanged persistent storage.

The evaluation also lacks a representative recent low-bit competitor such as SOLO https://arxiv.org/pdf/2505.00347

Questions for authors

Does the LM-head intervention also prevent TorchAO’s 2.7B cooldown failure when its original codebooks are retained? The reported matched interventions establish the mechanism under Dyn4/Update-SR second moments, leaving this baseline-specific question open.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 68JE

Thank you for the thoughtful comments. We address the weaknesses and questions below.

Q1: Measured optimizer latency, whole-step throughput, and peak training memory

Table 3 in Section 5 reports efficiency on GPT-style 2.7B using eight B300 GPUs, sequence length 2048, and a per-GPU batch size of 8, without activation checkpointing or gradient accumulation.

OptimizerOptimizer-state memory (GiB)Peak GPU memory (GiB)Full-step time (ms)
FP32 AdamW20.67108.94 (0%)315
TorchAO 4-bit2.7591.02 (16.45%)373
ZIP-SR2.7591.01 (16.45%)392
ZE-EDEN2.7591.02 (16.45%)359

Peak allocated GPU memory and full-step time cover forward computation, backward computation, and the optimizer update. Percentages in parentheses report peak-memory reductions relative to FP32 AdamW. All three 4-bit methods provide essentially the same memory saving. Compared with TorchAO, ZIP-SR takes 5.2% longer per step, while ZE-EDEN takes 3.8% less time.

Appendix C.6, Table 15, compares all four optimizers at 834M under two training configurations with the same effective batch size. Combining activation checkpointing with smaller microbatches increases the relative peak-memory reduction of all three 4-bit methods from 4.3% to 26.6%, while the absolute saving remains about 5.3 GiB. Reducing activation memory therefore makes optimizer-state compression a larger contributor to the overall memory reduction.

Q2: Comparison with a recent low-bit optimizer such as SOLO

SOLO is closely related to our study, particularly its logarithmic quantization of the second moment. We evaluate this design in full GPT-style 834M pretraining over three seeds, using its logarithmic-coordinate SR with quantile 0.1 and block size 128.

We retain NF4-RTN first moments and the shared AdamW coefficients to compare the second-moment designs at 4-bit precision. SOLO’s complete recipe also introduces first-moment quantization and precision-dependent momentum choices. Holding these factors fixed lets us assess its logarithmic second-moment quantizer under the same AdamW configuration used for our analysis and controlled experiments.

The mean validation-loss gaps to FP32 AdamW are 0.0133 (0.0006) for ZIP-SR and 0.0208 (0.0007) for logarithmic-coordinate SR, where parentheses denote sample SDs. Appendix C.2, “Comparison with logarithmic-coordinate SR,” reports the setup and results. We identify this experiment explicitly as a comparison with SOLO’s second-moment quantization design.

Q3: Does LM-head SR prevent TorchAO’s 2.7B cooldown failure with its original codebooks?

The intervention substantially mitigates, but does not fully prevent, the loss increase. We now test this directly using three matched RTN/SR continuations from TorchAO checkpoints at 90% of training. We retain the SDyn4 first-moment and Lin4-NZ State-RTN second-moment configurations, changing only the LM-head first-moment rounding rule.

The final validation loss decreases from 3.9435 (0.1347) with RTN to 2.8486 (0.0585) with SR, where parentheses denote sample SDs. Nevertheless, the SR loss rises from 2.5049 (0.0047) at 98% of training to 2.8486 (0.0585) at the endpoint. Each of the three SR continuations shows an increase, ranging from 0.2858 to 0.3939.

Table 13 in Appendix C.5 reports the comparison, and the preceding paragraph gives the late-training checkpoint losses. Section 4.2 summarizes the finding in the discussion of the LM-head intervention.

Reviewer_SuYcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that 4-bit AdamW state quantizers should choose between adjacent levels in the coordinate that matters for the update, not in second-moment state space. For methods, ZIP-SR (zero-inclusive Dyn4 + Update-SR) and ZE-EDEN (zero-excluding Dyn4-NZ + EDEN block calibration), both with an NF4 first moment and stochastic rounding of the LM-head first moment in the final 10% of training. In GPT-style (162M–2.7B) and Llama-style (130M–1.1B) pretraining with three paired seeds, both reduce the gap of TorchAO 4-bit AdamW to 32-bit AdamW at every size (by up to 70.1% at 1.4B) at the same 1.0625 B/parameter storage; in full-parameter SFT of Qwen3-8B-Base and Llama-3.2-3B (five seeds), both reach lower validation loss than TorchAO.

Critical strengths

The paper shows a simple, well-founded reframing with a clean controlled test. Proposition 1 gives a rule-independent ordering r(b) < E[r(QU)] ≤ r(ṽ) ≤ E[r(QS)] and a sharp zero-write criterion (bounded preconditioner error if and only if P(Q = 0) = O(ε)). Figure 1 shows that, with the same zero-inclusive Dyn4 codebook and FP32 first moments, State-RTN/SR become unstable while Update-RTN/SR track 32-bit AdamW. The paper shows a useful diagnostics beyond the method. Matched checkpoint interventions localize the 2.7B late-cooldown failure to deterministic rounding of the LM-head first moment (final validation loss 2.579 vs. 2.440, Table 10). Table 11 shows that NF4-SR stays stable despite larger magnitude-based errors (e.g., 14.75% vs. 9.21% relative RMS error), a useful caution that reconstruction error does not predict optimizer-state stability.

Critical weaknesses

The end-to-end gains are not attributed to the proposed second-moment designs. With TorchAO’s SDyn4 first moment held fixed at 834M, replacing TorchAO’s Lin4-NZ State-RTN second moment with ZIP-SR’s Dyn4 Update-SR moves the gap only from 0.0246 to 0.0239 (both single runs), and ZE-EDEN’s Dyn4-NZ + EDEN is worse than TorchAO (0.0308 vs. 0.0246). Lin4-NZ is never paired with NF4, so the improvement over TorchAO cannot be separated from the SDyn4 → NF4 first-moment swap, which is the largest single step in Table 3 (−0.0105). Within the zero-inclusive route, Update-SR is not isolated either: step 3b changes codebook and rule together in a single run, and because Dyn4-NZ is Dyn4 without zero while Update-SR writes zero with probability at most of order ε/√ṽ (Eq. 1), it is unclear whether zero inclusion or interior-cell rounding drives the 0.0207 → 0.0140 reduction. The comparison set and practical evidence are narrow. TorchAO 4-bit is the only baseline; closely related methods the paper cites, notably logarithmic-coordinate SR (Xu et al., 2025), are not compared, and the LM-head SR fix, derived from continuations of a single 2.7B checkpoint, is given to ZIP-SR and ZE-EDEN but not to TorchAO, whose 2.7B run diverges (+1.5435). Step time and peak memory are not measured (only analytical storage is reported), and in SFT TorchAO’s gap is already only +0.00060 / +0.00175, with every downstream difference within one SD.

Questions for authors

Can you add the following controls? At 834M with three seeds: (i) NF4 first moment with TorchAO’s Lin4-NZ State-RTN second moment, so that the second-moment designs are compared with TorchAO’s at a matched first moment; (ii) Dyn4-NZ with Update-SR, whose comparison with ZIP-SR isolates zero inclusion and whose comparison with Table 3 step 2 isolates the rounding rule; and (iii) logarithmic-coordinate SR (Xu et al., 2025) as an alternative rounding coordinate. At 2.7B and at least one smaller size: (iv) TorchAO with the same LM-head SR rule. Please also report the Dyn4 Update-SR gap with FP32 first moments, which is “—” in Table 8 although Section 3.1 reports L_SR − L_RTN = +1.08×10⁻⁴ for this setting. What are the step-time and peak-memory overheads of Update-SR and EDEN relative to TorchAO’s 4-bit kernels? In practice, what fraction of second-moment writes under ZIP-SR land on zero? This determines how much ZIP-SR actually differs from a zero-excluding quantizer.

3 comments
AuthorsOpenReview ↗

Response to Reviewer SuYc (1/3)

Thank you for the thoughtful comments and suggestions. In the revised manuscript, Table 9 in Appendix C.1 now includes the previously missing single-seed validation-loss gap for Dyn4 Update-SR with FP32 first moments (+0.0008 relative to 32-bit AdamW). We address the remaining weaknesses and questions below.

Q1: Isolating second-moment gains with matched NF4 first moments (834M, three seeds)

Thank you for suggesting this control. With NF4-RTN first moments fixed, Lin4-NZ State-RTN achieves lower mean validation loss than our original Dyn4-based ZIP-SR and ZE-EDEN configurations at 834M. We agree that NF4 accounts for a substantial part of the improvement over the original TorchAO recipe. We therefore also evaluate Update-SR with the linear codebook. Over three seeds, the final validation losses are 2.6502 (0.0016) for Lin4-NZ State-RTN, 2.6483 (0.0018) for Lin4-NZ Update-SR, and 2.6481 (0.0011) for Lin4 Update-SR, with NF4-RTN first moments throughout. Parentheses denote sample SDs.

The same improvement appears with Dyn4-NZ: switching from State-RTN to Update-SR reduces the mean gap to FP32 AdamW from 0.0207 to 0.0128. Thus, Update-SR improves on State-RTN with both tested positive second-moment codebooks, with each comparison keeping both moment codebooks fixed. Its benefit is therefore not specific to the dynamic codebook. This supports the central argument in our title and contributions: the rounding space is an important design choice for optimizer-state quantization. Update-SR also allows stable training with the tested zero-inclusive codebooks, as discussed in Q2.

The 834M results suggest that using a linear second-moment codebook may further improve our recipes at larger model sizes. These comparisons were conducted during rebuttal, and time constraints prevented extending the linear variants to larger sizes; our larger-model experiments therefore retain the original Dyn4-based recipes. We leave this extension and further codebook optimization to future work. Our focus is on the benefit of choosing rounding probabilities in preconditioner space, rather than tuning codebooks to claim state-of-the-art performance. Table 4 and the revised Section 5.1 separate the effects of first-moment choice, second-moment codebook, and rounding rule; Appendix C.1 reports additional first-moment controls.

Q2: Separating zero inclusion from rounding using Dyn4-NZ + Update-SR (834M, three seeds)

We have added the requested Dyn4-NZ + Update-SR control on GPT-style 834M over three random seeds, with NF4-RTN first moments throughout training. Table 4 in Section 5.1 reports the results.

With Dyn4-NZ fixed, switching from State-RTN to Update-SR reduces the mean validation-loss gap from 0.0207 (0.0010) to 0.0128 (0.0008). Retaining zero under Update-SR gives a similar gap of 0.0133 (0.0006), where parentheses denote sample SDs. The difference between the two Update-SR means is small relative to the observed run-to-run variability, and these results do not establish a performance advantage from either retaining or excluding zero.

Our key insight is that choosing rounding probabilities in preconditioner space allows a zero-inclusive quantizer to train stably. Update-SR adapts the probability of selecting zero to the induced preconditioner values, making zero unlikely when it would produce an excessively large preconditioner. Proposition 1(a) in Section 3.1 further shows that retaining zero reduces the conditional one-step bias caused by a positive floor. This correction applies below the floor; between shared positive levels, both configurations use the same Update-SR rule. Its effect on final loss therefore depends on the states encountered and how the resulting corrections affect training. Our 834M experiments show comparable final losses, while a larger benefit remains possible in settings where the positive floor more frequently suppresses updates.

Q3: Comparison with logarithmic-coordinate SR (Xu et al., 2025)

Xu et al. (2025) use logarithmic quantization to address signal swamping, where rounding prevents the second-moment EMA from responding to small updates. We study how errors in the stored second moment affect AdamW’s preconditioner. Near zero, the inverse-square-root transformation can amplify small state errors, so Update-SR sets rounding probabilities using the induced preconditioner values.

We compare ZIP-SR and logarithmic-coordinate SR on GPT-style 834M with NF4-RTN first moments throughout. Both use 4-bit second moments with block size 128 and their respective codebooks. The logarithmic quantizer uses quantile 0.1. Over three seeds, their mean paired validation-loss gaps to 32-bit AdamW are 0.0133 (0.0006) for ZIP-SR and 0.0208 (0.0007) for logarithmic-coordinate SR, where parentheses denote sample SDs. ZIP-SR therefore achieves lower final validation loss than the tested logarithmic-coordinate SR configuration in this setting. The setup and results are in Appendix C.2.

AuthorsOpenReview ↗

Response to Reviewer SuYc (2/3)

Q4: TorchAO with the same LM-head SR rule (2.7B and at least one smaller model)

We have added this control at 405M, 834M, and 2.7B, using three matched RTN/SR continuations per size from checkpoints at 90% of training. Both branches retain TorchAO’s SDyn4 first-moment and Lin4-NZ State-RTN second-moment configurations; only the LM-head first-moment rounding rule changes.

At 2.7B, applying SR reduces the final validation loss from 3.9435 (0.1347) to 2.8486 (0.0585), where parentheses denote sample SDs. However, the SR loss rises from 2.5049 (0.0047) at 98% of training to 2.8486 (0.0585) at the endpoint. All three SR continuations show this increase, ranging from 0.2858 to 0.3939. Thus, the intervention substantially mitigates, but does not fully prevent, the degradation with TorchAO’s original codebooks. The endpoint changes at 405M and 834M are small.

Table 13 in Appendix C.5 reports the full comparison, and the preceding paragraph gives the late-training checkpoint losses. Section 4.2 summarizes this finding where we introduce the final-10% LM-head SR rule.

Q5: Measured step-time and peak-memory overheads of Update-SR and EDEN versus TorchAO

Table 3 in Section 5 reports efficiency on GPT-style 2.7B using eight B300 GPUs, sequence length 2048, and a per-GPU batch size of 8 (global batch size 64), without activation checkpointing or gradient accumulation.

OptimizerOptimizer-state memory (GiB)Peak GPU memory (GiB)Full-step time (ms)
FP32 AdamW20.67108.94 (0%)315
TorchAO 4-bit2.7591.02 (16.45%)373
ZIP-SR2.7591.01 (16.45%)392
ZE-EDEN2.7591.02 (16.45%)359

Peak allocated GPU memory and full-step time cover forward computation, backward computation, and the optimizer update. Percentages in parentheses report peak-memory reductions relative to FP32 AdamW. All three 4-bit methods provide essentially the same memory saving. Compared with TorchAO, ZIP-SR takes 5.2% longer per step, while ZE-EDEN takes 3.8% less time.

Appendix C.6, Table 15, also compares all four optimizers at 834M under two training configurations with the same effective batch size. Combining activation checkpointing with smaller microbatches increases the relative peak-memory reduction of all three 4-bit methods from 4.3% to 26.6%, while the absolute saving remains about 5.3 GiB. Reducing activation memory therefore makes optimizer-state compression a larger contributor to the overall memory reduction.

Q6: Fraction of second-moment writes landing on zero under ZIP-SR

Table 10 in Appendix C.1, also referenced in Section 3.1, reports stored-zero frequencies by component at 5%, 20%, 50%, and 90% of training, using a GPT-style 834M run on FineWeb-Edu as an example. Across these checkpoints, the frequencies range from 1.37% to 1.54% in the embedding and from 0.19% to 0.83% in the LM head, while remaining below 0.002% in attention and MLP.

The concentration in vocabulary layers is consistent with their token-dependent gradients. Rare input tokens receive gradients only when they appear in a batch, so their second moments can decrease while the tokens are absent. In the LM head, low-probability non-target tokens contribute small softmax gradients [1]. Non-uniform token frequencies and their effects on embedding gradients are also discussed in [2, Appendix C]. As discussed alongside Table 10 in Appendix C.1, our GPT-style models use untied embeddings. With the same architecture, a corpus covering less of the vocabulary could leave more input-token rows unupdated and produce a higher fraction of zero second moments.

Retaining zero preserves zero states exactly and gives small positive states an alternative to being forced up to a positive floor. Update-SR controls this choice through a state-dependent rounding probability, making zero unlikely when its induced preconditioner would be excessively large. The frequencies reported here describe one model and corpus; the token-coverage differences discussed above could produce higher zero fractions in other settings. Where small second moments are more common, retaining zero offers more opportunities to reduce floor-induced preconditioner bias, and this could make its effect on training more visible.

AuthorsOpenReview ↗

Response to Reviewer SuYc (3/3)

Q7: Interpretation of the small SFT loss gains and downstream differences within one SD

We agree that the absolute SFT gains are small. Both proposed methods achieve lower mean Tulu validation loss than TorchAO. On downstream tasks, all three 4-bit configurations remain close to 32-bit AdamW: the mean paired score gaps are within one reported SD, with no consistent advantage across tasks for either proposed method.

Variability in downstream scores across training seeds and checkpoints is documented in prior benchmark studies [3, 4]. These studies also find that continuous likelihood metrics computed on benchmark answers often have higher signal-to-noise ratios than accuracy or exact-match scores. This supports interpreting small downstream differences together with their variability, which we report over five paired seeds.

We have revised Section 5.3 (“Supervised Fine-tuning”) to state these findings explicitly alongside Table 6. Appendix B.3 describes the SFT setup and evaluation. The SFT experiments assess whether the recipes remain effective when adapting pretrained models; the pretraining experiments show clearer performance differences among the configurations.

References

[1] Ofir Press and Lior Wolf. Using the Output Embedding to Improve Language Models. EACL, 2017.

[2] Tim Dettmers et al. 8-bit Optimizers via Block-wise Quantization. ICLR, 2022.

[3] Lovish Madaan et al. Quantifying Variance in Evaluation Benchmarks. arXiv:2406.10229, 2024.

[4] David Heineman et al. Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation. NeurIPS, 2025.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overview of rebuttals & revisions

We thank the area chair and reviewers for their constructive feedback. Reviewer SuYc and Reviewer ULqc recognized the rounding-space perspective and its theoretical support. Reviewer SuYc highlighted the controlled evidence that changing the rounding rule can prevent instability with the same zero-inclusive codebook.

Reviewers also valued the broad experimental coverage (Reviewer 68JE) and the matched interventions that isolate the LM-head instability (Reviewer SuYc, Reviewer 68JE). These diagnostics show why smaller reconstruction errors need not imply more stable training.

To address the remaining concerns, we strengthened the component attribution, expanded the baseline comparisons, and measured training memory and step time. The main revisions are summarized below.

  1. Component attribution. New three-seed 834M controls with matched NF4 first moments show that Update-SR improves on State-RTN with both tested positive codebooks, Lin4-NZ and Dyn4-NZ (Section 5.1, Table 4).

  2. Logarithmic-coordinate SR. We compare ZIP-SR with the second-moment quantizer of Xu et al. (2025) over three seeds under matched first-moment settings (Appendix C.2).

  3. TorchAO with LM-head SR. Tests at 405M, 834M, and 2.7B show that the intervention substantially reduces, but does not eliminate, TorchAO's late loss increase at 2.7B (Section 4.2; Appendix C.5, Table 13).

  4. Efficiency. Table 3 reports measured memory and full-step time at 2.7B without activation checkpointing or gradient accumulation. Appendix C.6 compares all four optimizers at 834M and shows how training configuration affects the relative memory saving.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.