acceptodds
Under review as a conference paper at ICLR 2027

ResQAdam: Residual-Preserving Adaptive Optimization for Memory-Efficient Quantization Aware Training

Abstract

Quantization-aware training (QAT) is the most reliable way to obtain accurate low-bit large language models (LLMs), but training with AdamW keeps two dense FP32 states for every weight matrix. We find that low-rank memory-efficient optimizers such as GaLore and LDAdam degrade substantially in QAT. A considerable part of the QAT momentum lies outside its top- singular subspace, and projection-based methods discard this part at every step. We further observe that this residual remains useful when only its sign is kept. Based on these observations, we propose ResQAdam. It keeps a single BF16 momentum, applies Adam only to the coordinates of the principal subspace, and adds a normalized sign update computed from the residual. We evaluate ResQAdam with two from-scratch QAT methods and three QAT fine-tuning pipelines, with weights quantized down to 2 bits. It reduces optimizer-state memory by 71% to 74% relative to AdamW, stays close to AdamW in most settings, and lowers training throughput by less than 2%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Quantization-Aware Training, Model Compression, Efficient Training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AZcLOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

ResQAdam combines adaptive updates in a low-rank subspace with a momentum-based residual update for quantization-aware training. It substantially reduces optimizer-state storage and achieves measurable peak-memory savings, though accuracy preservation varies across quantization settings.

Critical strengths
  1. Experiments cover several quantization recipes, bit widths, and model sizes, with ablations supporting the residual branch.
  2. Actual peak-memory and throughput measurements substantiate the efficiency benefits beyond optimizer-state accounting.
Critical weaknesses
  1. Although FiRA and FRUGAL were not originally evaluated for QAT, they also retain updates outside the low-rank subspace. A representative QAT comparison would help determine whether ResQAdam’s specific residual design offers benefits beyond simply retaining these updates.
  2. Accuracy preservation is uneven in demanding low-bit settings. For Qwen3-4B with LLM-QAT W2A16C, average perplexity increases from 33.84 with AdamW to 46.72, alongside a drop in accuracy.
Questions for authors

See weaknesses. The quantization-specific explanation is not established. The ablations show that residual updates are useful, but do not demonstrate that they specifically preserve quantization-compensation signals. A matched non-quantized control or direct analysis of quantization-induced changes would help substantiate this interpretation.

6 comments
AuthorsOpenReview ↗

Response to Reviewer AZcL (1/4)

Thanks for the valuable and constructive comments, which are definitely helpful for us to improve the quality of the paper. We have uploaded a revised version to accommodate all your suggestions and questions. Below, we further provide detailed clarifications for each of your concerns:

W1: Although FiRA and FRUGAL were not originally evaluated for QAT, they also retain updates outside the low-rank subspace. A representative QAT comparison would help determine whether ResQAdam's specific residual design offers benefits beyond simply retaining these updates.

A1: Thanks for pointing this out. Following your suggestion, we have added Fira and FRUGAL in Table 4 under the same QAT settings. The results answer the question in two steps: keeping the residual accounts for most of the gain over projection-only methods, and the specific residual design of ResQAdam gives a further gain in all four settings at a small memory cost. We first give the setup and results, then explain which design choices give the extra gain, and finally compare memory.

Setup. We follow the other baselines in Table 4: Llama2-297M for from-scratch QAT and Llama 3.2-1B for QAT fine-tuning, an LSQ quantizer with group size 128, and the same data, training steps and projection rank (r=64r = 64). We use the official implementations. Fira uses its default α=0.25\alpha = 0.25, and FRUGAL uses Adam in the subspace and signSGD outside it. The learning rate of every method is grid-searched and the best result is reported. Numbers are average perplexity on WikiText-2 and C4. The GaLore-AdamW column is taken from Table 4.

ParadigmQ-CfgResQAdamFRUGALFiraGaLore-AdamW
From-scratch QATW2A16G12820.0520.2521.1030.61
W4A4G12818.8920.0921.0632.93
QAT fine-tuningW2A16G12816.8917.5917.8119.25
W4A4G12815.0515.2715.3015.30

Keeping the residual gives most of the gain. Fira and FRUGAL close most of the gap left by the projection-only GaLore-AdamW, for example from 30.61 to 20.25 in from-scratch W2A16. This is consistent with the main observation of our paper: a large part of the useful QAT update lies outside the top-rr subspace.

The residual design of ResQAdam adds a consistent gain on top. ResQAdam is best in all four settings, by 0.20 to 1.20 ppl over FRUGAL and 0.25 to 2.17 ppl over Fira. The gap is largest in from-scratch W4A4 and smallest in fine-tuning W4A4, where the quantization error is small.

Where the difference comes from. The three methods differ in what the residual is computed from:

  • Fira and FRUGAL compute the residual from the current STE gradient. FRUGAL applies signSGD to it, and Fira rescales it by the norm ratio of the low-rank Adam update.
  • ResQAdam computes the residual from the momentum MtM_t, which averages the noisy STE gradient over steps. The projection bases also come from MtM_t, and the principal and residual updates are fused by the momentum energy ratio.

Two ablations already in the paper (Section 5.1) test these choices. Both keep the residual and change only one design choice, so they test the design rather than whether to keep the residual. Gradient-SVD Bases computes the bases from GtG_t instead of MtM_t. Global-Momentum Sign applies the sign to the whole MtM_t without splitting it into principal and residual parts. Both underperform ResQAdam, and Gradient-SVD Bases is clearly worse under W2A2.

Memory. The extra gain comes at a small memory cost. Fira and FRUGAL keep the same state as GaLore-AdamW, with a state-free residual, while ResQAdam additionally keeps one BF16 momentum.

SettingMetricFiraFRUGALResQAdamAdamW
From-scratch (Llama2-297M)Opt. state memory0.136 GB0.136 GB0.594 GB2.38 GB
WikiText-2 ppl18.3617.6516.2016.39
Fine-tuning (Llama 3.2-1B)Opt. state memory2.42 GB2.42 GB3.09 GB9.89 GB
WikiText-2 ppl13.8513.6213.2913.30

With 0.46 GB (from scratch) and 0.67 GB (fine-tuning) more optimizer state than Fira and FRUGAL, ResQAdam matches AdamW in WikiText-2 perplexity, while using 25% and 31% of AdamW's optimizer-state memory.

AuthorsOpenReview ↗

Response to Reviewer AZcL (2/4)

W2: Accuracy preservation is uneven in demanding low-bit settings. For Qwen3-4B with LLM-QAT W2A16C, average perplexity increases from 33.84 with AdamW to 46.72, alongside a drop in accuracy.

A2: We clarify that the cited numbers are correct, and ResQAdam performs worse than AdamW in this setting. We would like to provide additional context from downstream task accuracy, where the observed differences are relatively modest despite the larger perplexity gap.

To show this, we compare two low-bit rows of Table 2. For reference, the zero-shot tasks (HellaSwag, PIQA, ARC, WinoGrande, MMLU) have an average chance level of about 35%.

ModelMethodQ-CfgAvg. task acc. (AdamW / Ours)Avg. ppl (AdamW / Ours)
Llama 3.2-1BEfficientQATW2A16G12835.14 / 35.43351.78 / 379.52
Qwen3 4B-BaseLLM-QATW2A16C40.85 / 38.0233.84 / 46.72

For the cited Qwen3 setting, the downstream accuracy difference is 2.83 percentage points, compared with a perplexity increase from 33.84 to 46.72. Both models also exhibit substantial degradation relative to the 4-bit counterpart (69.29% accuracy), with their task accuracy approaching the chance level.

A similar pattern can be observed in the Llama 3.2-1B setting, where both optimizers achieve near-chance task accuracy (35.14% vs. 35.43%), while perplexity differs by 27.74. These observations suggest that when model performance is severely degraded under extreme low-bit quantization, perplexity differences can become more pronounced even when downstream accuracy differences remain relatively small.

AuthorsOpenReview ↗

Response to Reviewer AZcL (3/4)

Q: The quantization-specific explanation is not established. The ablations show that residual updates are useful, but do not demonstrate that they specifically preserve quantization-compensation signals. A matched non-quantized control or direct analysis of quantization-induced changes would help substantiate this interpretation.

A3: Thanks for the suggestion. To clarify, the ablations show that the residual is useful, but they do not link it to quantization. Following your suggestion, we have run three new analyses: (a) a direct analysis of the quantization-induced change in the momentum, (b) a matched full-precision control during training, and (c) a control at the level of single updates, which addresses the ablation question directly. Because of the length limit, (c) is in the next post.

The experiments for (a) and (b) have been added to Section 3.1 and Appendix E, while those for (c) are presented in Appendix D of the revised paper.

(a) Direct analysis of the quantization-induced change. We first measure how much of the quantization-induced signal a low-rank projection keeps.

Setup. We record two full ResQAdam runs (Llama2-297M, LSQ, G128, r=64r = 64, 12288 steps, all 84 linear layers), at W2A16 and at W4A4. At every step we compute two gradients on the same mini-batch, with and without the weight quantizer, gq=∇L(Q(W))g^{q} = \nabla L(Q(W)) and gfp=∇L(W)g^{fp} = \nabla L(W), and accumulate each into its own momentum with the optimizer's β1\beta_1, giving MqM^{q} and MfpM^{fp}. Their difference D=Mq−MfpD = M^{q} - M^{fp} is the part of the momentum caused by weight quantization. It contains no mini-batch noise and is exactly zero in full precision. Training uses only gqg^{q}, so the extra pass does not change the run. The bases U,VU, V are computed from MqM^{q} as in ResQAdam, so we test the subspace the optimizer actually uses.

Metric. The share of DD kept by this subspace, in-sub(D)=∥UU⊤DVV⊤∥F2/∥D∥F2\mathrm{in\text{-}sub}(D) = \Vert UU^\top D VV^\top \Vert_F^2 / \Vert D \Vert_F^2. A projection-based optimizer keeps only this part of its update.

stepin-sub(DD), W2A16in-sub(DD), W4A4
4000.3500.346
20480.2330.295
61440.0960.329
122880.1100.340

At W2A16 the subspace keeps 11.0% of DD at the end of training, so a projection-only update discards 89% of it. At W4A4 it keeps 34.0%. The lower the bit width, the more of the quantization-induced signal lies outside the principal subspace. This is not caused by outdated bases: grouped by the number of steps since the last refresh, in-sub(DD) stays within 0.102–0.119 at W2A16 and 0.272–0.289 at W4A4, so refreshing more often would not recover it. The residual-sign update keeps this part.

(b) Matched full-precision control during training. Next, we check whether quantization makes the momentum itself harder to capture with a low-rank subspace. We train four AdamW runs on Llama2-297M that differ only in the quantization setting: full precision, W4A16, W3A16 and W2A16. Every 200 steps, we take the AdamW first moment mm of the seven weight matrices in a middle layer and compute its residual energy res64(m)=1−∑i≤64σi2(m)/∑iσi2(m),\mathrm{res}_{64}(m) = 1 - \sum_{i \le 64} \sigma_i^2(m) \Big/ \sum_i \sigma_i^2(m), where σi(m)\sigma_i(m) are the singular values of mm. This is the share of the momentum energy outside its best rank-64 approximation, so a projection-based optimizer with rank 64 discards at least this share. We report the mean over the seven matrices.

training settingstep 5200 (mid-training)step 10200 (last recorded step)
full precision0.3500.340
W4A160.3680.364
W3A160.3810.375
W2A160.4140.418

The residual energy increases as the bit width decreases, and the full-precision run is the lowest at both steps. At step 10200, a rank-64 projection discards at least 34.0% of the momentum energy in full precision and 41.8% under W2A16. These results suggest that quantization error may increasingly spread into the residual subspace of the first moment as the bit width decreases, making low-rank projection less effective at capturing it.

Reviewer_toGdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ResQAdam, a memory-efficient optimizer for low-bit quantization-aware training (QAT). By applying Adam in the principal subspace and a sign update to the residual, ResQAdam achieves performance close to AdamW while reducing optimizer-state memory by approximately 71%–74%.

Critical strengths
  • The paper identifies that traditional low-rank optimizers may discard important residual optimization signals during QAT, providing a valuable research insight.
  • The proposed method combines low-dimensional Adam with a residual sign update, reducing optimizer-state memory while preserving full-parameter updates.
  • The experiments cover multiple QAT methods, models, and quantization configurations, with evaluations of model accuracy, memory usage, and training throughput.
Critical weaknesses
  • Insufficient validation of the motivation: The paper suggests that quantization-error compensation signals in QAT are more distributed outside the principal low-rank subspace, but lacks direct experimental evidence to support this hypothesis.
  • Limited scale of from-scratch training experiments: The from-scratch QAT experiments are limited to 200M and 430M models, leaving the effectiveness of the method on larger LLMs unclear.
  • Incomplete comparison with related optimizers: The paper does not compare against methods such as Fira and FRUGAL, which also consider updates outside the low-rank subspace, making it difficult to fully assess the advantages of ResQAdam
Questions for authors
  1. Could the authors compare the residual energy distributions between full-precision training and QAT with different quantization bit widths under the same training settings, and analyze whether these residuals are indeed related to quantization-error compensation?

  2. Could the authors provide additional from-scratch QAT experiments on models with at least 1B parameters to evaluate the convergence, training memory, and throughput of ResQAdam at a larger scale?

  3. Could the authors compare ResQAdam with other optimizers that preserve residual updates, such as Fira and FRUGAL, to further demonstrate its advantages over existing methods?

3 comments
AuthorsOpenReview ↗

Response to Reviewer toGd (1/2)

Thanks for the careful and constructive comments, which are definitely helpful for us to improve the quality of the paper. We have uploaded a revised version to accommodate all your suggestions. Below, we further provide detailed clarifications for each of your questions:

W1: Insufficient validation of the motivation: The paper suggests that quantization-error compensation signals in QAT are more distributed outside the principal low-rank subspace, but lacks direct experimental evidence to support this hypothesis. Q1: Could the authors compare the residual energy distributions between full-precision training and QAT with different quantization bit widths under the same training settings, and analyze whether these residuals are indeed related to quantization-error compensation?

A1: Thanks for the comment. Following your suggestion, we now measure it in three ways: 1) during training, we isolate the quantization-induced part of the momentum and measure how much of it the principal subspace keeps (W1), 2) following Q1, we compare the residual energy of the momentum between full-precision training and QAT at different bit widths under the same settings, and 3) on trained checkpoints, we split the gradient into a task part and a quantization part and compare them across bit widths.

1) Where the compensation signal lies. During two full ResQAdam runs on Llama2-297M (W2A16 and W4A4), we compute at every step the gradient with and without the weight quantizer on the same mini-batch and accumulate each into its own momentum. Their difference D=Mq−MfpD = M^{q} - M^{fp} is the compensation signal, and it is exactly zero in full precision. The rank-64 subspace that ResQAdam uses keeps only 11.0% of DD at the end of training at W2A16, so a projection-only update discards 89% of it, and keeps 34.0% at W4A4. This share does not depend on the number of steps since the last refresh, so refreshing more often would not recover it. The full setup, metric and time series are in our reply to Reviewer AZcL (Q, part (a)).

2) Full-precision training vs. QAT at different bit widths. We train four AdamW runs on Llama2-297M that differ only in the quantization setting and measure the residual energy of the first moment, res64(X)=1−∑i≤64σi2(X)/∑iσi2(X)\mathrm{res}_{64}(X) = 1 - \sum_{i \le 64} \sigma_i^2(X) / \sum_i \sigma_i^2(X), i.e., the share of energy outside the best rank-64 approximation of XX, averaged over the seven weight matrices of a middle layer. A rank-64 projection-based optimizer discards at least this share. As shown in the table below, at step 10200, the residual energy is 0.340 in full precision, 0.364 at W4A16, 0.375 at W3A16 and 0.418 at W2A16, and the same trend is observed at step 5200.

training settingstep 5200 (mid-training)step 10200 (last recorded step)
full precision0.3500.340
W4A160.3680.364
W3A160.3810.375
W2A160.4140.418

These results suggest that quantization error may increasingly spread into the residual subspace of the first moment as the bit width decreases, making low-rank projection less effective at capturing it.

3) Is the residual related to quantization-error compensation?

Setup. At the quantized weights, the QAT gradient splits exactly into two true gradients, ∇L(Q(W))=∇L(W)+c\nabla L(Q(W)) = \nabla L(W) + c, where cc exists only because of the quantization error Q(W)−WQ(W) - W and is zero in full precision. We take gradients with respect to the quantized weights directly, so no STE is involved, on final checkpoints trained with the same recipe at W8, W4, W3 and W2 (all A16).

Metrics. c/∇L(W)c/\nabla L(W) for the size of cc relative to the task gradient, and res64(c)−res64(∇L(W))\mathrm{res}_{64}(c) - \mathrm{res}_{64}(\nabla L(W)), which is positive when cc is harder than the task gradient for any rank-64 subspace to hold.

setting∥c∥/∥∇L(W)∥\|c\|/\|\nabla L(W)\|res64(c)−res64(∇L(W))\mathrm{res}_{64}(c) - \mathrm{res}_{64}(\nabla L(W))
W8A160.566−0.1058-0.1058
W4A160.968−0.0142-0.0142
W3A161.003+0.0044+0.0044
W2A161.217+0.0220+0.0220

As the bit width decreases, cc grows from 0.57 to 1.22 times the task gradient, so at low bit widths the compensation component is as large as the task gradient itself. The gap in residual energy increases monotonically and turns positive at 3 bits, so at 3 and 2 bits cc is harder than the task gradient for a rank-64 subspace to hold.

This experiment has been added to Section 3.1 and Appendix E of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer toGd (2/2)

W2: Limited scale of from-scratch training experiments: The from-scratch QAT experiments are limited to 200M and 430M models, leaving the effectiveness of the method on larger LLMs unclear. Q2: Could the authors provide additional from-scratch QAT experiments on models with at least 1B parameters to evaluate the convergence, training memory, and throughput of ResQAdam at a larger scale?

A2: Thanks for this suggestion. We clarify that, of the three quantities raised by the reviewer, memory and throughput are already evaluated at this scale in our original submission, therefore, our new experiments focus on convergence. We provide more details below.

Memory and throughput. These are reported in Table 3 on Llama 3.2-1B, Qwen2.5-1.5B and Qwen3 4B-Base. ResQAdam reduces optimizer-state memory by 72.2%, 71.3% and 73.2%, reduces total peak memory by 15.3% to 19.5%, and lowers throughput by at most 1.4%.

Convergence. Following your suggestion, we have trained one additional 1.2B Llama-style models from scratch under W2A16 with both frameworks used in the paper, QuEST and CAGE, and compared against full-state AdamW with the same data, schedule and hyperparameters.

Note that pretraining the 1.2B model is substantially more expensive than the 200M and 430M runs, our compute budget covered only W2A16, then we do not report W2A2 or W4A4 at this scale.

The QuEST and CAGE codebases give no configuration at this size. Extending their rule of halving the peak learning rate each time the model size doubles gives 5×10−55\times10^{-5}, but 1×10−41\times10^{-4} gives lower perplexity for both AdamW and ResQAdam under both frameworks, so all runs below use 1×10−41\times10^{-4}.

1.2B from-scratch QAT, W2A16G128. Each cell is AdamW / ResQAdam, as in Table 1.

QAT methodWikiText-2C4Avg. pplOpt. state memory
QuEST25.25 / 26.1714.87 / 15.1620.06 / 20.669.52 / 2.57 GB (↓73.0%)
CAGE25.48 / 27.0415.22 / 15.5320.35 / 21.289.52 / 2.57 GB (↓73.0%)

ResQAdam trains stably at this scale, and its training loss closely follows that of AdamW over the whole run with both frameworks. Its final validation loss is 0.023 higher than AdamW with QuEST and 0.025 higher with CAGE. ResQAdam also stays close to AdamW in average perplexity, while its optimizer state is 73.0% smaller. This matches what we observe at 200M and 430M.

The training-loss curves for both frameworks are in Appendix B of the revised paper.

W3: Incomplete comparison with related optimizers: The paper does not compare against methods such as Fira and FRUGAL, which also consider updates outside the low-rank subspace, making it difficult to fully assess the advantages of ResQAdam. Q3: Could the authors compare ResQAdam with other optimizers that preserve residual updates, such as Fira and FRUGAL?

A3: Thanks for this suggestion. We clarify that Fira and FRUGAL are indeed the closest existing methods, and leaving them out was a real omission. We have added both to Table 4, using the same settings as the other baselines and grid-searching the learning rate of every method. Reviewer AZcL asked for the same comparison, and the full setup, results table and memory accounting are in our reply there (W1). Below we summarize the advantages of ResQAdam over the two methods.

Perplexity. ResQAdam is best in all four settings (from-scratch QAT and QAT fine-tuning, each at W2A16G128 and W4A4G128). Its average perplexity on WikiText-2 and C4 is 0.20 to 1.20 lower than FRUGAL and 0.25 to 2.17 lower than Fira. The gap is largest in from-scratch W4A4 (18.89 against 20.09 and 21.06) and smallest in fine-tuning W4A4, where the quantization error is small. Both baselines also close most of the gap left by the projection-only GaLore-AdamW (30.61 to 20.25 in from-scratch W2A16), which supports the main observation of the paper that a large part of the useful QAT update lies outside the top-rr subspace.

Memory and quality. Fira and FRUGAL keep the same optimizer state as GaLore-AdamW, while ResQAdam additionally keeps one BF16 momentum. With 0.46 GB (from scratch) and 0.67 GB (fine-tuning) more optimizer state, ResQAdam matches AdamW in WikiText-2 PPL (16.20 vs. 16.39 and 13.29 vs. 13.30), while Fira and FRUGAL reach 18.36 and 17.65 from scratch and 13.85 and 13.62 in fine-tuning. ResQAdam uses 25% and 31% of AdamW's optimizer-state memory.

Why ResQAdam differs. Fira and FRUGAL compute the residual from the current STE gradient, while ResQAdam computes it from the momentum, which averages the noisy STE gradient over steps. ResQAdam also takes its projection bases from the momentum and fuses the two updates by the momentum energy ratio. The Gradient-SVD Bases and Global-Momentum Sign ablations in Section 5.1, which keep the residual but change these choices, both underperform ResQAdam.

We would be happy to add additional results if the reviewer has any questions.

Thanks for your response. I appreciate the extra work you put into showing how the residuals change with bit width, training a 1.2B model from scratch, and comparing ResQAdam with Fira and FRUGAL. These results address most of my concerns, so I will keep my positive rating.

Reviewer_6QfnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the memory budget challenges in quantization-aware training (QAT) for large language models, and found that low-rank memory-efficient optimizers such as GaLore and LDAdam degrade substantially in QAT due to the loss of useful optimization signals associated with quantization-error correction. They propose ResQAdam that keeps a single BF16 momentum, applies Adam only to the r × r coordinates of the principal subspace, and adds a normalized sign update computed from the residual. The author claim that the ResQAdam can reduce the optimizer-state memory by 71% to 74% relative to AdamW, stays close to AdamW in most settings, and lowers training throughput by less than 2%.

Critical strengths
  1. ResQAdam is a simple and understandable design which incorporate low-rank subspace extraction, adam-style adaptive updates and normalized sign update derived from the residual, fused using momentum-energy weights.
  2. The paper reports optimizer-state memory reductions of approximately 71–74% which is significant.
  3. Ablations against principal-update-only, residual-sign-only, gradient-SVD-basis, and sensitivity study of rank r is critical.
Critical weaknesses
  1. Experiments doesn't fully establish that the residual's usefulness is specifically due to quantization-error compensation, as opposed to a more general benefit from sign-based updates or from changing the update's coordinate-wise scaling. Can you compare residual-sign update against alternatives that control for these factors, including a normalized residual update without sign.
  2. Why should the residual update's norm match the principal update's norm before energy weighting? How sensitive is performance to the normalization rule, particularly when the principal update has an unusually small norm?
  3. The paper provides meaningful experimental details but several choices interact such as momentum precision, subspace rank, refresh frequency, Adam states, residual normalization, energy-based fusion, and the initialization/warm-up procedure which need to be well documented.
Questions for authors

See above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 6Qfn (1/3)

We thank the reviewer for the detailed and constructive comments. As suggested, we have run additional experiments for W1 and W2. For W3, we have added all interacting design choices in Appendix C of the revised paper. We also have uploaded a revised version to accommodate all your suggestions. Below, we further provide detailed clarifications for each of your questions:

W1: Experiments doen't fully establish that the residual's usefulness is specifically due to quantization-error compensation, as opposed to a more general benefit from sign-based updates or from changing the update's coordinate-wise scaling. Can you compare residual-sign update against alternatives that control for these factors, including a normalized residual update without sign.

A1: Thanks for the suggestion. Following it, we conducted these control experiments. The question raises two alternative explanations, which we test separately: (a) whether the benefit comes from the sign operation or from coordinate-wise rescaling, and (b, c) whether the information carried by the residual is specifically related to quantization.

(a) The benefit does not come from the sign. To separate the residual direction from the sign and the scaling, we change only the residual direction added to the principal update, using the family sign(Mt⊥)⊙∣Mt⊥∣p\mathrm{sign}(M_t^{\perp}) \odot |M_t^{\perp}|^{p}, rescaled to ∥Δt∥∥F\Vert \Delta_t^{\parallel} \Vert_F as in ResQAdam. p=0p = 0 is the sign rule of ResQAdam, and p=1p = 1 is the normalized residual without sign that the reviewer asks for. Setting: Llama2-297M from scratch, LSQ with group size 128, r=64r = 64, same data and steps. All variants use the same learning rate without separate tuning, so the p=0p = 0 row differs slightly from Table 4, which reports the best learning rate. Numbers are average perplexity on WikiText-2 and C4.

residual directionW2A16W4A4
none (Principal Update Only)44.5445.44
sign, p=0p = 0 (ResQAdam)20.4518.49
p=0.25p = 0.2519.4418.76
p=0.50p = 0.5019.4018.34
p=0.75p = 0.7519.0418.62
normalized residual, p=1p = 119.1140.92
  • Removing the sign keeps the gain. At W2A16, p=1p = 1 reaches 19.11, against 20.45 for the sign rule. If the benefit came from the sign operation, p=1p = 1 would lose it.
  • Coordinate-wise rescaling is not the source either. From p=0p = 0 to p=0.75p = 0.75, perplexity varies by at most 1.41 at W2A16 and 0.42 at W4A4, while adding the residual improves perplexity by 24.1 and 27.0.
  • Why we keep the sign. All variants with p≤0.75p \le 0.75 train stably at both bit widths, but p=1p = 1 fails at W4A4 (40.92, with a training-loss spike to 6.90 around step 3000). The sign rule is the simplest choice in the stable range and needs no extra state.

Besides stability, the sign also lowers communication cost. After the sign, the residual term is a ±1\pm 1 matrix with one scale, and the principal term UtA^t∥Vt⊤U_t \hat A_t^{\parallel} V_t^\top has rank rr. The full update of an n×mn \times m matrix therefore takes nmnm bits plus r(n+m)+r2r(n+m) + r^2 numbers, about 3 MB against 32 MB for a dense BF16 update at n=m=4096n = m = 4096 and r=64r = 64. This matters when the update or the model delta is sent over the network, for example in optimizer-state sharding, where each rank broadcasts the update of its own slice, and in local-update methods, where workers exchange model deltas over slower cross-node links. General compression methods reach a similar bandwidth only by approximating the update, usually with error feedback, while here the transmitted update is exactly the one the optimizer applies.

(b) The residual direction is informative about rounding. Given (a), we next ask what the residual direction does to the quantized weights, which the ablations cannot show. In our reply to Reviewer AZcL (Q, part (c)), starting from the final W2A16 checkpoint, we split each ResQAdam update into its principal and residual parts using its own bases, apply each part alone, and compare them with a random sign direction of the same norm as the residual part. We count level flips, i.e., weights whose quantized value changes, and the fraction of flips that reduce the loss. The residual part causes as many flips as the random direction (3.620e5 vs. 3.617e5 per unit update norm), but 64% of its flips reduce the loss, against 50% for the random direction and 53% for the principal part. The residual is not just a larger or differently scaled step. It points in a useful direction for rounding.

AuthorsOpenReview ↗

Response to Reviewer 6Qfn (2/3)

(c) The effect depends on the bit width. Finally, if the residual only reflected a general property of sign updates or low-rank optimization, its role should not change with the bit width. In our reply to Reviewer AZcL (Q, part (a)), we isolate the quantization-induced part of the momentum, D=Mq−MfpD = M^{q} - M^{fp}, which is exactly zero without quantization. The principal subspace keeps only 11.0% of DD at W2A16 and 34.0% at W4A4, so the part left to the residual grows as the bit width decreases.

W2: Why should the residual update's norm match the principal update's norm before energy weighting? How sensitive is performance to the normalization rule, particularly when the principal update has an unusually small norm?

A2: Thanks for raising this. We clarify that it contains three parts: 1) why the norms are matched, 2) how sensitive the results are to this rule, and 3) what happens when the principal update is small. We address them in turn.

1) Why we match the norms. There are two major reasons:

  1. Without matching, the residual step size would depend on the matrix shape rather than on the optimization state. For a matrix of size n×mn \times m, ∥sign(Mt⊥)∥F=nm\Vert \mathrm{sign}(M_t^{\perp}) \Vert_F = \sqrt{nm} at every step, regardless of the momentum magnitude, so larger matrices would take larger residual steps for no training-related reason.
  2. The residual has no second moment, so without matching it would need its own scale, which is an extra hyperparameter that interacts with the learning rate. Matching it to ∥Δt∥∥F\Vert \Delta_t^{\parallel} \Vert_F puts the residual on the scale of the Adam update in the principal subspace. The step size is then set by the Adam update, as in Adam, and the learning rate carries over.

2) Sensitivity to the rule. The closest test we have is the exponent sweep in our reply to W1, which changes the element-wise scaling of the residual while keeping the norm matching. Performance stays within 19.04–20.45 at W2A16 for all pp, and within 18.34–18.76 at W4A4 for p≤0.75p \le 0.75. Only p=1p = 1 fails at W4A4 (40.92, loss spike to 6.90 around step 3000).

3) When the principal update is small. Because UtU_t and VtV_t have orthonormal columns, ∥Δt∥∥F=αr∥A^t∥∥F\Vert \Delta_t^{\parallel} \Vert_F = \alpha_r \Vert \hat A_t^{\parallel} \Vert_F, where A^t∥=m^t∥/(v^t∥+ϵ)\hat A_t^{\parallel} = \hat m_t^{\parallel} / (\sqrt{\hat v_t^{\parallel}} + \epsilon) is the Adam update in the r×rr \times r space and αr=nm/r\alpha_r = \sqrt{nm}/r is fixed for each matrix. So ∥Δt∥∥F\Vert \Delta_t^{\parallel} \Vert_F is small only if all r2r^2 Adam ratios are small at the same time, which Adam's normalization prevents.

The measurements agree. Over two full 12288-step runs, with the learning rate divided out, ∥Δt∥∥F/∥W∥F\Vert \Delta_t^{\parallel} \Vert_F / \Vert W \Vert_F never falls below 2.05 at W2A16 or 2.38 at W4A4, with medians of 2.43 and 2.89. The 5th percentile is within 4% of the minimum, so there is no low tail.

Even if ∥Δt∥∥F\Vert \Delta_t^{\parallel} \Vert_F were small, the residual could not dominate. Δt⊥\Delta_t^{\perp} is rescaled to ∥Δt∥∥F\Vert \Delta_t^{\parallel} \Vert_F, so the ratio between the residual and principal terms is ρt⊥/ρt∥\rho_t^{\perp} / \rho_t^{\parallel}, which is set by the momentum energy and does not depend on ∥Δt∥∥F\Vert \Delta_t^{\parallel} \Vert_F. Since (ρt∥)2+(ρt⊥)2=1(\rho_t^{\parallel})^2 + (\rho_t^{\perp})^2 = 1, the full step satisfies ∥Δt∥F≤2∥Δt∥∥F\Vert \Delta_t \Vert_F \le \sqrt{2} \Vert \Delta_t^{\parallel} \Vert_F and shrinks together with the principal update.

AuthorsOpenReview ↗

Response to Reviewer 6Qfn (3/3)

W3: The paper provides meaningful experimental details but several choices interact such as momentum precision, subspace rank, refresh frequency, Adam states, residual normalization, energy-based fusion, and the initialization/warm-up procedure which need to be well documented.

A3: Thanks for this suggestion. We clarify that, in the original submission, these choices were spread across the method section, the experimental setup and the code. Below we list every setting related to the items the reviewer names, for both QAT paradigms.

itemsetting
Momentum precisionDense momentum MtM_t stored in BF16, with β1=0.9\beta_1 = 0.9. The projection UtUt⊤MtVtVt⊤U_tU_t^\top M_t V_tV_t^\top is computed in FP32 and rounded to BF16 before forming Mt⊥M_t^{\perp}. UtU_t, VtV_t, m∥m^{\parallel}, v∥v^{\parallel} are kept in the weight dtype
Subspace rankr=32r = 32 for QuEST and CAGE from scratch (Table 1). r=64r = 64 for the Llama2-297M runs (Table 4, the ablations and this response) and for QAT fine-tuning (Tables 2 and 3). Applied to every quantized linear weight (q, k, v, o, gate, up and down projections)
Refresh frequencyτ=400\tau = 400 steps from scratch and τ=200\tau = 200 in fine-tuning. The bases are the top-rr singular vectors of MtM_t (cast to FP32), from a randomized SVD with r+10r + 10 vectors and 3 power iterations
Adam statesm∥,v∥∈Rr×rm^{\parallel}, v^{\parallel} \in \mathbb{R}^{r \times r} per matrix, β=(0.9,0.95)\beta = (0.9, 0.95), ϵ=10−8\epsilon = 10^{-8}, with bias correction. At each refresh they are transported to the new bases with the LDAdam rule, not reset. No dense second moment is kept, and the residual has none. Embeddings, normalization layers and the LM head use AdamW with the same β\beta and ϵ\epsilon. LSQ step sizes use AdamW without weight decay
Principal updateΔt∥=αrUtA^t∥Vt⊤\Delta_t^{\parallel} = \alpha_r U_t \hat A_t^{\parallel} V_t^\top with αr=nm/r\alpha_r = \sqrt{nm}/r
Residual normalizationΔt⊥=∥Δt∥∥F sign(Mt⊥)/∥sign(Mt⊥)∥F\Delta_t^{\perp} = \Vert \Delta_t^{\parallel} \Vert_F \, \mathrm{sign}(M_t^{\perp}) / \Vert \mathrm{sign}(M_t^{\perp}) \Vert_F
Energy-based fusionρt∥=∥Mt∥∥F/∥Mt∥F\rho_t^{\parallel} = \Vert M_t^{\parallel} \Vert_F / \Vert M_t \Vert_F and ρt⊥=∥Mt⊥∥F/∥Mt∥F\rho_t^{\perp} = \Vert M_t^{\perp} \Vert_F / \Vert M_t \Vert_F, with ϵ\epsilon in the denominators, and Δt=ρt∥Δt∥+ρt⊥Δt⊥\Delta_t = \rho_t^{\parallel} \Delta_t^{\parallel} + \rho_t^{\perp} \Delta_t^{\perp}
Weight updateWt=(1−ηλ)Wt−1−ηΔtW_t = (1 - \eta\lambda) W_{t-1} - \eta \Delta_t, with λ=0.1\lambda = 0.1 in our codebase (Llama2-297M runs and Basic QAT). The other QAT frameworks follow their own codebases
Initialization and warm-upM0=0M_0 = 0 and m0∥=v0∥=0m^{\parallel}_0 = v^{\parallel}_0 = 0. U,VU, V come from a randomized SVD of the first gradient. From scratch, the refresh is suspended for the first 10% of training steps, because the momentum of a randomly initialized model changes quickly. In fine-tuning there is no suspension, and the bases are refreshed every τ\tau steps from the start

We have also added all of these settings to Appendix C in the revised paper. Regarding the interactions between these choices, the paper and this response already measure the following, all on the Llama2-297M model trained from scratch:

  • Rank rr (Section 5.4). r=4r = 4 is clearly worse (25.2 against 19.8 at r=64r = 64, W2A16). For r≥16r \ge 16, perplexity varies by at most 1.8 and not monotonically.
  • Refresh interval τ\tau (Section 5.4). Training converges for every τ\tau from 50 to 2000, and perplexity varies by about 1 with no consistent trend. τ=400\tau = 400 is within 0.9 of the best interval in every setting.
  • Source of the bases. Gradient-SVD Bases (GtG_t instead of MtM_t) is worse, most clearly under W2A2.
  • Principal and residual updates. Principal Update Only, Residual-Sign Only and Global-Momentum Sign (sign of the whole MtM_t) all fall short of the full method, so both the projected Adam update and the split contribute.
  • Residual direction and normalization. The exponent sweep in our reply to W1.

We would be happy to provide additional results or clarifications during the discussion if the reviewer has any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.