- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ResQAdam combines adaptive updates in a low-rank subspace with a momentum-based residual update for quantization-aware training. It substantially reduces optimizer-state storage and achieves measurable peak-memory savings, though accuracy preservation varies across quantization settings.
Critical strengths
- Experiments cover several quantization recipes, bit widths, and model sizes, with ablations supporting the residual branch.
- Actual peak-memory and throughput measurements substantiate the efficiency benefits beyond optimizer-state accounting.
Critical weaknesses
- Although FiRA and FRUGAL were not originally evaluated for QAT, they also retain updates outside the low-rank subspace. A representative QAT comparison would help determine whether ResQAdam’s specific residual design offers benefits beyond simply retaining these updates.
- Accuracy preservation is uneven in demanding low-bit settings. For Qwen3-4B with LLM-QAT W2A16C, average perplexity increases from 33.84 with AdamW to 46.72, alongside a drop in accuracy.
Questions for authors
See weaknesses. The quantization-specific explanation is not established. The ablations show that residual updates are useful, but do not demonstrate that they specifically preserve quantization-compensation signals. A matched non-quantized control or direct analysis of quantization-induced changes would help substantiate this interpretation.
6 comments
Response to Reviewer AZcL (2/4)
W2: Accuracy preservation is uneven in demanding low-bit settings. For Qwen3-4B with LLM-QAT W2A16C, average perplexity increases from 33.84 with AdamW to 46.72, alongside a drop in accuracy.
A2: We clarify that the cited numbers are correct, and ResQAdam performs worse than AdamW in this setting. We would like to provide additional context from downstream task accuracy, where the observed differences are relatively modest despite the larger perplexity gap.
To show this, we compare two low-bit rows of Table 2. For reference, the zero-shot tasks (HellaSwag, PIQA, ARC, WinoGrande, MMLU) have an average chance level of about 35%.
| Model | Method | Q-Cfg | Avg. task acc. (AdamW / Ours) | Avg. ppl (AdamW / Ours) |
|---|---|---|---|---|
| Llama 3.2-1B | EfficientQAT | W2A16G128 | 35.14 / 35.43 | 351.78 / 379.52 |
| Qwen3 4B-Base | LLM-QAT | W2A16C | 40.85 / 38.02 | 33.84 / 46.72 |
For the cited Qwen3 setting, the downstream accuracy difference is 2.83 percentage points, compared with a perplexity increase from 33.84 to 46.72. Both models also exhibit substantial degradation relative to the 4-bit counterpart (69.29% accuracy), with their task accuracy approaching the chance level.
A similar pattern can be observed in the Llama 3.2-1B setting, where both optimizers achieve near-chance task accuracy (35.14% vs. 35.43%), while perplexity differs by 27.74. These observations suggest that when model performance is severely degraded under extreme low-bit quantization, perplexity differences can become more pronounced even when downstream accuracy differences remain relatively small.
Response to Reviewer AZcL (3/4)
Q: The quantization-specific explanation is not established. The ablations show that residual updates are useful, but do not demonstrate that they specifically preserve quantization-compensation signals. A matched non-quantized control or direct analysis of quantization-induced changes would help substantiate this interpretation.
A3: Thanks for the suggestion. To clarify, the ablations show that the residual is useful, but they do not link it to quantization. Following your suggestion, we have run three new analyses: (a) a direct analysis of the quantization-induced change in the momentum, (b) a matched full-precision control during training, and (c) a control at the level of single updates, which addresses the ablation question directly. Because of the length limit, (c) is in the next post.
The experiments for (a) and (b) have been added to Section 3.1 and Appendix E, while those for (c) are presented in Appendix D of the revised paper.
(a) Direct analysis of the quantization-induced change. We first measure how much of the quantization-induced signal a low-rank projection keeps.
Setup. We record two full ResQAdam runs (Llama2-297M, LSQ, G128, , 12288 steps, all 84 linear layers), at W2A16 and at W4A4. At every step we compute two gradients on the same mini-batch, with and without the weight quantizer, and , and accumulate each into its own momentum with the optimizer's , giving and . Their difference is the part of the momentum caused by weight quantization. It contains no mini-batch noise and is exactly zero in full precision. Training uses only , so the extra pass does not change the run. The bases are computed from as in ResQAdam, so we test the subspace the optimizer actually uses.
Metric. The share of kept by this subspace, . A projection-based optimizer keeps only this part of its update.
| step | in-sub(), W2A16 | in-sub(), W4A4 |
|---|---|---|
| 400 | 0.350 | 0.346 |
| 2048 | 0.233 | 0.295 |
| 6144 | 0.096 | 0.329 |
| 12288 | 0.110 | 0.340 |
At W2A16 the subspace keeps 11.0% of at the end of training, so a projection-only update discards 89% of it. At W4A4 it keeps 34.0%. The lower the bit width, the more of the quantization-induced signal lies outside the principal subspace. This is not caused by outdated bases: grouped by the number of steps since the last refresh, in-sub() stays within 0.102–0.119 at W2A16 and 0.272–0.289 at W4A4, so refreshing more often would not recover it. The residual-sign update keeps this part.
(b) Matched full-precision control during training. Next, we check whether quantization makes the momentum itself harder to capture with a low-rank subspace. We train four AdamW runs on Llama2-297M that differ only in the quantization setting: full precision, W4A16, W3A16 and W2A16. Every 200 steps, we take the AdamW first moment of the seven weight matrices in a middle layer and compute its residual energy where are the singular values of . This is the share of the momentum energy outside its best rank-64 approximation, so a projection-based optimizer with rank 64 discards at least this share. We report the mean over the seven matrices.
| training setting | step 5200 (mid-training) | step 10200 (last recorded step) |
|---|---|---|
| full precision | 0.350 | 0.340 |
| W4A16 | 0.368 | 0.364 |
| W3A16 | 0.381 | 0.375 |
| W2A16 | 0.414 | 0.418 |
The residual energy increases as the bit width decreases, and the full-precision run is the lowest at both steps. At step 10200, a rank-64 projection discards at least 34.0% of the momentum energy in full precision and 41.8% under W2A16. These results suggest that quantization error may increasingly spread into the residual subspace of the first moment as the bit width decreases, making low-rank projection less effective at capturing it.
Response to Reviewer AZcL (1/4)
Thanks for the valuable and constructive comments, which are definitely helpful for us to improve the quality of the paper. We have uploaded a revised version to accommodate all your suggestions and questions. Below, we further provide detailed clarifications for each of your concerns:
A1: Thanks for pointing this out. Following your suggestion, we have added Fira and FRUGAL in Table 4 under the same QAT settings. The results answer the question in two steps: keeping the residual accounts for most of the gain over projection-only methods, and the specific residual design of ResQAdam gives a further gain in all four settings at a small memory cost. We first give the setup and results, then explain which design choices give the extra gain, and finally compare memory.
Setup. We follow the other baselines in Table 4: Llama2-297M for from-scratch QAT and Llama 3.2-1B for QAT fine-tuning, an LSQ quantizer with group size 128, and the same data, training steps and projection rank (r=64). We use the official implementations. Fira uses its default α=0.25, and FRUGAL uses Adam in the subspace and signSGD outside it. The learning rate of every method is grid-searched and the best result is reported. Numbers are average perplexity on WikiText-2 and C4. The GaLore-AdamW column is taken from Table 4.
Keeping the residual gives most of the gain. Fira and FRUGAL close most of the gap left by the projection-only GaLore-AdamW, for example from 30.61 to 20.25 in from-scratch W2A16. This is consistent with the main observation of our paper: a large part of the useful QAT update lies outside the top-r subspace.
The residual design of ResQAdam adds a consistent gain on top. ResQAdam is best in all four settings, by 0.20 to 1.20 ppl over FRUGAL and 0.25 to 2.17 ppl over Fira. The gap is largest in from-scratch W4A4 and smallest in fine-tuning W4A4, where the quantization error is small.
Where the difference comes from. The three methods differ in what the residual is computed from:
Two ablations already in the paper (Section 5.1) test these choices. Both keep the residual and change only one design choice, so they test the design rather than whether to keep the residual. Gradient-SVD Bases computes the bases from Gt instead of Mt. Global-Momentum Sign applies the sign to the whole Mt without splitting it into principal and residual parts. Both underperform ResQAdam, and Gradient-SVD Bases is clearly worse under W2A2.
Memory. The extra gain comes at a small memory cost. Fira and FRUGAL keep the same state as GaLore-AdamW, with a state-free residual, while ResQAdam additionally keeps one BF16 momentum.
With 0.46 GB (from scratch) and 0.67 GB (fine-tuning) more optimizer state than Fira and FRUGAL, ResQAdam matches AdamW in WikiText-2 perplexity, while using 25% and 31% of AdamW's optimizer-state memory.