- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies the weight distribution dynamics during QAT. They observe that while the loss for the quantized model improves during QAT, the loss for the full-precision model gets worse. They analyze the residual of weight changes and raise a new perspective that "signal imbalance is a source of asymmetric oscillations that favor weaker-gradient endpoints." From that analysis, they propose QAR, a method for QAT that favors strong signals and uses a smaller memory footprint. On three models, they achieve similar loss and accuracy while using less memory compared to the traditional QAT method.
Critical strengths
- I think formalizing the weight dynamics during QAT is informative for some communities. I see that this paper belongs to the optimization community and could potentially be interesting to people with more theoretical interests.
- A memory-efficient QAT method will be useful for many applications. QAT is necessary for some downstream tasks. But on non-industry-level GPUs, memory is indeed a big issue. QAR will be useful to enable some workflows on, for example, personal computers.
Critical weaknesses
- The paper is quite hard to follow. I would suggest making the main text a coherent story and introducing the analysis through words instead of equations and symbols. I feel like you don't need to talk about all the assumptions and modeling in the main text, as it just confuses people and gets worse with a lot of definitions/terminologies that are not commonly used in other literature (see my questions below, for example).
- The evaluation to confirm the efficiency of QAR is not complete. Table 2 shows that QAR saves memory, but we don't know if it saves time. The sampling and code manipulation in the QAR might introduce extra latency that is not disclosed in the current experiments. Also, the current PTQ method is RTN only. From my experience, I don't remember the degradation of PPL/accuracy being that big for 4-bit weight-only quantization for those better methods, and I wonder: a. how QAR works together with different PTQ methods and b. how QAR works with weight+activation quantization.
Questions for authors
Below are some questions that I seek clarification on regarding the paper:
- Do I understand correctly that "signals" are weight magnitudes? Strong signals mean big weights, and weak signals mean small weights.
- Do I understand correctly that QAR is more memory-efficient than standard QAT in that they do not work on full-precision weights, but instead works on quantized weights only? So the benefit over QAT is that QAR does not need to store a copy of the shadow weight.
Below are some technical questions:
-
The paper establishes that QAR-0 recovers DQT under the stated conditions. However, the main experiments compare QAR primarily against conventional QAT. Could the authors provide a direct comparison between QAR-0 and QAR-(gamma) on three LLMs, using comparable hyperparameter tuning budgets?
-
What are QAR's actual computational costs relative to QAT? Please report the time used to train QAT vs. QAR.
-
If we change the PTQ method, what happens to the two QAT model variants?
5 comments
Response to Reviewer GddK (2/3)
W2 & Q5: Other PTQ methods.
"If we change the PTQ method, what happens to the two QAT model variants?"
Thank you for this suggestion. The revision adds GPTQ as a PTQ baseline for every model at 4 and 8 bits (Table 1, three calibration sets).
The two kinds of training also start from different points. QAT keeps a full-precision shadow weight, so it starts from the pretrained weights and uses the local information they carry. DQT and QAR-γ are endpoint training: they keep no shadow weight and update the quantized weights directly. On the other hand, this lets them start directly from a PTQ solution and leverage what the PTQ method has already achieved, so in the revision both start from GPTQ and keep improving it. The comparison at the same start and the same memory is therefore QAR-γ against DQT (Q3). On Llama-2-7B at 4 bits, for example, GPTQ reaches a C4 loss of 2.209, and QAR-γ lowers it to 2.135; at 4 bits QAR-γ improves on its GPTQ start in C4 loss, WikiText-2 loss, and zero-shot accuracy on all four models.
Appendix B.9 compares an RTN and a GPTQ start on Qwen-2.5-0.5B, each method at its best learning rate (Table B.9, tuning C4 loss, three seeds):
| RTN, 4 bits | GPTQ, 4 bits | RTN, 8 bits | GPTQ, 8 bits | |
|---|---|---|---|---|
| PTQ only (no training) | 3.682 | 3.215 | 2.994 | 2.992 |
| DQT | 3.299 ± 0.002 | 3.177 ± 0.003 | 2.977 ± 0.000 | 2.976 ± 0.000 |
| QAR-γ | 3.258 ± 0.003 | 3.157 ± 0.001 | 2.976 ± 0.001 | 2.976 ± 0.001 |
A better start helps, most clearly at 4 bits. At 4 bits QAR-γ improves on DQT from either start (by 0.042 from RTN and 0.020 from GPTQ); at 8 bits the two are within 0.001.
W2: Weight and activation quantization.
"… and b. how QAR works with weight+activation quantization."
Our work focuses on weight-only quantization: the deployed weights live on a fixed grid, and the full-precision shadow weight is the memory cost that QAR removes. The Limitations name joint weight-activation quantization as a direction for extending the endpoint view.
QAR only changes how the weights are updated, so it combines directly with an activation quantizer in the forward pass. As a quick check, we quantized the input of every quantized linear layer to 8 bits (dynamic symmetric scale per token, straight-through gradients, the same quantization at evaluation) and trained Qwen-2.5-0.5B with 4-bit weights at the learning rates of Table B.2, with QAT started from the full-precision weights and QAR-γ from GPTQ, as in Table 1. We show the W4A8 evaluation loss here:
| C4 | WikiText-2 | |
|---|---|---|
| QAT | 3.361 | 3.161 |
| QAR-γ | 3.318 | 3.118 |
With 8-bit activations, QAR-γ keeps a lead of about 0.04 over QAT in both C4 and WikiText-2 loss. Lower-bit activations usually call for rotations or smoothing against activation outliers, such as QuaRot; these act on the activations and can be combined with QAR's weight update.
Response to Reviewer GddK (3/3)
W1: Readability of the main text.
"I would suggest making the main text a coherent story and introducing the analysis through words instead of equations and symbols."
Thank you for this suggestion. We would like to clarify the storyline of the paper and summarize the changes we made for readability.
The main text follows one storyline, where each step leads to the next (the new Contributions and roadmap paragraph of the Introduction gives the section or theorem behind each contribution):
- Observation: the deployed loss falls while the shadow loss rises (Section 1, Fig. 1). During QAT, the shadow loss can rise substantially while the deployed loss falls. Existing explanations look at the shadow weights and cannot account for this, so we turn to the deployed weights.
- Mechanism: residuals turn STE updates into one-cell moves (Section 3). To see how the deployed weights move, we track the residual of each shadow weight inside its cell. A straight-through estimator (STE) step smaller than a cell moves the deployed weight by at most one cell, and when this residual is close to uniform, the probability of the move is set by the size of the update. This one-cell mechanism is the basis for the rest of the paper.
- Signal imbalance motivates us to favor stronger signal for descent (Section 4.1). Through the same mechanism, we find that QAT tends to settle at endpoints with weak gradients: a weak signal lets the shadow drift slowly across a boundary, while a strong signal pulls it back quickly. This suggests giving strong signals more transitions, which is the idea behind our algorithm.
- Algorithmic framework: QAR (Sections 4.2 and 4.3). We therefore sample the one-cell moves directly on the quantized weights, without a shadow weight, and amplify the transitions of strong signals. QAR comes with convergence guarantees on the finite grid (Theorem 4).
- Evidence (Section 4.4). Experiments on four LLMs, including a 16B mixture-of-experts model, confirm this picture: at 4 and 8 bits, QAR-γ matches or outperforms QAT, and at 4 bits it stores 27–29% less persistent memory.
Changes for readability in the revision:
- The contributions in the Introduction are stated in more detail, each with the section or theorem where it is developed.
- Section 3 now explains in words how the residual phase follows from the pretrained weights.
- The notation in Sections 2 and 4 is streamlined.
- Terms such as the signal of a weight are defined in plain words (Section 4.1).
- The experiments state their findings in plain language.
We keep the two assumptions and the two crossing models in the main text because each of them carries a step of this storyline:
- Two classic assumptions, used in the main results. Assumption 1 (bounded update direction) and Assumption 2 (coordinate-wise smoothness, standard in sign-gradient analyses such as signSGD) are the standing assumptions of the analysis. Proposition 2, Theorem 3, and Theorem 4 all rest on them, so we state them once in Section 2 and the main results refer back to them.
- Two models that form one pipeline. Model 1 states the crossing law in general form, the probability that a step moves a deployed weight by one cell, and yields the endpoint descent bound (Theorem 3). Model 2 is its special case under a near-uniform residual, where this probability becomes . QAR with the uniform amplifier samples exactly this law, and the power amplifier reweights it toward strong signals (Section 4.2), so Model 2 is the bridge from the analysis of QAT to the algorithm.
- The residual analysis is kept short. The main text gives the residual phase in words, and its derivation is in Appendix C.
We would also welcome more specific suggestions: if there are parts of Sections 2–4 that you would prefer to see moved to the appendix or rephrased, we are glad to revise them accordingly.
We hope these answers address your concerns and are happy to clarify further.
References
[1] Zhao, K., Tabaru, T., Kobayashi, K., Honda, T., Yamazaki, M., and Tsuruoka, Y. Direct Quantized Training of Language Models with Stochastic Rounding. ACML, 2025.
[2] Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR, 2023.
[3] Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs. NeurIPS, 2024.
[4] Bernstein, J., Wang, Y.-X., Azizzadenesheli, K., and Anandkumar, A. signSGD: Compressed Optimisation for Non-Convex Problems. ICML, 2018.
Response to Reviewer GddK (1/3)
We thank Reviewer GddK for the review and the helpful questions. We address your concerns below and changes in the revised PDF are marked in green.
Q1: Are "signals" weight magnitudes?
The signal of a weight is the size of its gradient (more precisely, of the optimizer update) at the deployed quantized weight, not the size of the weight. A strong signal means a large update, regardless of the size of the weight. Section 4.1 defines the signal as the update magnitude.
Q2: Is the memory benefit that QAR stores no shadow weight?
Yes. QAR stores the integer codes, scales, and optimizer moments but no full-precision shadow weight, which saves 3 to 3.5 bytes per quantized weight at 4 and 8 bits compared with FP32-shadow QAT (Appendix B.10). In our measurements, this lowers the persistent memory at 4 bits by 27–29% on all four models (Table 2).
Q3: QAR-0 vs QAR-γ on the LLMs with comparable tuning budgets.
Table 1 now includes DQT on all four LLMs, tuned on the same grid with three seeds; the γ ablation (Appendix B.6, Fig. 12) also includes γ = 0, which coincides with tuned DQT at these learning rates. C4 loss at 4 bits:
W2 & Q4: Training time of QAT vs QAR.
Table 2 now reports the persistent memory and the time per training step. To make the comparison fair, we give QAT the same memory-saving implementation as QAR-γ: it applies a fused AdamW step to each weight during the backward pass, which by itself lowers QAT's peak memory by 8–22%. On DeepSeek-MoE-16B, both methods update all experts of a layer in one batched call, and the clock stops after all GPUs finish. QAR-γ compared with QAT at 4 bits: