- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes QUASAR, a quantization-aware training (QAT) method designed to lower the lower bound of training loss for large language models after low-bit quantization. QUASAR introduces a loss-aware weight reconstruction mechanism. Theoretical analysis shows that under certain assumptions, reducing the reconstruction error can tighten the convergence and final loss upper bounds of QAT. The experiments cover a variety of large language models as well as INT2, INT3, INT4 and NVFP4 quantization formats.
Critical strengths
This paper includes reasonably extensive experiments, while its theoretical explanations are concise.
Critical weaknesses
- When constructing the loss-aware reconstruction objective, the paper assumes that the first-order term can be neglected. However, this assumption is mostly valid only when the model is near convergence and may not hold in the early stage of training or during task adaptation. As a result, the proposed reconstruction objective may fail to accurately capture the true variation of quantization loss. The paper also lacks sufficient validation for the effectiveness of this assumption across different training stages.
- The paper employs the exponential moving average (EMA) of squared Adam gradients to approximate the Hessian diagonal and measure weight importance. Nevertheless, this statistic is not equivalent to the true second-order curvature and ignores mutual influences among different weights. Therefore, such an approximation may not precisely quantify how quantization error affects model loss, and its reliability requires more thorough verification.
Questions for authors
see weakness
3 comments
Response to Reviewer iGqX (2/2): W2
W2. The paper employs the exponential moving average (EMA) of squared Adam gradients to approximate the Hessian diagonal and measure weight importance. Nevertheless, this statistic is not equivalent to the true second-order curvature and ignores mutual influences among different weights. Therefore, such an approximation may not precisely quantify how quantization error affects model loss, and its reliability requires more thorough verification.
Adam's second moment is indeed not the true curvature, and a better estimate does give a lower final loss. But the better estimate is expensive, while Adam's is free because the optimizer already keeps it (Li et al., 2025) and correlates well with more expensive Hessian estimations (Figure 15).
We conducted the following experiment: we compare QUASAR at INT4 using 3 different curvature estimations (uniform saliency, Adam’s second moment as saliency, and finally saliency derived from a separate, one-off Hessian estimation on the full-precision model), keeping other experimental settings the same. The setting is QAD with the Qwen3-4B model, trained for 4,096 steps on OpenPerfectBlend. The better estimate we use is the curvature of the full-precision model's output, estimated once before training from the full-precision weights :
This estimate is better than Adam's in two ways: it is the curvature of the KL to the full-precision model itself (rather than a running average of squared gradients; Martens, 2020) and it keeps the interactions between the weights of a layer (Martens & Grosse, 2015), which Adam's lighter per-weight estimate ignores. Kronecker-factored curvature is already used to compress pretrained models (Wang et al., 2019; van der Ouderaa et al., 2024). Indeed, estimating requires calibration data and backward passes through the full-precision model. For each layer, is approximated by Kronecker factors, , which turns the loss-aware error of a layer with weights and reconstruction into .
| Curvature estimate | Held-out KL | Extra cost | |
|---|---|---|---|
| Worse | Uniform weighting (no curvature) | 0.0167 | none |
| QUASAR | Adam's second moment | 0.0157 | none |
| Better | Hessian of the full-precision model, | 0.0146 | Before training: 1.9 GPU-hours to collect , 52 GB to store it. During training: +3.6 GB memory per GPU |
All in all, the better Hessian estimate yields the lowest final KL loss but incurs substantial computational and memory overhead. QUASAR’s use of Adam’s second moment as a saliency estimate provides roughly half of that KL improvement without adding computational or memory overhead.
We added this experiment to the manuscript (Table A.1, Appendix A).
We thank the reviewer again! We hope this answers the reviewer's questions. If it does, we would be grateful if the reviewer would consider raising their score, and we are happy to discuss further.
Follow-up on our responses and additional analyses
Dear Reviewer iGqX,
Thank you again for your thoughtful feedback. We wanted to briefly follow up on the additional analyses we provided in response to your concerns: the first-order-term analysis (across training stages and during task adaptation; Table C.1 in the revised submission), and the comparison of curvature estimates (Table A.1).
When you have a chance we would appreciate hearing whether these results address your concerns or whether further clarification would be helpful. We would be happy to discuss any remaining questions.
Thank you for your time and consideration.
Response to Reviewer iGqX (1/2): W1
Many thanks to the reviewer for the thoughtful review, and for appreciating our extensive experiments and concise theory. Following the reviewer's comments, we have updated the submission with new analyses, which we point to below.
W1. When constructing the loss-aware reconstruction objective, the paper assumes that the first-order term can be neglected. However, this assumption is mostly valid only when the model is near convergence and may not hold in the early stage of training or during task adaptation. As a result, the proposed reconstruction objective may fail to accurately capture the true variation of quantization loss. The paper also lacks sufficient validation for the effectiveness of this assumption across different training stages.
There are two reasons to make the simplifying assumption that the gradient is null, even though during training it is technically not zero.
First, because QAT follows extensive pre-training, the weights already sit near an optimum when QAT starts. QAT usually comes as the last stage of a model's life, right before it is served: it follows pre-training, and often fine-tuning and reinforcement learning, and it is very short compared with pre-training (Liu et al., 2024; Gemma Team, 2025; Liu et al., 2025). As a consequence, the QAT model remains close to the converged full-precision model in the loss landscape: QAT does not create a new model, but explores the low-bit configurations around the full-precision one. To illustrate this empirically, in our Qwen3-4B runs, the latent weights move by only 0.9–3.4% in relative norm, ∥wT−w0∥/∥w0∥ (from INT4 to INT2).
Second, QUASAR uses Eq. 2 to compare various reconstructions r of the same latent weights w, and ultimately pick the optimal one. We observe empirically that across candidates r, the loss can vary widely. For instance, at INT2 on Qwen3-4B, the loss of 9 candidate reconstructions varies by up to 5.4 nats (considering the quantity maxrL(r)−minrL(r)). When we decompose this variation, we notice that at most 6% of the variation is attributable to the first-order term. This also holds at the other bit widths, where the first-order term accounts for at most 3% (INT3) and 1% (INT4), respectively. As such, the first-order term does not influence the optimal reconstruction picked, although it is technically not exactly zero. Another way to verify this: if we rank the candidates by their loss with and without the first-order term, both rankings pick the same reconstruction at all 12 checkpoints we tested (3 bit widths × 4 points from early to late training, steps 32 to 4,096). The same holds during task adaptation (Qwen3-4B-Base adapted to math, INT2), where the first-order term accounts for less than 1% of the variation and both rankings again pick the same reconstruction. All in all, the first order term does not influence the choice of the optimal loss-aware reconstruction.
Thank you for this question: we added this analysis to the manuscript (Table C.1, Appendix C).