acceptodds
Under review as a conference paper at ICLR 2027

QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction

12 views

Abstract

As large language model inference shifts toward lower precision, post-training quantization (PTQ) becomes increasingly brittle, making quantization-aware training (QAT) essential for preserving model quality. However, QAT has a structural mismatch: gradient updates are applied to latent full-precision weights, while the loss and gradients are computed on lossy reconstructions of those weights. This mismatch can lead to suboptimal training trajectories and a higher loss floor. Second-order PTQ methods address a similar problem by minimizing loss-aware reconstruction error, but applying such expensive reconstruction repeatedly during QAT as the weights evolve is impractical. We introduce QUASAR, a QAT method that brings lightweight, loss-aware reconstruction into the training loop. At each training step, QUASAR reconstructs the latent weights by searching over a small set of clipping ranges and fitting dequantization parameters through saliency-weighted least squares. We use an exponential moving average of squared gradients as the per-parameter saliency signal. Our theoretical analysis shows that optimizing QUASAR's reconstruction objective tightens both the convergence and final-loss bounds of QAT. We evaluate QUASAR across four model families and across INT4, INT3, INT2, and NVFP4 quantization formats. QUASAR consistently achieves lower training and evaluation loss than competitive QAT methods and outperforms QAT and PTQ baselines on downstream benchmarks. At INT2, QUASAR improves average accuracy over the best QAT baseline by 13.3 points with quantization-aware distillation and by 10.9 points with QAT on mathematical reasoning data. Notably, after distillation on only 600M tokens, QUASAR's INT4 Gemma-4 E4B checkpoint outperforms the corresponding QAT checkpoint released by Google, with 66% lower KL divergence and 1.8 points higher average accuracy.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
quantization-aware training, large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_iGqXOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes QUASAR, a quantization-aware training (QAT) method designed to lower the lower bound of training loss for large language models after low-bit quantization. QUASAR introduces a loss-aware weight reconstruction mechanism. Theoretical analysis shows that under certain assumptions, reducing the reconstruction error can tighten the convergence and final loss upper bounds of QAT. The experiments cover a variety of large language models as well as INT2, INT3, INT4 and NVFP4 quantization formats.

Critical strengths

This paper includes reasonably extensive experiments, while its theoretical explanations are concise.

Critical weaknesses
  1. When constructing the loss-aware reconstruction objective, the paper assumes that the first-order term can be neglected. However, this assumption is mostly valid only when the model is near convergence and may not hold in the early stage of training or during task adaptation. As a result, the proposed reconstruction objective may fail to accurately capture the true variation of quantization loss. The paper also lacks sufficient validation for the effectiveness of this assumption across different training stages.
  2. The paper employs the exponential moving average (EMA) of squared Adam gradients to approximate the Hessian diagonal and measure weight importance. Nevertheless, this statistic is not equivalent to the true second-order curvature and ignores mutual influences among different weights. Therefore, such an approximation may not precisely quantify how quantization error affects model loss, and its reliability requires more thorough verification.
Questions for authors

see weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer iGqX (1/2): W1

Many thanks to the reviewer for the thoughtful review, and for appreciating our extensive experiments and concise theory. Following the reviewer's comments, we have updated the submission with new analyses, which we point to below.

W1. When constructing the loss-aware reconstruction objective, the paper assumes that the first-order term can be neglected. However, this assumption is mostly valid only when the model is near convergence and may not hold in the early stage of training or during task adaptation. As a result, the proposed reconstruction objective may fail to accurately capture the true variation of quantization loss. The paper also lacks sufficient validation for the effectiveness of this assumption across different training stages.

There are two reasons to make the simplifying assumption that the gradient is null, even though during training it is technically not zero.

First, because QAT follows extensive pre-training, the weights already sit near an optimum when QAT starts. QAT usually comes as the last stage of a model's life, right before it is served: it follows pre-training, and often fine-tuning and reinforcement learning, and it is very short compared with pre-training (Liu et al., 2024; Gemma Team, 2025; Liu et al., 2025). As a consequence, the QAT model remains close to the converged full-precision model in the loss landscape: QAT does not create a new model, but explores the low-bit configurations around the full-precision one. To illustrate this empirically, in our Qwen3-4B runs, the latent weights move by only 0.9–3.4% in relative norm, ∥wT−w0∥/∥w0∥\lVert w_T - w_0\rVert / \lVert w_0\rVert (from INT4 to INT2).

Second, QUASAR uses Eq. 2 to compare various reconstructions rr of the same latent weights ww, and ultimately pick the optimal one. We observe empirically that across candidates rr, the loss can vary widely. For instance, at INT2 on Qwen3-4B, the loss of 9 candidate reconstructions varies by up to 5.4 nats (considering the quantity max⁡rL(r)−min⁡rL(r)\max_r \mathcal{L}(r) - \min_r \mathcal{L}(r)). When we decompose this variation, we notice that at most 6% of the variation is attributable to the first-order term. This also holds at the other bit widths, where the first-order term accounts for at most 3% (INT3) and 1% (INT4), respectively. As such, the first-order term does not influence the optimal reconstruction picked, although it is technically not exactly zero. Another way to verify this: if we rank the candidates by their loss with and without the first-order term, both rankings pick the same reconstruction at all 12 checkpoints we tested (3 bit widths × 4 points from early to late training, steps 32 to 4,096). The same holds during task adaptation (Qwen3-4B-Base adapted to math, INT2), where the first-order term accounts for less than 1% of the variation and both rankings again pick the same reconstruction. All in all, the first order term does not influence the choice of the optimal loss-aware reconstruction.

Thank you for this question: we added this analysis to the manuscript (Table C.1, Appendix C).

AuthorsOpenReview ↗

Response to Reviewer iGqX (2/2): W2

W2. The paper employs the exponential moving average (EMA) of squared Adam gradients to approximate the Hessian diagonal and measure weight importance. Nevertheless, this statistic is not equivalent to the true second-order curvature and ignores mutual influences among different weights. Therefore, such an approximation may not precisely quantify how quantization error affects model loss, and its reliability requires more thorough verification.

Adam's second moment is indeed not the true curvature, and a better estimate does give a lower final loss. But the better estimate is expensive, while Adam's is free because the optimizer already keeps it (Li et al., 2025) and correlates well with more expensive Hessian estimations (Figure 15).

We conducted the following experiment: we compare QUASAR at INT4 using 3 different curvature estimations (uniform saliency, Adam’s second moment as saliency, and finally saliency derived from a separate, one-off Hessian estimation on the full-precision model), keeping other experimental settings the same. The setting is QAD with the Qwen3-4B model, trained for 4,096 steps on OpenPerfectBlend. The better estimate we use is the curvature of the full-precision model's output, estimated once before training from the full-precision weights w⋆w^\star:

H=∇w2,Ex,KL(pw⋆(⋅∣x),∣,pw(⋅∣x))∣w=w⋆.H = \nabla_w^2 \\, \mathbb{E}_x \\, \mathrm{KL}\big(p_{w^\star}(\cdot \mid x) \\,\\|\\, p_w(\cdot \mid x)\big)\Big|_{w = w^\star}.

This estimate is better than Adam's in two ways: it is the curvature of the KL to the full-precision model itself (rather than a running average of squared gradients; Martens, 2020) and it keeps the interactions between the weights of a layer (Martens & Grosse, 2015), which Adam's lighter per-weight estimate ignores. Kronecker-factored curvature is already used to compress pretrained models (Wang et al., 2019; van der Ouderaa et al., 2024). Indeed, estimating HH requires calibration data and backward passes through the full-precision model. For each layer, HH is approximated by Kronecker factors, H≈Hout⊗HinH \approx H_{\text{out}} \otimes H_{\text{in}}, which turns the loss-aware error of a layer with weights WW and reconstruction RR into tr(Hout(R−W),Hin(R−W)⊤)\mathrm{tr}\big(H_{\text{out}}(R-W)\\,H_{\text{in}}(R-W)^\top\big).

Curvature estimateHeld-out KLExtra cost
WorseUniform weighting (no curvature)0.0167none
QUASARAdam's second moment0.0157none
BetterHessian of the full-precision model, HH0.0146Before training: 1.9 GPU-hours to collect HH, 52 GB to store it. During training: +3.6 GB memory per GPU

All in all, the better Hessian estimate yields the lowest final KL loss but incurs substantial computational and memory overhead. QUASAR’s use of Adam’s second moment as a saliency estimate provides roughly half of that KL improvement without adding computational or memory overhead.

We added this experiment to the manuscript (Table A.1, Appendix A).

We thank the reviewer again! We hope this answers the reviewer's questions. If it does, we would be grateful if the reviewer would consider raising their score, and we are happy to discuss further.

AuthorsOpenReview ↗

Follow-up on our responses and additional analyses

Dear Reviewer iGqX,

Thank you again for your thoughtful feedback. We wanted to briefly follow up on the additional analyses we provided in response to your concerns: the first-order-term analysis (across training stages and during task adaptation; Table C.1 in the revised submission), and the comparison of curvature estimates (Table A.1).

When you have a chance we would appreciate hearing whether these results address your concerns or whether further clarification would be helpful. We would be happy to discuss any remaining questions.

Thank you for your time and consideration.

Reviewer_x6ZbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes QUASAR, a quantization-aware training (QAT) method that incorporates loss-aware weight reconstruction into the training process. The approach combines clipping-range search with saliency-weighted least-squares fitting, using the exponential moving average of squared gradients to approximate parameter importance. The authors also provide theoretical analysis linking reconstruction error to QAT convergence and final loss.

Critical strengths
  1. The paper addresses a relevant problem in low-bit LLM quantization, and the motivation for improving weight reconstruction during QAT is reasonable.
  2. The experimental evaluation covers multiple model architectures, quantization formats, and downstream benchmarks, with substantial improvements observed under INT2 quantization.
  3. The proposed method is relatively efficient and can be integrated into existing QAT pipelines without additional inference overhead.
Critical weaknesses
  1. The proposed approach mainly combines existing techniques, including clipping-range search, weighted least-squares reconstruction, and gradient-based saliency estimation. The conceptual advancement beyond existing quantization optimization methods is not sufficiently compelling.
  2. The ablation study in Table 8 indicates that most improvements originate from clipping-range search. In particular, uniform fitting achieves a KL divergence of 0.114, compared with 0.112 for the proposed saliency-weighted fitting. This marginal difference raises questions about the necessity and effectiveness of the central loss-aware weighting mechanism.
  3. The analysis relies on relatively strong assumptions, particularly the global condition and the positive saliency lower bound. These assumptions are difficult to justify for large-scale LLM optimization, weakening the connection between the theoretical results and practical training behavior.
  4. While the gains at INT2 are significant, the improvements under INT4 are considerably smaller, with several downstream results comparable to existing methods. Moreover, the comparison with Google's QAT checkpoint involves different quantization group sizes and undisclosed training budgets, making it difficult to isolate the benefits of the proposed method. The efficiency evaluation is also limited to a single model and quantization configuration.
Questions for authors
  1. Given the marginal improvement of saliency-weighted fitting over uniform fitting in Table 8, what is the essential contribution of the proposed loss-aware weighting mechanism beyond conventional clipping-range optimization?
  2. How do the authors justify the practical relevance of the global PL assumption and the positive saliency lower bound for large-scale LLM training?
  3. Why are the improvements under INT4 substantially smaller than those under INT2? Can the authors clarify the conditions under which QUASAR provides meaningful advantages over existing QAT methods?
  4. How should the comparison with Google's QAT checkpoint be interpreted given the differences in quantization configurations and the unavailable information about the baseline's training budget?
5 comments
AuthorsOpenReview ↗

Response to Reviewer x6Zb (1/4): W1

We are grateful to the reviewer for the detailed feedback, and for noting that QUASAR is efficient and adds no inference overhead. The reviewer's questions led us to run new experiments, which we have added to the submission; we mention each addition in the relevant answer. We respond to each weakness raised and question below.

W1. The proposed approach mainly combines existing techniques, including clipping-range search, weighted least-squares reconstruction, and gradient-based saliency estimation. The conceptual advancement beyond existing quantization optimization methods is not sufficiently compelling.

Thank you for this insightful comment. While the individual tools QUASAR relies on (e.g. clipping-range search or gradient-based saliency estimation) are not new, we would like to argue that our main contribution and novelty lie elsewhere. Our key contribution is to show that, during QAT, a better reconstruction, done in a loss-aware way and at every step, leads to (1) a lower training loss floor, (2) a lower held-out loss, and (3) better performance on downstream benchmarks. To our knowledge, QUASAR is the first method to apply loss-aware reconstruction at every step of QAT.

Most existing QAT research has focused on aspects of quantization other than reconstruction, such as propagating gradients through rounding using the straight-through estimator or learning quantization parameters (e.g., LSQ). In contrast, loss-aware reconstruction has largely been studied in post-training quantization (PTQ), where it is applied once to a frozen model (e.g., GPTQ). Whether loss-aware reconstruction provides similar benefits during QAT has remained unclear. In particular, it could reduce the loss early in training without improving the final converged loss. Our results show that its benefits extend beyond early training. Theorem 1 establishes that the loss floor is controlled by the loss-aware reconstruction error. Consistent with this result, QUASAR achieves lower training and held-out losses than all QAT baselines across both evaluated models and every bit width (Figure 3). Furthermore, reconstruction error strongly predicts final loss (R2=0.98R^2=0.98; Figure 13).

We designed QUASAR so that loss-aware reconstruction is efficient enough to be able to run at every step of QAT. Using Adam's second moment as the saliency needs no calibration data, no additional backward passes and no additional optimizer state, and the scale search runs in parallel on the GPU, adding only about 1% overhead per step over Standard QAT (see W4 below for further details).

AuthorsOpenReview ↗

Response to Reviewer x6Zb (2/4): W2/Q1

W2. The ablation study in Table 8 indicates that most improvements originate from clipping-range search. In particular, uniform fitting achieves a KL divergence of 0.114, compared with 0.112 for the proposed saliency-weighted fitting. This marginal difference raises questions about the necessity and effectiveness of the central loss-aware weighting mechanism.

Q1. Given the marginal improvement of saliency-weighted fitting over uniform fitting in Table 8, what is the essential contribution of the proposed loss-aware weighting mechanism beyond conventional clipping-range optimization?

Thank you very much for this question. We have extended our ablation to all bit widths. We report the full ablation results below.

In the following table, we show the impact of the loss-aware weighting alone, the clipping search alone, and both combined (QUASAR), for Qwen3-4B during QAD using the OpenPerfectBlend dataset (new runs):

Held-out KL vs Standard QATINT2INT3INT4
Loss-aware weighting only (no clipping search)−7.7%−8.3%−11.3%
Clipping search only (uniform weighting)−34.7%−20.2%−8.1%
QUASAR (both)−35.3%−22.1%−13.8%

We find that at INT2 and INT3, the clipping search contributes more, while at INT4, the loss-aware weighting contributes more. Both ingredients are complementary across bitwidths.

In a second set of experiments below, we report the increase in downstream accuracy from the loss-aware weighting, as opposed to uniform weighting (clipping search being unchanged):

\text{Uniform weighting} \to \text{QUASAR} & \text{INT2} & \text{INT3} & \text{INT4} \\\\ \hline \text{Average accuracy}^{1} & 42.4 \to \mathbf{43.4} & 62.5 \to \mathbf{63.5} & 68.2 \to \mathbf{68.6} \\\\ \text{Long-context retrieval (RULER)} & 56.7 \to \mathbf{62.7} & 82.0 \to \mathbf{85.3} & 86.7 \to \mathbf{87.8} \end{array}$$ ¹ Average over the 5 tasks of Table 1: MATH-500, SuperGPQA, LiveCodeBench v6, LongBench-v2 and RULER. *We added this ablation to the manuscript (Table E.2, Appendix E).*
AuthorsOpenReview ↗

Response to Reviewer x6Zb (3/4): W3/Q2

W3. The analysis relies on relatively strong assumptions, particularly the global condition and the positive saliency lower bound. These assumptions are difficult to justify for large-scale LLM optimization, weakening the connection between the theoretical results and practical training behavior.

Q2. How do the authors justify the practical relevance of the global PL assumption and the positive saliency lower bound for large-scale LLM training?

Due to the complexity and highly non-convex nature of LLM training, assumptions are necessary to derive theoretical guarantees, and our analysis is no exception. But the assumptions we use check out empirically, and they are on par or milder than those of prior theory of quantized training. Finally, our main convergence result doesn’t require the global PL condition or the positive saliency lower bound.

  • Empirical checks. Given the gap between the idealized assumptions and the actual dynamics of large-scale LLM training, we ran extensive empirical checks of the theory on real training runs (Appendix C), and all of them check out:
    • the gradient mismatch between the reconstructed and the latent weights grows with the loss-aware error, as Assumption 3 states. This is the case for every bit width (Figure 14);
    • along the training trajectory, the squared gradient norm grows with the remaining loss gap, as the PL inequality states (Figure 16);
    • Adam's second moment tracks the actual curvature of each layer (Figure 15);
    • the bound of Theorem 1 (part i) is within ~3× of the observed squared gradient norm, and the reconstruction error is its largest term (Figure 17);
    • as the theorem predicts, the reconstruction error predicts the final KL: it tracks the final KL across methods and bit widths (Figure 10), and restricting QUASAR's search raises both together (Figure 12). QUASAR also keeps its advantage under plain SGD, the actual setting of the theory (Figure 18).
  • Our assumptions are on par or milder than prior theory. PL, which we use only for the final-model bound, does not require convexity and is weaker than strong convexity (Karimi et al., 2016). On the contrary, prior theory of quantized training assumes more:
AssumptionPrior theory of quantized training (Li et al., 2017; Hou et al., 2019)Theorem 1
Convex lossrequired by bothnot required
Strongly convex lossrequired in some results (Li et al., 2017)not required (PL, which is weaker, only for part ii)
Bounded gradientsrequired by bothnot required
Weights confined to a bounded setrequired (Hou et al., 2019)not required
  • Results we have without these assumptions. Convergence (Theorem 1, part i) uses neither PL nor the saliency lower bound. Indeed, PL is used only for the final-model bound (part ii), and the saliency lower bound is only one sufficient condition for Assumption 3, that the gradient mismatch is bounded by the loss-aware error (Appendix B.1, which is confirmed empirically).

Our goal is to use relatively light assumptions, that check out empirically, to explain, with an interpretable theoretical result, why loss-aware reconstruction lowers the loss floor of QAT.

References

  • Karimi, Nutini and Schmidt. Linear convergence of gradient and proximal-gradient methods under the Polyak–Łojasiewicz condition. ECML PKDD, 2016.
  • Li, De, Xu, Studer, Samet and Goldstein. Training quantized nets: A deeper understanding. NeurIPS, 2017.
  • Hou, Zhang and Kwok. Analysis of quantized models. ICLR, 2019.
Reviewer_h1R3OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies quantization-aware training of large language models. Its main motivation is that QAT updates full-precision weights using gradients evaluated at their quantized reconstructions, which can be inaccurate. Therefore, this paper proposes to leverage a second-order approximation around the latent weight to obtain a quantized reconstruction as close to the latent weight as possible, a similar idea to existing PTQ methods. This paper first uses the diagonal Fisher information matrix as an approximation for the Hessian. Then, at each step, the method searches clipping ranges for each weight group and fits dequantization parameters by weighted least squares, using AdamW's second moment to weight reconstruction errors.

Additionally, this paper shows a theorem that bounds QAT convergence and final quantized loss in terms of this error under stated assumptions.

In experiments, this paper compares the proposed method with Standard QAT, LSQ, Denoising QAT, BitDistiller, and post-training quantization baselines. The experiments cover Qwen, Llama, and Gemma models in INT2–4 and NVFP4, using distillation and math adaptation. At INT2 precision, average accuracy improves by 13.3% for Qwen distillation and 10.9% for math adaptation over the best QAT baselines.

Critical strengths
  • This paper considers reducing the reconstruction error (i.e., the loss of the quantized reconstruction minus the loss at latent weight) at each step of QAT using the second-order approximation technique and searches for the best scales and clip values.
  • This paper shows that reducing this reconstruction error improves the convergence of QAT and downstream accuracy of the final quantized models. The INT2 results are strong across two training settings. The proposed method improves average accuracy by 13.3 points on Qwen distillation and 10.9 points on math adaptation over the best QAT baselines.
Critical weaknesses
  • The main question I have for this paper is the validity of the second-order approximation (in Equation 2). The assumption of the second-order approximation is that (1) the latent weight is near a stationary point and (2) the distance between the quantized and latent weight is small. However, this is likely not true in quantization-aware training, especially for lower bits (less than or equal to 2-bit). The first-order term can still be quite large, and the latent weight is not a local minimum. I wonder how the authors can justify this point for their method design.

  • Following the above question, I wonder whether the performance improvement comes from making the quantized reconstruction closer to the latent weight.

  • Another main question is how well the approximation matches the full Hessian matrix. This paper uses se the diagonal Fisher information matrix as a proxy for the Hessian. I wonder how accurate this approximation is, and how this approximation error affects the training (for example, if we have a worse/better approximation of the Hessian, does QAT become better?)

  • The full training algorithm is unclear from the reading. This paper mainly discusses the quantization step. It would be great to have a summary or overview of the final training method.

Questions for authors
  • If the method needs to search for the quantization parameters (like scaling and clips) at each training step, what are the final quantization parameters used for inference?
  • How much does saliency weighting improve downstream accuracy over per-step clipping search with uniform weights at INT2, INT3, and INT4?
5 comments
AuthorsOpenReview ↗

Response to Reviewer h1R3 (1/3): W1, W2

We thank the reviewer for the careful review, and for highlighting our strong INT2 results. Taking the reviewer's comments into account, we have updated the submission in several places, and each answer below points to the corresponding change. We address the weaknesses and questions one by one.

W1. The main question I have for this paper is the validity of the second-order approximation (in Equation 2). The assumption of the second-order approximation is that (1) the latent weight is near a stationary point and (2) the distance between the quantized and latent weight is small. However, this is likely not true in quantization-aware training, especially for lower bits (less than or equal to 2-bit). The first-order term can still be quite large, and the latent weight is not a local minimum. I wonder how the authors can justify this point for their method design.

There are two reasons to make the simplifying assumption that the gradient is null, even though during training it is technically not zero.

First, because QAT follows extensive pre-training, the weights already sit near an optimum when QAT starts. QAT usually comes as the last stage of a model's life, right before it is served: it follows pre-training, and often fine-tuning and reinforcement learning, and it is very short compared with pre-training (Liu et al., 2024; Gemma Team, 2025; Liu et al., 2025). As a consequence, the QAT model remains close to the converged full-precision model in the loss landscape: QAT does not create a new model, but explores the low-bit configurations around the full-precision one. To illustrate this empirically, in our Qwen3-4B runs, the latent weights move by only 0.9–3.4% in relative norm, ∥wT−w0∥/∥w0∥\lVert w_T - w_0\rVert / \lVert w_0\rVert (from INT4 to INT2).

Second, QUASAR uses Eq. 2 to compare various reconstructions rr of the same latent weights ww, and ultimately pick the optimal one. We observe empirically that across candidates rr, the loss can vary widely. For instance, at INT2 on Qwen3-4B, the loss of 9 candidate reconstructions varies by up to 5.4 nats (considering the quantity max⁡rL(r)−min⁡rL(r)\max_r \mathcal{L}(r) - \min_r \mathcal{L}(r)). When we decompose this variation, we notice that at most 6% of the variation is attributable to the first-order term. This also holds at the other bit widths, where the first-order term accounts for at most 3% (INT3) and 1% (INT4), respectively. As such, the first-order term does not influence the optimal reconstruction picked, although it is technically not exactly zero. Another way to verify this: if we rank the candidates by their loss with and without the first-order term, both rankings pick the same reconstruction at all 12 checkpoints we tested (3 bit widths × 4 points in training). All in all, the first order term does not influence the choice of the optimal loss-aware reconstruction.

Thank you for this question: we added this analysis to the manuscript (Table C.1, Appendix C).

W2. Following the above question, I wonder whether the performance improvement comes from making the quantized reconstruction closer to the latent weight.

QUASAR performs best when the reconstruction minimizes the loss-aware error, as opposed to the plain error.

Indeed, there are two ways to measure how close a reconstruction rr is to the latent weights ww:

  • plain error ∥r−w∥2\lVert r-w\rVert^2, where every weight counts the same;
  • loss-aware error ∑ihi (ri−wi)2\sum_i h_i\,(r_i-w_i)^2, where weights that matter more for the loss count more (here, hh is still Adam's curvature estimate).

In the table below, we train Qwen3-4B for 4,096 steps both using the plain error and using the loss-aware error, across bit widths (new runs). We observe that training the model with the loss-aware error, which is QUASAR's default reconstruction, results in a lower held-out KL.

Trained withPlain error measured (×10⁻⁶)Loss-aware error measured (×10⁻⁶)Held-out KLHeld-out KL vs plain error
INT2
Plain error63.864.70.11398—
Loss-aware error (QUASAR)66.161.40.11290−0.9%
INT3
Plain error18.418.60.04822—
Loss-aware error (QUASAR)19.317.50.04703−2.5%
INT4
Plain error4.864.880.01669—
Loss-aware error (QUASAR)5.134.620.01566−6.2%

At every bit width, training with the plain error gives the lower plain error, but QUASAR gives the lower loss-aware error and reaches the lower final KL loss.

We added this comparison to the manuscript (Table E.1, Appendix E).

AuthorsOpenReview ↗

Response to Reviewer h1R3 (2/3): W3, W4

W3. Another main question is how well the approximation matches the full Hessian matrix. This paper uses the diagonal Fisher information matrix as a proxy for the Hessian. I wonder how accurate this approximation is, and how this approximation error affects the training (for example, if we have a worse/better approximation of the Hessian, does QAT become better?)

We find that a better estimate results in a lower final loss, which confirms that curvature matters. But the better estimate below is expensive, while Adam's is free because the optimizer already keeps it (Li et al., 2025) and correlates well with more expensive Hessian estimations (Figure 15).

We conducted the following experiment: we compare QUASAR at INT4 using 3 different curvature estimations (uniform saliency, Adam’s second moment as saliency, and finally saliency derived from a separate, one-off Hessian estimation on the full-precision model), keeping other experimental settings the same. The setting is QAD with the Qwen3-4B model, trained for 4,096 steps on OpenPerfectBlend. The better estimate we use is the curvature of the full-precision model's output, estimated once before training from the full-precision weights w⋆w^\star:

H=∇w2,Ex,KL(pw⋆(⋅∣x),∣,pw(⋅∣x))∣w=w⋆.H = \nabla_w^2 \\, \mathbb{E}_x \\, \mathrm{KL}\big(p_{w^\star}(\cdot \mid x) \\,\\|\\, p_w(\cdot \mid x)\big)\Big|_{w = w^\star}.

This estimate is better than Adam's in two ways: it is the curvature of the KL to the full-precision model itself (rather than a running average of squared gradients; Martens, 2020) and it keeps the interactions between the weights of a layer (Martens & Grosse, 2015), which Adam's lighter per-weight estimate ignores. Kronecker-factored curvature is already used to compress pretrained models (Wang et al., 2019; van der Ouderaa et al., 2024). Indeed, estimating HH requires calibration data and backward passes through the full-precision model. For each layer, HH is approximated by Kronecker factors, H≈Hout⊗HinH \approx H_{\text{out}} \otimes H_{\text{in}}, which turns the loss-aware error of a layer with weights WW and reconstruction RR into tr(Hout(R−W) Hin(R−W)⊤)\mathrm{tr}\big(H_{\text{out}}(R-W)\,H_{\text{in}}(R-W)^\top\big).

Curvature estimateHeld-out KLExtra cost
WorseUniform weighting (no curvature)0.0167none
QUASARAdam's second moment0.0157none
BetterHessian of the full-precision model, HH0.0146Before training: 1.9 GPU-hours to collect HH, 52 GB to store it. During training: +3.6 GB memory per GPU

All in all, the better Hessian estimate yields the lowest final KL loss but incurs substantial computational and memory overhead. QUASAR’s use of Adam’s second moment as a saliency estimate provides roughly half of that KL improvement without adding computational or memory overhead.

We added this experiment to the manuscript (Table A.1, Appendix A).

W4. The full training algorithm is unclear from the reading. This paper mainly discusses the quantization step. It would be great to have a summary or overview of the final training method.

This is a great suggestion, thank you very much. We have added a second algorithm to the manuscript (Algorithm 2), right below the QUASAR algorithm, which shows how the QUASAR reconstruction blends into a complete, end-to-end training step.

AuthorsOpenReview ↗

Response to Reviewer h1R3 (3/3): Q1, Q2

Q1. If the method needs to search for the quantization parameters (like scaling and clips) at each training step, what are the final quantization parameters used for inference?

After the very last training step, we run the reconstruction one last time on the final weights and keep the result. This produces a final scale and a final offset for each group of weights, and the final reconstructed weights, which are used at inference. These codes, scales and offsets are saved in the same standard format as min–max rounding, so nothing is searched at inference time.

While our main algorithm fits a scale and an offset per group, QUASAR extends to other formats by fitting only what each format stores. For instance, our Gemma-4 INT4 checkpoint uses groups of 64 weights with one scale each and no offset.

Finally, we notice that by end of training the choice has settled: over the last 10% of training, at least 99.89% of weight groups keep the same clipping range.

Q2. How much does saliency weighting improve downstream accuracy over per-step clipping search with uniform weights at INT2, INT3, and INT4?

Using saliency weighting, as opposed to uniform weighting, improves average accuracy at every bit width, especially on long-context tasks. Using QUASAR's loss-aware weighting as opposed to uniform weighting (scale search unchanged) gives the following results:

\text{Uniform weighting} \to \text{QUASAR} & \text{INT2} & \text{INT3} & \text{INT4} \\\\ \hline \text{Average accuracy}^{1} & 42.4 \to \mathbf{43.4} & 62.5 \to \mathbf{63.5} & 68.2 \to \mathbf{68.6} \\\\ \text{Long-context retrieval (RULER)} & 56.7 \to \mathbf{62.7} & 82.0 \to \mathbf{85.3} & 86.7 \to \mathbf{87.8} \end{array}$$ ¹ Average over the 5 tasks of Table 1: MATH-500, SuperGPQA, LiveCodeBench v6, LongBench-v2 and RULER. We thank the reviewer again! We hope this answers the reviewer's questions. If it does, we would be grateful if the reviewer would consider raising their score, and we are happy to discuss further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.