acceptodds
Under review as a conference paper at ICLR 2027

JacQuant: STE‑Free Quantization‑Aware Training via Learned Jacobian Surrogates

Abstract

Quantization-aware training (QAT) usually propagates gradients through hard quantizers with a fixed straight-through estimator (STE). At ultra-low precision, this backward rule ignores how quantized weights respond to finite perturbations. We introduce JacQuant, a QAT framework that learns a lightweight surrogate of quantizer sensitivity while retaining the hard forward quantizer. A periodic Gaussian probe fits one scalar gain per weight group; the gains are averaged over time and reused between refreshes. We identify the population regression target through Gaussian smoothing and separate it from the clipped estimator used in practice. For a local linearized surrogate, we establish non-convex stationarity bounds and linear convergence to an error neighborhood under a Polyak–Łojasiewicz condition, with explicit bias and variance terms. Across 1B–8B LLaMA and Qwen models with 1–2-bit weights, JacQuant improves both perplexity and mean zero-shot accuracy over the ParetoQ and WinQ baselines. On LLaMA3-1B W2A16, it lowers ParetoQ perplexity by 8.9% and recovers roughly one third of the gap to FP16 in both metrics. Gains persist at all three 8B checkpoints through 90K steps. In its default configuration, JacQuant incurs only a 1.6–3.5% reduction in training throughput on 1B models and 0.76% on 8B relative to the corresponding STE baselines. It requires no additional computation at inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Quantization-Aware Training, Large Language Models, Ultra-Low-Bit Quantization, Straight-Through Estimator, Learned Jacobian Surrogates, Gradient Estimation, STE-Free Quantization, Model Compression

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_tZfSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes JacQuant, an alternative to the straight-through estimator for ultra-low-bit quantization-aware training. Instead of using a fixed backward rule, JacQuant learns group-wise gradient scaling factors from the quantizer's responses to Gaussian weight perturbations, while preserving the hard quantizer in the forward pass. These factors are updated periodically and reused across training steps to limit computational overhead.

The paper also provides a theoretical analysis of the learned sensitivity estimates and conditional convergence guarantees for a local surrogate objective. Experiments on 1B–8B LLaMA and Qwen models show improvements in perplexity and zero-shot accuracy over ParetoQ and WinQ baselines, with relatively low reported training overhead and no additional inference cost.

Critical strengths
  1. The proposed method is easy to follow. JacQuant estimates quantizer sensitivity using lightweight probes and maintains only one scalar gain per weight group. The periodic update strategy also helps reduce the additional computational cost.

  2. Promising experimental results . The method is evaluated on different model families, model sizes, and quantization settings. The results show consistent improvements in perplexity and average zero-shot accuracy over the corresponding baselines.

Critical weaknesses
  1. Insufficient ablation studies. The main results in Table 1 combine sensitivity learning with variance reduction, making it difficult to assess the individual contribution of the proposed method. It is also unclear whether the Rotation Trick baseline uses the same variance-reduction setting.

  2. Limited discussion of the theoretical analysis. The convergence guarantees rely on a local surrogate with fixed quantization codes and parameters, while actual training uses AdamW with evolving codes and scales. The connection between the theoretical results and practical optimization remains unclear.

  3. Insufficient implementation details. Appendix E describes different variance-reduction strategies without specifying which is used in the main experiments. This limits reproducibility and makes it difficult to assess the reported training overhead in the main paper.

Questions for authors
  1. Could the authors provide a comparison of STE and JacQuant, both with and without variance reduction, under the same experimental setting, such as ParetoQ on LLaMA3-1B W2A16?

  2. Does the proposed method still show improvements compared with group-shuffled gains, where the gain distribution and refresh timing are kept the same at each step?

  3. Could the authors clarify which variance-reduction estimator is used for each quality and runtime result?

Reviewer_w3WEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduced a variation of STE for QAT. It adds learnable scalar parameters per group of weights, acting as scales for gradients of the respective weights. In other words, it's a form of (soft) gradient masking with adaptive mask.

Concretely, weights are divided in consecutive(?) groups of size gg. For each group its gradients get scaled by a learnable factor bib_i before being passed to the optimizer. bib_i are updated with lower frequency than the weights; they are also constrained to [0,1][0,1]. The update of bib_i is based on a scalar product between a Gaussian perturbation δi∼N(0,σ2I).\delta_i \sim N(0, \sigma^2 I). and a quantization error computed on that perturbation.

This modification adds memory overhead for the additional parameters bib_i and extra compute to update them. Extra costs only concern training phase, not inference.

The experimental evaluation involves training on FineWebEdu of LLaMa3 and Qwen3 family models. Authors do low-precision weight-only QAT starting from released full-precision model checkpoints. Main results are improvements over ParetoQ [1] and WinQ[2] methods in their respective setups. I believe this evidence is insufficient and many other methods need to be compared against.

Authors should demonstrate the gains of their method compared with works such as LC_QAT [3], Hestia [4], StableQAT [5], LOTION [6], QuEST[7]. Until then, the proposed method is not sufficiently supported by experiments.

[1] https://arxiv.org/abs/2502.02631

[2] https://arxiv.org/abs/2605.17471

[3] https://arxiv.org/abs/2606.10531

[4] https://arxiv.org/abs/2601.20745

[5] https://arxiv.org/abs/2601.19320

[6] https://arxiv.org/abs/2510.08757

[7] https://arxiv.org/abs/2502.05003

Critical strengths
  1. The main idea of the method is interesting.
Critical weaknesses
  1. Experimental results are unconvincing. Evaluation lacks comparison with many relevant works: LC_QAT [3], Hestia [4], StableQAT [5], LOTION [6]. The closest one methodologically would be QuEST[7].

PPL on validation set of WikiText-2 were used for both hparameters choice and the main results table. Held-out test set PPL should be reported instead.

  1. Authors do not separate the effects of gradient scaling, which is the core method, and variance reduction.
Questions for authors

None

Reviewer_U7v2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In this paper the authors propose a new approach, JacQuant, for quantization-aware training for ultra-low (1-2 bit weights, 8/16 bit activations) precision in LLMs. Particularly for low-precision, the typical approach to obtaining gradients through the quantization operator, the (clipped) straight-through estimator, does not reflect the actually relatively large fluctuations that can happen (parameters switching between hard quantization buckets) under relatively small perturbations. The authors propose to address this by scaling each group’s gradient by its sensitivity (in quantized space) to perturbations in the weights, using learned per-group gains applied to the gradient of the weights (a surrogate Jacobian). Sensitivities are learned by adding small Gaussian noise to the weights, quantizing the original and noised weights, then fitting a scalar (single least squares slope) to model the response (basically a count of the number of parameters in the group that switched to a different quantized value in response to the noise, weighted by the magnitude of the noise and normalized).

The authors derive bounds on convergence and bias on a smoothed local approximation of the quantized loss, showing that training with the proposed learned gain converges to within a neighborhood of a stationary point. In experiments on LLaMA3 (1B-8B) and Qwen3-1.7B, at 1, 1.58, and 2-bit weights and 8/16-bit activations, they show that the proposed approach consistently improves over standard STE in terms of WikiText-2 perplexity and mean zero-shot accuracy across eight language understanding benchmarks.

Critical strengths
  • Quantization is a nearly ubiquitous method for improving computational and memory efficiency in deployed neural networks such as LLMs. An efficient method for achieving high-quality (i.e. as lossless as possible) low-bit width quantization could have high impact in these settings.
  • The proposed approach is reasonably theoretically grounded alongside its empirical results.
Critical weaknesses
  • Experimental results don’t clearly separate the effect of the proposed approach (specifically, the surrogate gradient) from the STE baseline, or from noise. This is unfortunately buried in the appendix, but experiments using the proposed approach don’t just vary the backward surrogate, they also include additional variance reduction (e.g. SVRG) that isn’t performed for the baseline, and no control (adding VR to baseline STE) is done. Results are also largely reported over single seeds, while Table 9 shows similar differences in perplexity just from tuning hyperparameters. It’s not clear whether the reported improvements would go away if averaged over multiple runs.
  • As the appendix also points out, there are interactions between the optimizer and the proposed approach (fixed gains cancel out under moment normalization in AdamW, so improvements come from changes in the gain throughout training) that could be easily controlled but aren’t. This seems like an important result to include in the main paper in order to improve our understanding of the approach.
  • The paper is not very well written. It’s dense and makes a relatively simple approach (usually a good thing!) seem more complex than it is, and the approach is not sufficiently anchored in relation to prior work. More details/examples:
    • The paper lists as its first contribution “Learning the backward rule for ultra-low-bit QAT,” but that has been done before (e.g. MetaQuant (Chen et al., NeurIPS 2019), DSQ (Gong et al., ICCV 2019), both not cited), its the specific mechanism they’re proposing that is new, and that should be more clearly articulated. Currently the contribution is overstated.
    • Relatedly, there has been a ton of work on QAT, important related works are missing and for what is there, is it not clearly described how JacQuant fits in. Just describing the other methods and this method without the “why” is not useful. Well written related work should go beyond just summarizing the methods to give the reader a sense of when would I choose one method over the other and why (particularly for the most closely related methods, but also where the method sits among other work in the area more broadly.) Discussion of post-training quantization, on the other hand, is not needed, IMO (and can make room for the discussion that is needed of more closely related work).
    • Fig 1: less focus on the overall training setup (“task pass: every training step” part), which is fairly straightforward, spend more space/explanation in the figure on the “gain refresh” which is the actual approach being proposed (and the figure does not clearly breakdown/depict at all). Ideally, most of the figure would be devoted to conveying the proposed approach and its intuitive interpretation (i.e. the weighted, normalized sum of parameters that switch quantization buckets in response to the perturbation.) Because the figure and its caption are not sufficiently clear, there is no clear explanation of the approach in the intro that is sufficient to understand how it compares to the prior work that is discussed in the intro (e.g. the rotation trick, ParetoQ).
    • Terms used without being clearly introduced/defined (e.g. “codeword” line 075, yes this is a reasonable concept in quantization, but it hasn’t been used or relevant so far in this paper or defined; for context, the related work it’s being used to describe (from ICLR 2025) clearly introduces the interpretation/idea of a quantization fn as a codebook in the abstract and intro before then describing the method in that way.)
    • The paper refers to the proposed approach as “STE-free” but I’m not sure that’s an accurate characterization, since the proposed approach is really some scalar values multiplied by the STE “gradient”. Describing it as “STE-free” doesn’t really add any clarity for understanding the approach, and in fact just obscures/confuses the reader more, in my opinion.
    • I don’t find that the discussion/conclusion add much to the paper, so reducing or removing this section this could be a way to make room for the suggested improvements to related work, and including ablations and throughput results in the main paper.
Questions for authors
  • In appendix E, the authors state: “The main JacQuant configurations are reported as VR-enabled, so their results assess gain learning and control variates together. A matched four-way comparison of STE/JacQuant with and without the same control variate would isolate their respective contributions.” Similarly, in Appendix F: “Fixed- or shuffled-gain controls would distinguish response-dependent gain learning from other effects of time-varying gradient scaling.” I agree on both counts. Could you please run those experiments along with variance over multiple seeds for the main results and report, and/or explain why it was not included in the paper? This would help address weaknesses 1 and 2 above.
  • What configuration was used for the experiments in Tables 7 and 8, and is it the same one for which accuracy is reported in Tables 1 and 3-6 (i.e. VR-enabled)? If so, this would add an extra forward/backward per step, but the reported throughput reductions are only 1.6-5.2% on 1B and under 1% on 8B which is surprising.
Reviewer_f7gcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper addresses the fact that the straight-through estimator used in quantization-aware training ignores how a hard quantizer actually responds to weight changes.
  • It proposes JacQuant, which keeps the hard quantizer in the forward pass and scales the backward gradient by one learned gain per weight group, fitted from occasional random perturbations.
  • The method achieves small but consistent gains over two recent low-bit training pipelines on language models up to 8B parameters, with little extra training cost and no inference cost.
Critical strengths
  1. The method is a cheap drop-in change that improves results in every reported setting. It adds little training time and no inference cost,
Critical weaknesses
  1. The experiments do not show that the learned gains cause the improvement. The main results also turn on a variance-reduction estimator that the baselines do not use, and there is no fixed-gain or shuffled-gain control. With single seeds, many gains are about as large as the spread across the method's own hyperparameter settings.
Questions for authors
  1. How do STE with the same variance-reduction estimator and JacQuant without it perform?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.