acceptodds
Under review as a conference paper at ICLR 2027

JARQ: Joint Alternating Refinement for Quantization

Abstract

Group-wise post-training quantizers for large language models round weights onto a grid that is not refit to the resulting integer codes. We show that this leaves accuracy on the table: the best grid depends on the codes, input correlations couple the errors of different groups, and useful code changes often involve many codes at once. We propose JARQ (Joint Alternating Refinement for Quantization), a plug-in refinement that starts from any group-wise quantizer and alternates a joint least-squares fit of all group scales with bounded Babai proposals that move many codes of a group together on the current grid. The problem is a bilinear box-constrained mixed-integer least-squares problem; the solver is backpropagation-free, does not increase the layer-wise objective under exact scale solves, and keeps the host's bit width, groups, zero points, and inference cost. Across Llama-2, Llama-3, and Qwen models with RTN, GPTQ, OmniQuant, and AWQ hosts, JARQ lowers perplexity in 90 of 96 comparisons, cuts three-bit RTN perplexity by up to 36%, raises mean multiple-choice accuracy in 23 of 24 configurations, and improves QEP, QuaRot, and OJBKQ outputs, at under a minute per 7B block.

Primary area:
optimization
Keywords:
Quantization, Optimization, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_pFKtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work introduces a joint alternating refinement method for standard integer quantizers, which can serve as a drop-in replacement for conventional integer quantization in a PTQ framework. According to the reported results, the proposed method generally improves the performance of existing PTQ frameworks, although in some cases it yields negligible or even negative gains. However, the integration details with existing PTQ frameworks remain unclear. For instance, OmniQuant already jointly optimizes the quantization scale and offset through block-wise learning. Since the proposed method also refines the quantization scale, it is unclear how the two optimization procedures are combined without conflict. Moreover, several important baselines are missing. In particular, HQQ [1] also adopts an alternating optimization scheme for quantization scales and zero-points, while COMQ [2] directly performs joint optimization of the quantization scale and discrete quantization codes. Besides, this work formulates quantized code optimization as a Babai nearest-plane problem and solves it in a group-wise manner. This is closely related to GPTQ, which has recently been shown to be equivalent to Babai’s nearest-plane algorithm under an appropriate formulation [3]. However, the paper does not discuss or compare against this connection. Therefore, the paper requires further improvement.

[1] Badri, H. and Shaji, A. Half-quadratic quantization of large machine learning models, November 2023. URL https://mobiusml.github.io/hqq_blog/.

[2] Zhang A, Yang Z, Wang N, et al. Comq: A backpropagation-free algorithm for post-training quantization[J]. IEEE Access, 2025, 13: 99879-99891.

[3] Chen, Jiale, et al. "The geometry of llm quantization: Gptq as babai's nearest plane algorithm." International Conference on Learning Representations. Vol. 2026. 2026.

Critical strengths
  1. The proposed joint refinement method is designed as a generic replacement for standard integer quantization, making it potentially applicable to a wide range of existing PTQ frameworks.
  2. In most reported cases, the proposed method further improves the performance of the compared PTQ approaches.
Critical weaknesses
  1. The paper does not clearly explain how the proposed refinement is integrated into existing PTQ frameworks such as OmniQuant, which already optimizes quantization scales and offsets, or GPTQ, which has recently been shown to be equivalent to Babai’s nearest-plane algorithm. This makes it unclear whether the proposed optimization complements, replaces, or potentially overlaps with the original optimization procedures.
  2. The comparison with closely related prior work is also incomplete. Methods such as HQQ [1] and COMQ [2] are not included, and the relationship between the proposed group-wise Babai code optimization and GPTQ’s interpretation as Babai’s nearest-plane algorithm [3] is not sufficiently discussed, making the methodological novelty difficult to assess.

[1] Badri, H. and Shaji, A. Half-quadratic quantization of large machine learning models, November 2023. URL https://mobiusml.github.io/hqq_blog/.

[2] Zhang A, Yang Z, Wang N, et al. Comq: A backpropagation-free algorithm for post-training quantization[J]. IEEE Access, 2025, 13: 99879-99891.

[3] Chen, Jiale, et al. "The geometry of llm quantization: Gptq as babai's nearest plane algorithm." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors
  1. The characterization of HQQ in the related work seems somewhat incomplete, since HQQ also performs iterative refinement through half-quadratic alternating optimization rather than simply optimizing the zero point once. Could the authors clarify the conceptual and algorithmic difference between HQQ’s alternating refinement and JARQ’s proposed alternating optimization?
  2. Could you directly compare GPTQ+HQQ with GPTQ+JARQ under matched settings? Please keep the other experimental conditions as controlled as possible, including the initialization, bit width, group size, and calibration data.
  3. Although K=3K=3 is chosen because it captures most of the perplexity improvement in the Llama-2 experiments, what happens when KK is further increased for downstream tasks? Does the calibration objective continue to decrease while held-out or downstream performance begins to degrade due to overfitting to the calibration data?
Reviewer_dMmrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents JARQ, a plug‑in post‑training quantization refinement module for group‑wise LLM PTQ pipelines. The authors identify three key failure modes in existing group‑wise quantizers: grid staleness, cross‑group coupling, and single‑code stationarity, where rounding onto a fixed grid does not jointly re‑optimize quantization scales and integer codes. JARQ formulates the layer‑wise reconstruction task as a bilinear box‑constrained mixed‑integer least‑squares problem, and uses a backpropagation‑free alternating solver: it jointly least‑squares‑fits all per‑group scales given current integer codes, then applies bounded Babai nearest‑plane proposals to update multiple codes within each group together on the updated grid. JARQ reuses the host quantizer’s bit‑width, grouping configuration and zero‑points, adding no inference‑time overhead.

Critical strengths
  1. JARQ can wrap arbitrary existing group‑wise PTQ outputs without changing inference specifications. It provides a monotonic descent guarantee for layer‑level reconstruction loss under exact scale solves, with formal propositions explaining why refitting quantization grids unlock improvements unreachable by pure fixed‑grid integer search.
  2. The authors evaluate across diverse model families (Llama‑2/3, Qwen‑2.5, Qwen‑3 including 32B scale), multiple host quantizers covering naive rounding and state‑of‑the‑art Hessian‑ / activation‑aware PTQ, both 3‑bit and 4‑bit weight quantization.
Critical weaknesses
  1. While local per‑module reconstruction error almost always improves after JARQ refinement, propagated block‑level error can occasionally degrade for certain host‑model combinations (e.g. some OmniQuant configurations). Although the paper notes this phenomenon and points to QEP‑style correction as a potential remedy, JARQ’s main experiments use standard layer‑target reconstruction without propagation‑aware loss, and there is no built‑in mechanism within JARQ itself to suppress such propagated error drift. This creates risk that perplexity improvements do not always fully translate to end‑task gains for certain setups.
  2. Even with K=3 iterations, JARQ adds significant wall‑clock overhead for model quantization (≈30+ minutes for full Llama‑2‑7B). Additionally, the ridge regularization fallback for rank‑deficient scale systems breaks the monotonic descent guarantee; the paper only uses a fixed ridge strength (10−410^{-4}) without ablation studying alternative stabilization strategies.
Questions for authors
  1. When ridge regularization is triggered for ill‑conditioned Gram matrices in scale fitting, the monotonic descent guarantee no longer holds. Have you observed concrete cases in your experiments where ridge stabilization increases layer reconstruction loss? Could you provide guidance on when practitioners should expect this risk, and practical heuristics to mitigate it?
  2. Your layer‑wise objective optimizes towards the full‑precision layer input, not accounting for quantization errors coming from preceding layers. You show that this can lead to occasional increases in propagated block‑output error. Could you quantitatively compare JARQ performance when using QEP‑style propagation‑corrected targets versus your standard target across more of your main evaluation set, not just limited‑scope compatibility tests?
  3. In your ablations you fix maximum iteration budget K=3K=3. Are there lightweight early‑stopping criteria (beyond your current per‑channel residual threshold) that could reliably reduce average runtime while retaining most of JARQ’s accuracy gains across diverse hosts and models?
4 comments
AuthorsOpenReview ↗

Response to Comments on Numerical Stability and Propagation-Aware Reconstruction

We sincerely thank the reviewer for the positive assessment of JARQ's theoretical contributions, general applicability, and experimental evaluation. We also appreciate the thoughtful suggestions regarding numerical stability, error propagation, and computational efficiency.

Q1. Ridge Stabilization and Monotonic Descent

Thank you for raising this important question. We conducted an additional numerical audit across 8 configurations (Llama-2-7B/Qwen3-8B × RTN/GPTQ × W3/W4).

Among 33,128,444 scale solves, ridge stabilization was triggered only 59 times (0.000178%), all in Qwen3-8B/GPTQ/W3, blocks 2–5. Importantly, none of these updates increased the original reconstruction objective.

All 59 cases involved groups with zero centered integer-code vectors, which provides a structural explanation. The scale-fitting system includes a positive diagonal contribution proportional to ν2∥ci∥2\nu^2\|c_i\|^2, where ν=0.6\nu=0.6 is the weight-reconstruction regularization parameter. When ci=0c_i=0, the corresponding scale is unidentifiable. In practice, the risk is therefore confined to groups whose centered codes are all zero, which we observed only at three bits.

As discussed in Remark 1, a simple safeguard is to keep a ridge-regularized scale update only when it does not increase the original objective and to retain the previous scales otherwise. With this safeguard, the monotonicity guarantee (Theorem 1(i) and Corollary 1) holds without the exact-solve assumption; only Part (iii) still requires exact scale solves. Alternatively, one can retain the scales of inactive zero-code groups and solve the reduced least-squares system. We will clarify these safeguards and the empirical stability results in the revision.

Q2. Propagation-Aware Reconstruction

We appreciate the reviewer highlighting the distinction between local reconstruction and propagated block-output error. We first clarify that JARQ's standard target is X~W\widetilde{X}W rather than XWXW: each layer is fitted on the inputs it actually receives from the quantized preceding layers. We conducted additional experiments to examine the choice of reconstruction target and its interaction with propagation-aware quantization.

Direct comparison of reconstruction targets. Following Appendix E.5, we compared the standard target (μ=1\mu=1, used throughout the paper) with the propagation-aware interpolated target (μ=0.5\mu=0.5, corresponding to QEP's default correction strength), while keeping the host initialization, calibration data, group size (G128), and iteration budget (K=3) fixed.

The experiments cover three models (Llama-3-8B, Qwen2.5-7B, Qwen3-8B), two hosts (GPTQ and OmniQuant), and both W3A16 and W4A16. The standard target (μ=1\mu=1) achieves lower WT2 and C4 perplexity in all 24 model–host–bit-width–corpus comparisons, with absolute PPL differences ranging from 0.32 to 10.02.

For example, on Qwen3-8B/GPTQ/W3, WT2 PPL is 10.54 with the standard target (Table 1), compared with 13.24 with μ=0.5\mu=0.5. These controlled results provide empirical support for the standard reconstruction objective adopted in JARQ. We will include the full ablation in the revision.

Compatibility with propagation-aware quantization. Importantly, JARQ can directly complement QEP. As demonstrated in Table 3(a), applying JARQ after QEP+GPTQ improves WT2 perplexity in all four settings:

ModelW/AQEP+GPTQ+ JARQ
Qwen2.5-7BW3A167.677.58
Qwen2.5-7BW4A167.027.00
Qwen3-8BW3A1611.0010.52
Qwen3-8BW4A169.949.88

This establishes that propagation-aware correction in the host stage and JARQ refinement are complementary.

Propagated error under the standard target. Our existing experiments further show that JARQ improves local reconstruction in 92.9% of 7,616 block–module comparisons on held-out WT2/C4 inputs. It reduces final block-output error in 23/32 configurations: 8/8 for RTN, 7/8 for GPTQ, 6/8 for AWQ, and 2/8 for OmniQuant. OmniQuant already optimizes block outputs, leaving less room for further improvement.

Together, these results support the standard μ=1\mu=1 objective while demonstrating that JARQ can be effectively combined with propagation-aware host quantization.

AuthorsOpenReview ↗

Response to Comments on Computational Efficiency and Early Stopping

Q3. Early Stopping and Computational Efficiency

Thank you for this suggestion. We agree that reducing preprocessing time would further improve JARQ's practicality.

JARQ introduces a one-time offline refinement cost. With K=3K=3, it adds approximately 53–57 seconds per Llama-2-7B block, and a complete Llama-2-7B RTN+JARQ run, including calibration, takes approximately 32 minutes. Importantly, JARQ does not change the quantized representation or introduce any inference-time overhead.

Early stopping is already part of JARQ. As shown in Appendix G.5, the per-channel rule stops a column at the latest one iteration after it reaches the fixed point of Theorem 1, and earlier when the decrease falls below the threshold. A further lightweight criterion is to terminate a channel as soon as a complete Babai code-update sweep produces no accepted integer-code updates, which removes that final iteration using acceptance information already computed during refinement. Under exact deterministic scale solves, this state is a scale-optimal, Babai-stable fixed point, consistent with Theorem 1.

We thank the reviewer again for the constructive feedback. We will improve the presentation of numerical stability conditions, propagation-error analysis, and computational trade-offs to make these aspects clearer.

AuthorsOpenReview ↗

Follow-up on our response and revised manuscript

Dear Reviewer dMmr,

We have uploaded a revised manuscript including the ridge audit, target ablation, and early-stopping clarifications discussed in our response. The changes are marked in blue.

We would appreciate it if you could take another look at our response and the revised manuscript when you have a chance. Please let us know if any concerns remain. Thank you again for your time.

Thanks for the detailed responses, most of my comments are addressed. I will maintain my positive score.

Reviewer_f4nCOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper starts with the key insight that the quantization grid and code should be jointly optimized due to grid staleness, cross-group coupling, and single-code stationarity. The paper then shows a layer-wise objective, a weighted sum of output activation reconstruction error and weight reconstruction error. To optimize for the layer-wise objective, the authors propose JARQ, a method that alternatively optimizes the grid (e.g. scale) and the code. A least-squares problem is solved to update scales. For every group, a Babai nearest proposal is computed to (possibly) update all codes. The theoretical analysis shows that JARQ makes non-zero progress until a fixed point is reached, that the fixed point can be reached in finitely many iterations, and that JARQ cannot do worse than its host (scale and code at initialization time) in terms of the layer-wise objective. Empirical results suggest JARQ can generally improve model performance. Ablation studies validate the key insights.

Critical strengths

The key insight that the grid and code must be jointly optimized, along with the three reasons/observations (grid staleness, cross-group coupling, and single code stationarity) are novel. The paper is easy to read. The proposed method is simple and elegant. The empirical results are thorough and convincing.

Critical weaknesses

None.

Questions for authors

Here are some questions I want to ask out of curiosity. Answering is optional. Your answers will not materially affect my assessment.

  1. While the layer-wise objective is useful, it attempts to minimize reconstruction error in every element (output activation/weight) equally. Are there any trivial extensions of JARQ that enable the use of saliency metrics? For example, the SqueezeLLM paper uses the diagonal of the empirical Fisher as a way to specify some elements are more important than others. If such an extension is reserved for future work, what do you think is the main theoretical/practical challenge?
  2. To update integer codes given fixed scales, have you considered trivial baselines like learning/updating the codes using Adam/Muon with gradients obtained using the Straight Through Estimator? An advantage is you can jointly update all groups at the same time instead of going through each group sequentially. I understand Babai has strong theoretical guarantees. However, I think it is possible that you get better results in practice with Adam/Muon. This does not violate the spirit of JARQ as you are still doing joint alternating refinement (least squares + GD as opposed to least squares + Babai).
2 comments
AuthorsOpenReview ↗

Response to Reviewer f4nC: Saliency Metrics and STE-Based Code Updates

We sincerely thank the reviewer for the careful reading and encouraging assessment. We are glad that the three observations and the motivation for jointly refining grids and codes came across clearly. We address the two questions below.

Q1. Saliency metrics. Our objective is a regularized layer-wise reconstruction loss, which already weights weight errors by X~⊤X~+ν2I\widetilde{X}^\top\widetilde{X}+\nu^2 I, but does not explicitly incorporate task-loss sensitivity. JARQ extends naturally to fixed positive-semidefinite quadratic saliency metrics that remain separable across output columns: the subproblems retain their least-squares structure, and the descent argument carries over under exact scale solves. For element-wise diagonal Fisher weights as in SqueezeLLM, the main practical challenge is that the group QR factors would become column-dependent and could no longer be shared across output columns. A full Fisher with cross-column interactions would additionally break the column-wise separability of our solver.

Q2. STE with Adam/Muon. We agree that STE-based code updates fit the spirit of JARQ, and their ability to update multiple groups jointly is appealing. Gradient-based rounding methods such as AdaRound and SignRound could also serve as hosts. We chose Babai because the fixed-scale subproblem has known quadratic structure: one back-substitution proposes coordinated changes to many codes within a group, requires no learning-rate or optimizer tuning, and an acceptance test ensures monotone descent. We have not evaluated STE-based updates inside JARQ, so we cannot rule out better empirical results. STE-generated proposals, including simultaneous updates across groups, could be incorporated by accepting them only when they reduce the full-column objective. This would preserve the monotone-descent guarantee under exact scale solves, though the fixed-point characterization would no longer be Babai-specific. We see this as a promising direction for future work.

Thank you again for the thoughtful questions and support.

Reviewer_f4nCOpenReview ↗

Thank you.

I appreciate the authors' response. My concerns have been addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.