acceptodds
Under review as a conference paper at ICLR 2027

CAJQUANT: LARGE LANGUAGE MODEL QUANTIZATION VIA COMPENSATION-AWARE JOINT GRID SELECTION

Abstract

Large language models (LLMs) impose substantial memory and inference costs, motivating weight-only post-training quantization (PTQ) for efficient deployment. However, preserving low-bit accuracy while keeping quantization overhead low remains challenging. In column-wise quantization with error compensation, the candidate grid determines the quantization errors in the current column, which drive compensation updates to weights in later columns. Static group-wise grid evaluation overlooks these candidate-dependent updates by scoring grids on the weights before intra-group compensation. Moreover, fixing the zero point during clipping search can overlook grids with smaller quantization errors for weights outside the clipping interval. To address these issues, we propose Compensation-Aware Joint Grid Selection for LLM Quantization (CAJQuant), which jointly selects the clipping ratio and zero point using compensation-aware grid evaluation. This evaluation measures reconstruction loss under optimal compensation while accounting for the weight updates induced by each candidate grid. Experiments on Llama models from 7B to 70B parameters demonstrate strong performance at 2, 3, and 4 bits, with substantial improvements in model quality at 2 bits. In this setting, CAJQuant outperforms all evaluated baselines in WikiText-2 perplexity and average downstream accuracy across all four models. On a single NVIDIA A100-80GB GPU, CAJQuant completes 2-bit quantization of Llama-2-70B with a group size of 128 in approximately 5 hours.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
large language model, quantization

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_VdiUOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

A - Paper Summary

The paper identifies that in column-wise compensation methods (GPTQ-style), the choice of quantization grid for a group determines the quantization errors in the current column, which drive the compensation updates applied to later columns. Yet existing grid selectors score candidates statically, on weights before intra-group compensation. The authors propose CAJQuant, (1) a compensation-aware grid evaluation that simulates column-wise quantization and optimal compensation within the group and sums per-step reconstruction losses; (2) performs joint search over clipping ratio and zero point, since at fixed scale the two zero-point choices differ only in tail-weight errors, which static evaluation cannot distinguish.

B - Paper Contribution

(1) A compensation-aware reconstruction-loss criterion (Eq. 8) that accounts for candidate-dependent weight updates;

(2) joint clipping-ratio/zero-point selection under that criterion;

(3) preservation of the standard group-wise affine representation (drop-in, no kernel changes); and

(4) efficiency gains over search-based baselines.

C - Paper Validation

Models: Llama-1/2 models from 7B to 70B at 2/3/4 bits.

Metrics: WikiText-2 and C4 perplexity, zero-shot downstream accuracy on PIQA/ARC-e/ARC-c/BoolQ/HellaSwag/WinoGrande/MMLU, quantization time and peak memory.

Baselines: GPTQ, GPTAQ, OmniQuant, LeanQuant.

Ablation separates compensation-aware clipping from joint zero-point search.

Critical strengths
  1. The observation is correct, simple, and underexploited, and the fix is cheap. The coupling between grid choice and compensation updates is overlooked by static grid evaluators, and the compensation-aware criterion is a small change that preserves the standard affine format. This is consequential because it makes CAJQuant a drop-in that requires no kernel changes, which is a practical benefit for a PTQ method.

  2. Strong, consistent 2-bit results with better efficiency than search-based baselines. Llama-2-70B W2 WikiText-2 5.15 vs. 5.82 (LeanQuant) and MMLU 55.68% vs. 25.87% (GPTAQ) / 44.35% (OmniQuant) / 38.01% (LeanQuant). CAJQuant is also faster than both LeanQuant (2.3×) and OmniQuant (4.1×) on 70B. This is consequential because the 2-bit regime is where grid selection matters most, and being faster while searching is a huge advantage.

Critical weaknesses
  1. Marginal 3/4-bit gains with no variance estimates. At W4, CAJQuant vs. LeanQuant on Llama-2-7B is 5.575 vs. 5.574 — a tie or slight loss. Several downstream cells differ by <0.5 points without confidence intervals. This matters because the paper claims the method is competitive across bit widths, but the evidence supports that claim only at 2 bits, and the reader cannot distinguish signal from noise in the 3/4-bit cells.

  2. The contribution over LeanQuant is modest and the zero-point search is only two candidates. LeanQuant already makes the grid objective loss-aware via inverse-Hessian weighting; CAJQuant's delta is (a) simulating compensation updates and (b) searching two zero points. The ablation isolates these against GPTQ's ℓ2\ell_2 clipping, not against LeanQuant's objective, so the source of the gain over the closest baseline is not isolated. This matters because the paper's positioning as a significant advance over LeanQuant is not fully supported by the evidence as structured.

Questions for authors
  1. Can you report seed variance (at least 3 seeds) for the 3- and 4-bit downstream cells, so the "competitive at all bit widths" claim can be assessed?

  2. Can you ablate compensation-awareness on top of LeanQuant's inverse-Hessian-weighted objective (rather than GPTQ's ℓ₂ clipping), to isolate how much of the gain over LeanQuant comes from compensation-awareness vs. zero-point search?

  3. Why only two zero-point candidates for even b? A sensitivity study over a wider zero-point range would clarify whether {2b−1,2b−1−12^{b−1}, 2^{b−1}−1} is near-optimal or just convenient.

  4. What are the calibration set size and sequence length? These are referenced to Section 4 but not visible in the main text, and they matter for interpreting the 2-bit results.

2 comments
AuthorsOpenReview ↗

Multi-Seed Evaluation, Scoring Ablation, and Quantization Settings

We thank the reviewer for the insightful comments and suggestions.

Variability Across Calibration Seeds at 3 and 4 Bits

We have added experiments at 3 and 4 bits on Llama-2-7B using calibration seeds 0, 1, and 2 to evaluate GPTQ, OmniQuant, LeanQuant, and CAJQuant. The table below reports the mean ±\pm standard deviation across the three seeds of the average accuracy over seven tasks. For 3-bit quantization, the average accuracy of CAJQuant is 1.07 and 0.37 percentage points higher than that of LeanQuant and OmniQuant, respectively. At 4 bits, CAJQuant’s mean accuracy is 0.05 and 0.25 percentage points higher than that of LeanQuant and OmniQuant, respectively.

Bit widthGPTQOmniQuantLeanQuantCAJQuant
3-bit59.40 ±\pm 1.3363.49 ±\pm 0.2462.79 ±\pm 0.2863.86 ±\pm 0.50
4-bit65.77 ±\pm 0.2866.15 ±\pm 0.0366.35 ±\pm 0.1166.40 ±\pm 0.14

Ablation Study of Compensation-Aware Scoring

We thank the reviewer for suggesting additional ablation experiments. To further investigate the contribution of compensation-aware scoring, we constructed a controlled static-scoring variant: given the same candidate clipping configurations and a fixed zero point, it uses LeanQuant's inverse-Hessian-weighted objective to select the grid before column-wise quantization and compensation within the group. The compensation-aware variant uses the same candidate configurations and fixed zero point but simulates within-group quantization and compensation during candidate evaluation.

The table below reports WikiText-2 PPL on Llama-2-7B.

Bit widthStatic scoring (LeanQuant objective)Compensation-aware scoring (fixed zero point)
2-bit58.99028.854
3-bit6.0526.014
4-bit5.5955.597

Under this controlled setting, compensation-aware scoring substantially reduces PPL at 2 bits, from 58.990 to 28.854, and slightly reduces it at 3 bits, from 6.052 to 6.014. At 4 bits, PPL remains nearly unchanged, increasing by 0.002.

By using LeanQuant’s inverse-Hessian-weighted objective as the static baseline while holding the candidate configurations and zero point fixed, this ablation isolates the contribution of compensation-aware scoring within our framework, independently of zero-point search. However, it does not fully decompose the overall performance gap between CAJQuant and the original LeanQuant, which also differs in its candidate space and search over asymmetric clipping bounds.

Investigation of a Larger Zero-Point Search Space

Thank you for suggesting a broader zero-point search. Reviewer ZCpU raised the same question. Please refer to our response to Reviewer ZCpU for the corresponding experiments and discussion.

Calibration Set Size and Sequence Length

Thank you for the comment. The default calibration set consists of 128 segments randomly sampled from WikiText-2, each containing 2,048 tokens, as described in Section 4 of the original paper.

I thank the authors for addressing my questions and concerns. I will maintain my positive score.

Reviewer_Avt8OpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper observes that column-wise error compensation in GPTQ-like methods changes subsequent weights, so static grid selection before compensation may rank candidate grids incorrectly. Jointly choosing the clipping ratio and zero point also affects tail-weight error. CAJQuant evaluates candidate clipping/zero-point configurations at group boundaries while accounting for subsequent compensation, then selects the grid with the lowest compensation-aware reconstruction loss. The authors compare against GPTQ, GPTAQ, OmniQuant, and LeanQuant on LLaMA-2 7B/13B/70B and LLaMA-1 30B at 2–4 bits and report gains, especially at low bit widths, along with runtime and memory analyses.

Critical strengths
  1. The paper identifies the coupling between static grid search and sequential compensation. Its candidate score directly accounts for reconstruction loss after compensation, giving the method a clear motivation.
  2. Joint search over clipping ratios and two zero-point choices is a local modification compatible with the GPTQ pipeline; substantial gains are reported at 2 bits, particularly for LLaMA-2-70B.
  3. Evaluation spans multiple model sizes, bit widths, and downstream tasks, with runtime, memory, and reproduction details included.
Critical weaknesses
  1. Improvements are not uniform across tasks and settings, and gains over strong baselines at 4 bits are limited. Variance/significance across calibration samples should be reported, along with when gains are large enough to justify additional search.
  2. Table 4 reports 2-bit quantization time and peak memory: for LLaMA-2-70B, CAJQuant takes 5 h 14 min versus 1 h 37 min for GPTQ and 11 h 58 min for LeanQuant. CAJQuant is about 3.2x slower than GPTQ, although faster than the other grid-search methods. The paper should discuss whether this absolute cost is acceptable in practical update/deployment workflows and whether candidate pruning can reduce it.
Questions for authors
  1. How much additional quantization wall-clock time and peak memory does CAJQuant require relative to GPTQ? Can candidate pruning reduce the cost without losing quality?
  2. What is the variance of the main results across calibration seeds, datasets, and group sizes? Can the gain from joint zero-point search be separated from the gain due to compensation-aware scoring?
4 comments
AuthorsOpenReview ↗

Quantization Overhead, Candidate Pruning, and Robustness Across Settings

We thank the reviewer for the positive comments on our motivation and methodology. Below we address quantization costs, candidate pruning, robustness, and zero-point search.

Quantization Time and Memory Overhead

As shown in Table 4 of the original paper, CAJQuant takes 3.24×–3.49× GPTQ's wall-clock time across the four models on 2-bits, which is 2.24×–2.49× more time, and uses 0.64–2.27 GB more peak memory. However, this is just a one-time overhead which is normally acceptable for model deployment where quantization quality is much more important. On Llama-2-70B at 2 bits, WikiText-2 perplexity falls from 7.990 to 5.150.

Comparing CAJQuant with GPTQ is not the most appropriate choice: both share column-wise second-order compensation, but GPTQ keeps a group-wise min-max grid and a fixed symmetric zero point, while CAJQuant replays each clipping ratio and boundary zero point through that compensation and selects the lowest-loss pair. A more appropriate comparison is with more recent methods such as LeanQuant and OmniQuant. Table 4 shows that CAJQuant reduces quantization wall-clock time by 56%–59% and 76%–79% compared with LeanQuant and OmniQuant, respectively, across the four models.

On Llama-2-7B, narrowing the clipping-ratio search range from 0.2–1 to 0.5–1 decreases quantization time by ~25%, with nearly unchanged PPL for Llama-2-7B at 3/4 bits. However, the PPL rises from 9.645 to 16.987 at 2 bits for Llama-2-7B, showing that quality preservation depends on bit width.

The current results support narrowing the search range in some higher-bit settings.

ModelBit widthPPL (α\alpha: 0.2–1)PPL (α\alpha: 0.5–1)Time costTime cost (α\alpha: 0.5–1)
Llama-2-7B2-bit9.64516.98731 min23 min
Llama-2-7B3-bit5.9295.92730 min23 min
Llama-2-7B4-bit5.5755.57930 min23 min

Robustness Across Experimental Settings

We evaluated robustness across calibration seeds, group sizes, and datasets on Llama-2-7B and will include these results in the appendix of the final version of the paper.

Random Seed

WikiText-2 PPL over seeds 0, 1, and 2 (mean ±\pm standard deviation):

Method2-bit3-bit4-bit
GPTQ62.982±9.00162.982 \pm 9.0016.384±0.0256.384 \pm 0.0255.631±0.0125.631 \pm 0.012
OmniQuant11.190±0.10711.190 \pm 0.1076.097±0.0036.097 \pm 0.0035.593±0.0035.593 \pm 0.003
LeanQuant12.919±0.42212.919 \pm 0.4225.942±0.0185.942 \pm 0.0185.574±0.004\mathbf{5.574 \pm 0.004}
CAJQuant9.936±0.346\mathbf{9.936 \pm 0.346}5.925±0.012\mathbf{5.925 \pm 0.012}5.577±0.0065.577 \pm 0.006

CAJQuant has the lowest mean PPL at 2/3 bits. At 4 bits, its 0.003 gap to LeanQuant is smaller than either method's standard deviation across seeds.

Group Sizes

With seed 1 and group sizes 64, 128, and 256, CAJQuant achieves the lowest 2/3-bit PPL. At 4 bits, neither CAJQuant nor LeanQuant consistently wins; their absolute gap is at most 0.023.

MethodBit widthGroup size 64Group size 128Group size 256
GPTQ2-bit29.00257.886118.630
OmniQuant2-bit9.60411.23712.252
LeanQuant2-bit12.35113.31113.949
CAJQuant2-bit9.0099.84410.284
GPTQ3-bit6.1926.3676.498
LeanQuant3-bit5.8665.9276.014
OmniQuant3-bit5.9696.1016.212
CAJQuant3-bit5.8425.9115.962
GPTQ4-bit5.6005.6425.643
OmniQuant4-bit5.5725.5965.614
LeanQuant4-bit5.5465.5775.574
CAJQuant4-bit5.5535.5725.597

Datasets

With seed 0, we calibrated and evaluated separately on WikiText-2 and on C4. CAJQuant achieves the lowest PPL at 2/3 bits. At 4 bits, the gap to LeanQuant is negligible, at 0.001 on WikiText-2 and 0.002 on C4.

MethodBit widthWikiText-2 calibration & evaluationC4 calibration & evaluation
GPTQ2-bit73.37488.697
OmniQuant2-bit11.06814.546
LeanQuant2-bit12.97623.627
CAJQuant2-bit9.64514.471
GPTQ3-bit6.4128.258
OmniQuant3-bit6.0957.807
LeanQuant3-bit5.9377.689
CAJQuant3-bit5.9297.685
GPTQ4-bit5.6317.165
OmniQuant4-bit5.5937.121
LeanQuant4-bit5.5747.118
CAJQuant4-bit5.5757.120

Zero-Point Search

Table 3 of the original paper isolates the contribution of zero-point search by keeping compensation-aware scoring fixed and comparing a fixed symmetric zero point with joint search. On 2-bit Llama-2-70B, this zero-point step alone reduces WikiText-2 perplexity from 8.300 to 5.150.

Thank you for your response. By the way, I am curious about the results of CAJQuant on MoE-architecture LLMs. Any existing results? Fine if not.

Thank you for your response and for raising the score. Regarding your question on MoE-based models, we do not currently have results, but we will do our best to add some evaluations.

We additionally evaluated GPTQ, OmniQuant, LeanQuant, and CAJQuant on DeepSeek-MoE-16B-base under the same WikiText-2 setting. Perplexity is reported below.CAJQuant achieves the lowest perplexity at 2 and 3 bits, while CAJQuant and LeanQuant achieve similar perplexity at 4 bits.

Bit widthMethodWikiText-2
BF16—11.387
2-bitGPTQ83.901
2-bitOmniQuant24.406
2-bitLeanQuant22.022
2-bitCAJQuant20.292
3-bitGPTQ13.235
3-bitOmniQuant12.963
3-bitLeanQuant12.470
3-bitCAJQuant12.443
4-bitGPTQ11.709
4-bitOmniQuant11.729
4-bitLeanQuant11.608
4-bitCAJQuant11.613
Reviewer_ZCpUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the degradation in weight-only post-training quantization (PTQ) of LLMs at low bit-widths (especially 2-bit). The authors identify two primary limitations in standard column-wise error-compensated quantization (e.g., GPTQ): (1) static group-wise grid search evaluates candidates on original weights before intra-group compensation, ignoring candidate-dependent updates; and (2) fixed zero points restrict tail weights outside clipping bounds. To resolve this, the paper proposes CAJQuant, which simulates column-wise quantization and error compensation within each group to calculate an accumulated reconstruction loss under optimal compensation, and jointly optimizes the clipping ratio α\alpha and zero point zz. Evaluated on LLaMA-1/2 models across 2, 3, and 4 bits, CAJQuant demonstrates strong perplexity reductions and downstream accuracy improvements over baselines like GPTQ, OmniQuant, and LeanQuant, while maintaining fast runtime.

Critical strengths
  1. Elegant Coupling of Simulation and Error Compensation: Simulating sequential compensation within candidate evaluation directly fixes the mismatch between the static evaluation metric and dynamic column-wise updates without adding complex iterative learning.
  2. Empirical Gains and Efficiency at Extreme Low Bits: The method delivers substantial gains at 2-bit quantization across multiple LLaMA scales (e.g., LLaMA-2-70B achieving 5.15 perplexity and completing in ~5 hours on a single A100 GPU), providing an effective balance between reconstruction quality and quantization overhead.
Critical weaknesses
  1. Narrow Scope of Model Architectures and Formats: The empirical validation is restricted entirely to LLaMA-1 and LLaMA-2 models. Testing on modern open architectures (e.g., LLaMA-3, Qwen, or Mistral) with grouped-query attention or SwiGLU variations is absent, leaving generalizability across newer architectures unverified.
  2. Restricted Zero-Point Candidate Set: The joint search limits zero-point candidates to only two symmetric-boundary options Z={2b−1−1,2b−1}\mathcal{Z} = \{2^{b-1}-1, 2^{b-1}\}. While motivated by symmetric clipping and tail coverage, the paper provides limited theoretical or empirical justification as to whether a wider zero-point search would yield further gains or destabilize the simulation.
Questions for authors
  1. How does CAJQuant perform on more recent model families such as LLaMA-3 or Qwen2.5/Qwen3, especially regarding activation outlier behavior and tail weight distributions?

  2. Did the authors investigate extending the zero-point search space beyond the two candidates in Equation (9), and what is the trade-off in quantization runtime versus perplexity?

1 comment
AuthorsOpenReview ↗

Evaluation on Recent Models and Zero-Point Search Trade-offs

We thank the reviewer for the positive assessment of our methodology. We also appreciate the questions on the effectiveness of CAJQuant on more recent models and the investigation of more possible zero-point candidates. We address these questions below.

Effectiveness of CAJQuant on Recent Models

We evaluated CAJQuant, GPTQ, OmniQuant, and LeanQuant on downstream tasks using Llama-3-8B and Qwen3-8B at 2–4 bits. CAJQuant achieves the highest average accuracy in all evaluated settings except 4-bit Llama-3-8B, where it scores slightly below LeanQuant (72.78 vs. 72.88). Specifically, at 2 bits, CAJQuant outperforms the strongest baseline by 2.46 and 2.31 percentage points on Llama-3-8B and Qwen3-8B, respectively.

Llama-3-8B

MethodPIQAARC-EARC-CHSBoolQWinoMMLUAvg.
GPTQ-4bit79.1174.4151.8877.9781.2272.6962.7871.44
OmniQuant-4bit79.6076.5250.6078.5880.7671.7463.6371.63
LeanQuant-4bit79.9878.7054.6178.4281.3873.5663.5172.88
CAJQuant-4bit79.3879.1253.7578.3681.0774.1163.7072.78
GPTQ-3bit73.4564.7741.0472.5569.9768.1938.7061.24
OmniQuant-3bit76.3963.3840.5373.4774.2868.1953.6664.27
LeanQuant-3bit77.3771.9748.0474.7980.4673.8056.6469.01
CAJQuant-3bit76.5074.8747.7875.1381.0172.5357.7169.36
GPTQ-2bit49.7325.5523.9825.4740.3749.8025.2234.30
OmniQuant-2bit53.9728.3722.7028.1937.8351.9326.0635.58
LeanQuant-2bit61.8649.3727.4748.4967.3460.1426.7748.78
CAJQuant-2bit65.2952.6129.9552.7967.2263.3027.5051.24

Qwen3-8B

MethodPIQAARC-EARC-CHSBoolQWinoMMLUAvg.
GPTQ-4bit79.8781.7357.4277.6284.5671.0375.2575.35
OmniQuant-4bit79.8279.8856.7476.5782.7572.4575.5474.82
LeanQuant-4bit80.0381.2357.3477.0882.2371.4375.2274.94
CAJQuant-4bit79.9281.5756.9177.4483.5272.7775.4775.37
GPTQ-3bit76.8272.3947.3573.7973.5267.0167.9568.40
OmniQuant-3bit78.4078.5851.1972.9880.8670.7270.3171.86
LeanQuant-3bit78.5676.3552.3073.6280.7669.5370.8771.71
CAJQuant-3bit77.9779.5954.8674.0282.1771.7471.1973.08
GPTQ-2bit52.0129.0023.6328.1243.1551.7824.5336.03
OmniQuant-2bit68.9360.1036.0950.6267.7158.0929.6353.02
LeanQuant-2bit63.7648.2728.6747.7462.6056.8337.5049.34
CAJQuant-2bit69.4857.1137.2054.0763.2762.0444.1555.33

Investigation of a Larger Zero-Point Search Space

We expanded the zero-point search space on Llama-2-7B at 2 and 4 bits. Considering both PPL and downstream task accuracy, searching two zero-point candidates provides the best overall trade-off among the evaluated settings. For quantization at 2 and 4 bits, expanding the zero-point candidate set does not improve average downstream accuracy but increases quantization time.

The two zero-point candidates in CAJQuant, z∈{2b−1−1,2b−1}z \in \lbrace 2^{b-1}-1,2^{b-1}\rbrace, yield identical dequantized values for weights inside the clipping interval but different values for tail weights, allowing the grid to extend toward either tail. Expanding the candidate set may clip the integer codes of weights inside the clipping interval to the valid range, altering their dequantized values and reconstruction errors.

ModelBit widthZero pointPPLPIQAARC-EARC-CHSBoolQWinoMMLUAvg.Time cost
Llama-2-7B4-bit{7,8}\{7,8\}5.57578.4073.3245.0577.3475.1769.3845.4866.3131 min
Llama-2-7B4-bit{6,7,8,9}\{6,7,8,9\}5.55678.6772.6944.1175.5176.8568.6745.0365.9350 min
Llama-2-7B4-bit{0,…,15}\{0,\ldots,15\}5.56277.5375.5143.0056.2778.2969.3846.1163.73175 min
Llama-2-7B2-bit{1,2}\{1,2\}9.64566.2749.2429.0149.5565.7259.0426.8149.3831 min
Llama-2-7B2-bit{0,1,2,3}\{0,1,2,3\}9.82064.9150.6325.5136.9863.2459.6727.7146.9551 min

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.