- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
A - Paper Summary
The paper identifies that in column-wise compensation methods (GPTQ-style), the choice of quantization grid for a group determines the quantization errors in the current column, which drive the compensation updates applied to later columns. Yet existing grid selectors score candidates statically, on weights before intra-group compensation. The authors propose CAJQuant, (1) a compensation-aware grid evaluation that simulates column-wise quantization and optimal compensation within the group and sums per-step reconstruction losses; (2) performs joint search over clipping ratio and zero point, since at fixed scale the two zero-point choices differ only in tail-weight errors, which static evaluation cannot distinguish.
B - Paper Contribution
(1) A compensation-aware reconstruction-loss criterion (Eq. 8) that accounts for candidate-dependent weight updates;
(2) joint clipping-ratio/zero-point selection under that criterion;
(3) preservation of the standard group-wise affine representation (drop-in, no kernel changes); and
(4) efficiency gains over search-based baselines.
C - Paper Validation
Models: Llama-1/2 models from 7B to 70B at 2/3/4 bits.
Metrics: WikiText-2 and C4 perplexity, zero-shot downstream accuracy on PIQA/ARC-e/ARC-c/BoolQ/HellaSwag/WinoGrande/MMLU, quantization time and peak memory.
Baselines: GPTQ, GPTAQ, OmniQuant, LeanQuant.
Ablation separates compensation-aware clipping from joint zero-point search.
Critical strengths
-
The observation is correct, simple, and underexploited, and the fix is cheap. The coupling between grid choice and compensation updates is overlooked by static grid evaluators, and the compensation-aware criterion is a small change that preserves the standard affine format. This is consequential because it makes CAJQuant a drop-in that requires no kernel changes, which is a practical benefit for a PTQ method.
-
Strong, consistent 2-bit results with better efficiency than search-based baselines. Llama-2-70B W2 WikiText-2 5.15 vs. 5.82 (LeanQuant) and MMLU 55.68% vs. 25.87% (GPTAQ) / 44.35% (OmniQuant) / 38.01% (LeanQuant). CAJQuant is also faster than both LeanQuant (2.3×) and OmniQuant (4.1×) on 70B. This is consequential because the 2-bit regime is where grid selection matters most, and being faster while searching is a huge advantage.
Critical weaknesses
-
Marginal 3/4-bit gains with no variance estimates. At W4, CAJQuant vs. LeanQuant on Llama-2-7B is 5.575 vs. 5.574 — a tie or slight loss. Several downstream cells differ by <0.5 points without confidence intervals. This matters because the paper claims the method is competitive across bit widths, but the evidence supports that claim only at 2 bits, and the reader cannot distinguish signal from noise in the 3/4-bit cells.
-
The contribution over LeanQuant is modest and the zero-point search is only two candidates. LeanQuant already makes the grid objective loss-aware via inverse-Hessian weighting; CAJQuant's delta is (a) simulating compensation updates and (b) searching two zero points. The ablation isolates these against GPTQ's clipping, not against LeanQuant's objective, so the source of the gain over the closest baseline is not isolated. This matters because the paper's positioning as a significant advance over LeanQuant is not fully supported by the evidence as structured.
Questions for authors
-
Can you report seed variance (at least 3 seeds) for the 3- and 4-bit downstream cells, so the "competitive at all bit widths" claim can be assessed?
-
Can you ablate compensation-awareness on top of LeanQuant's inverse-Hessian-weighted objective (rather than GPTQ's ℓ₂ clipping), to isolate how much of the gain over LeanQuant comes from compensation-awareness vs. zero-point search?
-
Why only two zero-point candidates for even b? A sensitivity study over a wider zero-point range would clarify whether {} is near-optimal or just convenient.
-
What are the calibration set size and sequence length? These are referenced to Section 4 but not visible in the main text, and they matter for interpreting the 2-bit results.
2 comments
I thank the authors for addressing my questions and concerns. I will maintain my positive score.
Multi-Seed Evaluation, Scoring Ablation, and Quantization Settings
We thank the reviewer for the insightful comments and suggestions.
Variability Across Calibration Seeds at 3 and 4 Bits
We have added experiments at 3 and 4 bits on Llama-2-7B using calibration seeds 0, 1, and 2 to evaluate GPTQ, OmniQuant, LeanQuant, and CAJQuant. The table below reports the mean ± standard deviation across the three seeds of the average accuracy over seven tasks. For 3-bit quantization, the average accuracy of CAJQuant is 1.07 and 0.37 percentage points higher than that of LeanQuant and OmniQuant, respectively. At 4 bits, CAJQuant’s mean accuracy is 0.05 and 0.25 percentage points higher than that of LeanQuant and OmniQuant, respectively.
Ablation Study of Compensation-Aware Scoring
We thank the reviewer for suggesting additional ablation experiments. To further investigate the contribution of compensation-aware scoring, we constructed a controlled static-scoring variant: given the same candidate clipping configurations and a fixed zero point, it uses LeanQuant's inverse-Hessian-weighted objective to select the grid before column-wise quantization and compensation within the group. The compensation-aware variant uses the same candidate configurations and fixed zero point but simulates within-group quantization and compensation during candidate evaluation.
The table below reports WikiText-2 PPL on Llama-2-7B.
Under this controlled setting, compensation-aware scoring substantially reduces PPL at 2 bits, from 58.990 to 28.854, and slightly reduces it at 3 bits, from 6.052 to 6.014. At 4 bits, PPL remains nearly unchanged, increasing by 0.002.
By using LeanQuant’s inverse-Hessian-weighted objective as the static baseline while holding the candidate configurations and zero point fixed, this ablation isolates the contribution of compensation-aware scoring within our framework, independently of zero-point search. However, it does not fully decompose the overall performance gap between CAJQuant and the original LeanQuant, which also differs in its candidate space and search over asymmetric clipping bounds.
Investigation of a Larger Zero-Point Search Space
Thank you for suggesting a broader zero-point search. Reviewer ZCpU raised the same question. Please refer to our response to Reviewer ZCpU for the corresponding experiments and discussion.
Calibration Set Size and Sequence Length
Thank you for the comment. The default calibration set consists of 128 segments randomly sampled from WikiText-2, each containing 2,048 tokens, as described in Section 4 of the original paper.