- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes ReCaC, a method for constructing a small calibration set (K=128) for post-training quantization of LLMs. Each candidate sample is represented by hidden states from a few layers, concatenated and randomly projected to 384 dimensions. Selection starts from the highest-local-density sample and then greedily adds the candidate that most reduces the residual energy of the whole candidate pool after projection onto the span of the selected set (a greedy column-subset / pivoted-QR style criterion). A multi-domain extension (JGDC) adds a lambda-weighted term that prioritizes the currently least-covered domain. The method is evaluated on Llama-3.1-8B-Instruct, Gemma-2-9B-it and Qwen3.5-9B with GPTQ and AWQ (W3/W4A16) and SpinQuant and FlatQuant (W3/W4A4), against COLA and ZipCal, on 11 intra-domain and 5 held-out tasks, plus OOD tasks (Alpaca, HumanEval, MBPP) and ablations on initialization and lambda. The main claim is that maximizing activation-space coverage gives competitive or better quantized models, especially at 3-bit.
Critical strengths
-
The selection criterion is simple, deterministic and model-agnostic, operates directly on the activations that PTQ actually calibrates on, and needs no prior about the target task distribution (unlike ACDM). This makes it easy to adopt in existing PTQ pipelines.
-
The evaluation matrix is broad: three models, four PTQ methods, four bit settings, 16 downstream tasks including 5 held-out tasks and OOD tasks, with full per-task tables in the appendix. Gains are small but consistent in the activation-quantized settings (SpinQuant/FlatQuant W4A4) and large at GPTQ W3A16.
Critical weaknesses
-
The headline gains rest on a single setting where a baseline collapses, and the rest of the evidence is within noise. All large numbers quoted in the text (17.75 / 20.45 points, 25.83-point drop) come from GPTQ W3A16 on Llama, where COLA reaches mean 36.23 (single-domain) / 30.53 (multi-domain), GSM8k 4.33 / 0.23 and HellaSwag 26.68, i.e. at or below chance. Everywhere else (W4A16, AWQ, all Gemma settings) differences are mostly 0.1 to 1 point from a single run with no variance, even though the random projection is itself a seeded component; the initialization ablation (Table 5) separates density-max from random by 0.19 points. There is no random-selection baseline and no standard 128-sample C4/WikiText default, which is what practitioners actually use, so it is not possible to tell whether any of the three compared methods beats the default, or whether COLA's collapse is a GPTQ configuration issue (group size, act-order and damping are not stated).
-
Evaluation anomalies undermine the reported numbers. Quantized models exceed the dense model on MMLU-K by large margins: Gemma ReCaC GPTQ-W4A16 46.84 vs dense 33.83 (the dense value is itself implausibly low for Gemma-2-9B-it, and Gemma MMLU-K ranges from 24.87 to 46.84 across settings); Qwen3.5 GPTQ-W4A16 73.71 to 75.12 vs dense 66.36. On Llama, MMLU-M at lambda=1.5 is 32.59 while every other lambda gives 21 to 22 and dense is 24.44. Dense Gemma MMLU-K differs between Table 1 (33.83) and Table 2 (33.93). Since MMLU-K is what separates the methods on Gemma, the Gemma conclusions are not trustworthy as reported. In addition, lambda is selected on the same evaluation tasks that are reported (Table 9 covers the tasks of Table 2), and the lambda=1.5 MMLU-M outlier feeds the reported means.
Secondary: the selection rule is greedy residual-energy subset selection (pivoted QR / column subset selection / OMP-style landmark selection) on projected activations, and none of that literature (CSSP, Nystrom landmarks, kernel herding, k-center greedy coresets) is cited or compared. Fig. 1b measures coverage in exactly the space ReCaC optimizes, so ReCaC leading on coverage holds by construction, and three points do not establish that coverage causes performance. Several details needed for reproduction are missing (see questions).
Questions for authors
- Which splits (train/test) and how many candidates per domain form the calibration pool? Is there any overlap with the evaluation data for the intra-domain tasks?
- How is a 2048-token sequence reduced to a single activation vector (mean pooling, last token?), and from which hidden state?
- What are the GPTQ settings (group size, act-order, damping)? Does COLA's W3A16 collapse on Llama persist under a different damping value or seed, and why does it happen?
- Why do quantized models exceed the dense model by 7 to 13 points on MMLU-K for Gemma and Qwen? What is the dense Gemma MMLU-K evaluation setup, and why do Tables 1 and 2 report different dense values?
- Can you report random selection and the default C4/WikiText 128-sample set, over at least three seeds including the random projection seed?
- Was lambda tuned on held-out data? What are the means without the MMLU-M outlier at lambda=1.5?
- What is the selection cost (wall-clock and complexity in the pool size N) relative to the quantization time?
- Coverage is capped by r=384 with K=128. How do results change with r=128 or r=1024?