acceptodds
Withdrawn from ICLR 2027

When Does the Calibration Source Matter for Low-Bit GPTQ Quantization of LLMs? A Seed-Paired Study

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Earlier studies found that the calibration source has little effect on four-bit weight quantization of current language models, while a three-bit study of one reasoning model found GPTQ highly sensitive to the calibration domain. We measure both precisions on the same calibration draws. Qwen3-8B, Llama-3.1-8B and Mistral-7B-v0.3 are quantized with weight-only GPTQ at three and four bits using WikiText, random-token or prompted model-generated calibration text, which we call Diverse, with five calibration draws shared across sources and precisions and a budget of tokens, and are evaluated on WikiText-2 perplexity and all 14,042 MMLU questions. On the recorded MMLU readout, the Diverse-minus-WikiText contrast is , and points at four bits and , and points at three bits, every three-bit seed-paired 95% interval lying above zero, while WikiText has the lower perplexity in all 15 model–seed pairs; the source-by-precision interaction averages points, with a 95% interval of over seed blocks. Choosing the source by WikiText perplexity thus forgoes , and MMLU points at three bits relative to the best source in hindsight and under one point at four bits, and choosing it by accuracy on held-out questions lowers this three-bit regret for Qwen3-8B and Llama-3.1-8B. Three-bit calibration draws vary – times more than four-bit draws. Registered replacement runs that score every prompt untruncated reproduce the three-bit gain in every seed of all three models, and on 15 fresh Qwen3-8B draws in three registered batches the gain and the interaction average and points with intervals above zero, as they do on ARC-Challenge for the ten of these draws scored on it.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
post-training quantization, calibration data, activation smoothing, efficient language models, model compression

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_kVPTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors evaluate the effect of changing calibration dataset for the GPTQ quantizer.

Critical strengths

None

Critical weaknesses
  • Lack of Novelty

    • The key results are well known in the quantization community (e.g. "Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolbox" 2024 Liu et al.) They are also inuitively fairly obvious: When the quantization problem is ~harder mistakes are more costly.
    • The practical relevance of the observation that the calibration gap is larger at lower bits is limited or at least not well elaborated.
  • Study design is flawed

    • As the authors note "The studied regime. The budget of 32 × 128 tokens is 64 times smaller than GPTQ’s reference setup of 128 × 2048 tokens, and short sequences limit the context seen by each activation." -- This is too far outside the practically relevant regime, especially when the main contribution are empirical results.
    • "Perplexity selects the wrong source where the choice matters." is not well supported. The perplexity for selection should be evaluated OOD.
Questions for authors

None

Reviewer_KGz2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how much the choice of calibration source matters for downstream performance. The authors study three choices: WikiText, random tokens, or diverse text that the model generates itself from 32 domain prompts. The model is then quantized with GPTQ and evaluated on WikiText-2 perplexity and MMLU. The results show that the diverse text is best for MMLU at 3 bits and WikiText calibration is best for WikiText-2 perplexity.

Critical strengths
  1. The calibration source is an important choice that determines the quality of the final quantized model.

  2. The paper conducts significance tests and shows actual improvements in perplexity or MMLU.

Critical weaknesses
  1. The experiments are conducted in a very small setting (4096 tokens in total, 64 times fewer than GPTQ's reference setting), and no motivation or explanation is given for deviating from the reference setting. With only 4096 tokens, GPTQ is likely under-calibrated, and the results are shown only for an unusual, data-limited regime.

  2. The conclusion seems self-explanatory. It surprises no one that a model calibrated on WikiText-2 performs better on WikiText-2 evaluation than with other sources. There is limited novelty: Liu et al. (COLM 2025) already found that self-generated calibration text beats WikiText for 3-bit GPTQ.

Questions for authors

Please explain the non-standard GPTQ data budget, or at least run several experiments under the full setting.

Reviewer_bxjPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the impact of calibration data on 3-bit weight quantization. It quantizes Qwen3-8B, Llama-3.1-8B, and Mistral-7B-v0.3 at W3 and W4 using three calibration sources. The paper makes the interesting observation that perplexity selects the wrong calibration source at W3. It also finds that at W3, accuracy across calibration draws varies by up to 10x compared to W4.

Critical strengths
  • The paper does a thorough evaluation spanning multiple sources and precisions
  • The findings on W3 and the shortcomings of perplexity-based ordering are practically useful
Critical weaknesses

The main results use 32*128 tokens, where the authors note that the Hessian is rank-deficient. Is this sufficient to understand the real impact of W3 at practical budgets?

Questions for authors
  • What is the W3 gain at 128*2048 tokens for at least one model?
  • Do W3 WikiText models show a skewed distribution of predicted letters?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.