- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents a theoretically grounded approach for calibration data selection for post-training quantization. The central assumption of the work is that there is no need to know the exact performance of every single hyperparameter combination for quantization but to preserve the relative ranking (i.e., price gap in terms of KL divergence between the full and quantized models) between options. The paper provides a proved theoretical error bound that breaks total price-gap distortion into three components consisting of the distribution shift error (difference between calibration and deployment data), the sampling error from using a finite pool of data, and the calibration data compression error (calibration data representativeness when subsampling). The later is the only term that can directly be controlled by ensuring that the average activations of the selected calibration set match those of the full data (e.g., the whole of C4).
The proposed PPCC algorithm selects an optimal calibration set by greedily sampling one data point at a time by adapting the Herding algorithm. Empirical results show that it offers better downstream performance for Llama 3.1 8B while results on the Qwen3 8B family are somewhat mixed.
Critical strengths
-
Theoretically grounded and principled method for choosing calibration data for post-training quantization.
-
The proof of the theoretical error bound that breaks total price-gap distortion is the strongest contribution.
Critical weaknesses
-
The empirical evidence is somewhat limited (and rather short in the paper). The method is only applied on smallish models (1-8B). Results on Qwen are not as strong as for the Llama model. It would make the work much more complete if the evaluation include larger models (~30B or higher).
-
In general, the PPCC runtime is higher and more compute demanding compared to models that do not require the model for data selection.
Questions for authors
- How applicable is the proposed approach to architectures that are not dense, e.g. MoE?
3 comments
Thank you for your clarifications, I'm keeping my score. Best of luck with your submission.
Thank you for your reply and your kind wishes. We hope our responses have fully addressed your concerns. Please feel free to let us know if you have any further questions.
We thank Reviewer cFX9 for the positive assessment of the error decomposition and for the suggestions on model scale, runtime, and MoE.
On Qwen3-8B, PPCC still achieves the highest average score in all four settings for the Base model, including a gain of 1.98 points and the lowest PPL in bitwidth selection, and in all three settings for the Instruct model in Appendix D. We believe that the margins in the other three Base settings are smaller than on Llama because Qwen3-8B is less sensitive to quantization in our setting. For example, in bitwidth selection at 3.25 bits, the WikiText-2 PPL of Qwen3-8B Base stays at or below 9.83 (7.00 in BF16) under every selector, while that of Llama-3.1-8B Base reaches 32.26 (6.24 in BF16), so on Qwen the calibration data has little room to change the result. This agrees with prior findings that Llama-3 models degrade notably under low-bit quantization [1] and that the effect of calibration data differs across model families [2].
To evaluate a larger model, we ran bitwidth selection on Qwen3-32B at 2.75 and 3.00 bits, following the Qwen3-8B Instruct setting in Appendix D (non-thinking mode, AWQ INT2/3/4 with group size 128, N=64). Below we report the scores on GSM8K, AIME24, AIME25, and GPQA and their average.
At 2.75 and 3.00 bits, where quantization costs more, PPCC achieves the best score on all four tasks and the highest average, 1.75 and 1.84 points above the strongest baseline, respectively.
We will include these results and discussion in the revision.
We agree that PPCC takes longer than ZipCal, because ZipCal selects sequences by vocabulary coverage without any forward pass. However, we believe this cost is acceptable, since PPCC takes 12–14 minutes on one H100 GPU for an 8B model, comparable to the other model-aware selector CoverCal, and this cost is paid only once before deployment.
In return, PPCC leads to better configurations in Sec. 5.2. Averaged over all selection settings, the average downstream score of PPCC exceeds that of the strongest baseline by 1.11, 1.53, and 1.37 points on Llama-3.1-8B Base, Qwen3-8B Base, and Qwen3-8B Instruct, where the strongest baseline is ZipCal, Random, and CoverCal, respectively. PPCC is also more reliable, with its WikiText-2 PPL on Llama-3.1-8B staying between 8.47 and 8.71 across the four settings of Table 1, while each baseline exceeds 13 in at least one setting. Since the selected configuration determines the quality of the quantized model throughout its deployment, we consider a one-time cost of about 13 minutes a worthwhile trade-off.
Yes, PPCC can be extended to MoE models. The price is defined for each linear layer and depends on the data only through the input moments of that layer. In an MoE model, the linear layers of each expert can be treated as separate layers, whose input moments are computed over the tokens routed to that expert, and the price-weighted moment matching and Alg. 1 can then be applied to these layers. Matching these moments also takes into account how often each expert is used in the pool, which is relevant to MoE calibration, where some experts receive few calibration tokens.
However, a complete extension is more involved, because our price and the analysis in Secs. 2 and 3 assume that each token passes through the same layers before and after quantization, whereas in an MoE model, quantization can change which experts a token is routed to. Extending the price and the analysis to account for these routing changes requires substantial further work, which goes beyond the scope of this paper.
Thank you for this suggestion. We leave the extension of PPCC to MoE models as future work and will add this discussion to the revision.
We hope the new results on Qwen3-32B and the cost analysis resolve your concerns, and we are happy to provide any additional details if needed.