ReCaC: Residual-Guided Calibration via Activation Coverage for LLM Quantization

As a crucial technique for efficient deployment of large language models (LLMs), post-training quantization (PTQ) performs low-bit quantization for LLM relying on a pre-defined calibration dataset. Logically, the calibration dataset determines quantization parameters and so clearly influences PTQ performance. However, effective calibration construction is still underexplored. Particularly, existing methods typically focus on selecting representative and task-related samples while overlooking the complementarity among selected samples, which hardly guarantees coverage of the calibration dataset and limits the generalization ability of quantized models. Therefore, this paper proposes a Residual-guided Calibration via activation Coverage (ReCaC) method, aiming to construct an effective calibration dataset by maximizing the sample coverage in the activation space under a small fixed budget. Specifically, given a pre-defined benchmark for constructing the calibration dataset, our ReCaC uses residual-guided calibration construction (RGC) to progressively seek and pick the samples with the smallest residual score after being added to the currently selected calibration set, which effectively prioritizes complementary activation information and maximizes calibration coverage in activation space. Furthermore, ReCaC can be flexibly extended to multi-domain scenarios by joint global and domain coverage (JGDC), which selects the samples that provide complementary contributions to the overall activation space while maintaining sufficient coverage for each domain. As such, ReCaC can capture complementary activation patterns among heterogeneous domains and so enlarge the coverage of calibration dataset. Experiments are conducted by using different language models across various post-training quantization methods and settings, and results on a dozen downstream tasks demonstrate ReCaC achieves highly competitive or superior performance compared with existing state-of-the-art calibration methods in both iintra-domain, inter-domain and out-of-domain downstream tasks, particularly under the setting of lower-bit quantization.