acceptodds
Under review as a conference paper at ICLR 2027

Not All Calibration Data Are Equal: Optimizing Calibration Data for Post-Training Quantization

Abstract

An often overlooked source of performance variation in post-training quantization (PTQ) is the calibration data used to estimate quantization-related statistics. Even under the same model and quantization configuration, different calibration data can lead to substantially different performance. We propose CalibSelect, a calibration data assessment and selection framework based on model responses. CalibSelect introduces two metrics: the calibration activation range score (CARS), which measures the spread of activation responses while mitigating the influence of extreme samples, and the layer sensitivity discrimination score (LSDS), which measures how clearly and stably calibration data reveal differences in quantization sensitivity across layers. Based on these metrics, CalibSelect adopts a two-stage selection strategy that first identifies several promising calibration data sets from the candidate data pool and then refines them through indicator-guided local optimization. CalibSelect is compatible with different PTQ methods and does not modify their underlying algorithms. Experiments on Qwen2.5-7B and LLaMA3.1-8B using AWQ and AutoRound show that CalibSelect consistently outperforms random sampling and existing selection methods. The largest reduction in quantized model perplexity reaches 8.23%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Post-Training Quantization, Calibration Data, Low-Bit Quantization, Model Compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DkNKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces the calibration activation range score (CARS) and the layer sensitivity discrimination score (LSDS)., CARS assesses activation response coverage by analyzing the mean absolute activation, root mean square, and high-percentile activation values of calibration data across Transformer layers. LSDS evaluates how clearly and stably calibration data capture layer-wise differences in quantization sensitivity. Also proposes a calibration data assessment and selection framework called CalibSelect. Which integrates CARS and LSDS to guide data selection for PTQ.

Critical strengths

This paper addresses a practical and important problem. and introduced a quantization aware data quality assessment framework. From experiments side, it compatible with existing PTQ methods.

Critical weaknesses

I see some results on 4-bit weight quantization but not other number of bits. have test on other number of bits? The novelty of the CARS metric is limited. It's a combination of existing methods like Mean absolute activation, RMS, percentile and quantile range. What's the motivation to combine these pieces together as a metric.

Questions for authors
  1. All the experimental results are tested on LLMs, doesn't mean this method is designed for LLM architecture.

  2. Can this model used on other VLMs, diffusion models or other types of model?

  3. How about the computational cost and efficiency compare with other methods?

  4. From table 7, on the right columns "Ours", the performance for LSDS seems increase when budget from 16-->128 while CARS decreases. I am confused if a higher score is needed or lower score is needed for the 2 metrics.

4 comments

Response to Critical Weaknesses: Thank you for your comment. We have addressed the two concerns as follows. (i) Quantization bit-width. We have added experiments with 2-bit weight quantization. Detailed results are reported in Appendix E. CalibSelect achieves an overall average PPL of 15.4509, a 2.79% reduction relative to the strongest baseline, ACDM (15.8940), and a 4.92% reduction relative to SelectQ (16.2501). These additional experiments evaluate the applicability of CalibSelect under a lower quantization bit-width. (ii) Motivation for CARS. We agree that RMS, mean absolute activation, and quantiles are established elementary statistics. Their combination in CARS, however, is motivated by the relationship between PTQ reconstruction error and second-order activation information. For a linear layer y=Wxy = Wx, the output reconstruction error induced by weight quantization can be written as $\mathcal{L}_{S}(\Delta W) =\underset{x\sim S}{\mathbb{E}} \left[\|\Delta W x\|_2^2\right] = \operatorname{tr}(\Delta WG_S \Delta W^\top)$, where GS=Ex∼S[xx⊤]G_S=\underset{x\sim S}{\mathbb{E}} \left[xx^\top\right] is the second-moment matrix of the calibration activations. This expression shows that second-order activation information directly affects the estimation of quantization reconstruction error. However, using high-dimensional second-moment matrices to compare activation responses across calibration samples incurs substantial computational and storage costs. We therefore employ several complementary low-cost statistics as approximate descriptors of activation characteristics. Squared RMS corresponds to the normalized trace of the second-moment matrix and reflects overall second-order activation energy. Mean absolute activation characterizes average magnitude, while Q95 and Q99 supplement information about higher-magnitude responses. Since no single statistic fully describes the activation-magnitude distribution, we combine these descriptors to assess activation states from the perspectives of overall energy, average magnitude, and upper-tail behavior.

Response to Questions (1) and (2): Thank you for your comments. To address whether CalibSelect is restricted to LLMs architectures and whether it can be extended to other model types, we have further examined its applicability and added a VLMs experiment. The design of CalibSelect is not tailored to a particular LLM architecture, so it may extend to other models. Following your suggestion, we evaluate CalibSelect on the Qwen2-VL-7B-Instruct using AWQ. We construct the candidate calibration pool by combining the GQA and TextVQA training sets and form a single evaluation set by combining their validation sets. Under calibration budgets of 16, 32, 64, and 128, we compare CalibSelect with Random, CoLA, ACDM, and SelectQ. As shown in Fig. 9 of Appendix I in the manuscript, CalibSelect achieves the lowest PPL at three of the four budgets (16, 64, and 128). Although it is not the best method at a budget of 32, the results indicate promising quantization performance on the VLMs. These findings provide additional empirical evidence that the approach can extend beyond text-only LLMs to other models. The corresponding results have been added to Appendix I.

Response to Question (3): Thank you for your comment. We have added a runtime comparison with other calibration data selection methods. Under the same GPU configuration, the average runtimes of CoLA, ACDM, and SelectQ are approximately 258s, 127s, and 99s, respectively, whereas the complete CalibSelect procedure takes approximately 1541s. Although our method incurs higher computational overhead, its primary advantage is improved quantization performance rather than faster calibration data selection. In the Qwen2.5-7B AWQ experiments, CalibSelect achieves an average PPL of 12.4765, outperforming CoLA (12.7154), ACDM (12.7615), and SelectQ (12.7057). Thus, CalibSelect trades additional offline calibration data selection cost for better quantization performance. Because calibration data selection is performed offline before model deployment, it does not add latency to online inference, and we consider this overhead acceptable in practical applications. All methods were evaluated on the same single-GPU configuration, so elapsed runtime provides a direct comparison of GPU time consumption; accordingly, we do not separately report the equivalent values in GPU-hours. The corresponding results have been added to Appendix H.

Response to Question (4): Thank you for your comment. We clarify that the proposed method aims to select calibration data by maximizing a joint score combining CARS and LSDS. Importantly, the metrics should be compared under the same calibration budget, rather than across different budgets. CalibSelect aims to select a calibration data set that achieves favorable CARS and LSDS scores under a fixed calibration budget. Thus, the relevant comparison in Table 7 is between Ours and Random at each individual budget. For Table 7, the key comparison is therefore between Ours and Random at the same calibration budget. As shown, across all four calibration budgets (16, 32, 64, and 128), our method consistently achieves higher CARS and LSDS scores than random sampling, while yielding lower PPL after quantization. These results indicate that, under a given calibration budget, our method can select calibration data that exhibit more diverse activation responses and more clearly distinguishable differences in layer-wise quantization sensitivity.

Reviewer_DkNKOpenReview ↗

feedback

Thank you for the detailed responses and additional experiments. however the substantially higher computational overhead of CalibSelect relative to existing methods remains a concern. I will keep my original score.

Tank you for carefully considering our response and for the continued attention to the computational overhead of CalibSelect. As calibration data selection is performed offline, our initial implementation did not prioritize execution efficiency. To address this concern, we further optimized CalibSelect by introducing multi-GPU parallelization while preserving the definitions of CARS and LSDS and the original two-stage selection procedure. Specifically, we distribute layer-wise quantization sensitivity evaluations across multiple GPUs for concurrent execution. This optimization reduces the overall wall-clock runtime from 1540.73s to 387.47s. This parallelization is enabled by the independence of the layer-wise sensitivity evaluations used in LSDS. For each evaluation, only the target Transformer layer is fake-quantized, while all other layers remain in full precision. The resulting change in NLL is measured through a full-model forward pass. Since each evaluation is independent of the results for other layers, these evaluations can be performed concurrently, and their results are subsequently aggregated to compute LSDS. Thus, parallelization changes only the execution strategy without altering the metric definitions or calibration data selection procedure. We hope these additional results help address your concerns regarding the computational overhead of CalibSelect. Thank you again for your valuable feedback.

Reviewer_FGMLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript under review aims to address selection of calibration data sets in PTQ. The authors propose CalibSelect. The method comprises two stages: Stage 1, generating candidate sets, and Stage 2, optimizing candidate sets.

Stage 1 consists of three steps:

Step1: M equal size calibration data sets are generated, 

Step 2: Two measures, the CARS (Calibration Activation Range Score) and LSDS (Layer Sensitivity Discrimination Score), are computed. 
   CARS aims to give an overall summary of activation range for a data set over all layers using the distance, in logarithmic scale, between high, say 90%, and low, say 10%, quantiles of different activation statistics. 
    LSDS aims to provide an overall measure of sensitivity of the model for a given data set. To this end, the authors first define sensitivity of a layer, say layer l, for a sample, say sample i,  using the discrepancy between the negative log-likelihood of the full precision model and the same model whose layer l is quantized. To come up with a stable estimate of sensitivity of a layer, they use the idea of repeated subsampling and calculate an average sensitivity measure for each layer, denoted by \mu_{l}, l=1, 2, ... , L. They also calculate variance of the sensitivity measure for each layer, denoted by \sigma^{2}_{l}, l=1, 2, ... , L. Having defined these measures for each layer, the LSDS is defined as the ratio of standard deviation of \mu_{l} divided by the  average of \sigma_{l}, with some adjustment in the denominator. Having calculated the two scores for all M data sets, they use the median and scaled inter-quartile of the CARS and LSDS values and map the values to (0 , 1) interval using the logistic function to make a standardized version of CARS and LSDS. Their final score for ranking the data sets, the so-called joint score in the manuscript, is a weighted version of the standardized CARS and LSDS.    

Step 3: Using the joint score data sets are ranked and the top K are selected. 

Stage 2 also consists of three steps:

Step 1: It initializes with the top k calibration data sets selected in Step 3 of Stage 1. 
Step 2: It iteratively improves each data set by replacing one selected sampled with one unselected and checked to see if the joint score improves.
Step 3: The algorithm stops if either there is no improvement in a number of pre-specified steps (N_{stop}) or we reach a pre-specified max number of iterations (N_{max}). 

The manuscript includes experiments on Qwen2.5-7B and LLaMA3.1-8B using AWQ and AutoRound on different calibration sets and compared their result with CoLa and ACMD.

Critical strengths
  1. The topic is important: Quantization is an important topic and although it has reached some level maturity now, there are still important issues are remaining that needs to be addressed. Sensitivity of quantization methods to calibration data sets is one of them.
Critical weaknesses
  1. The proposed method sounds interesting to me, but it seems half-cooked. Given its strong statistical basis, being aligned with ideas in design of experiment for choosing samples that increase information about the target of inference, I would hope to see a formal statement showing that the proposed method works better than its competitors. What presented in the article is some limited experiments. And although you have some improvement in PPL, the improvement in accuracy is almost negligible, about 0.5%

  2. I would also consider SpQR and Hessian-base quantizations (https://ojs.aaai.org/index.php/AAAI/article/view/33807)

  3. I did not see any comparison with SelectQ. (https://www.techrxiv.org/doi/full/10.36227/techrxiv.21456291.v2)

  4. Aside from the above points, there are methods which are zero-shot and calibration free. I think you should compare your methods with such methods. (https://arxiv.org/abs/2405.13358).

Questions for authors

Please see the weaknesses.

4 comments

Response to Comment (1): Thank you for your comment. In response, we have strengthened the analysis and added further experimental evaluations. We appreciate your expectation of a formal theoretical analysis. However, CARS and LSDS are heuristic proxy metrics motivated by the underlying quantization mechanisms, and a rigorous theoretical guarantee linking these metrics to final quantization performance remains difficult to establish. We therefore provide a more detailed explanation of their design rationale, clarify their advantages over existing calibration data selection strategies, and support our analysis with additional experimental evidence.

Existing calibration data selection methods primarily focus on sample representativeness in activation space, activation diversity, or alignment with a reference activation distribution. For example, CoLA selects representative samples based on hidden-state representations in the activation space, ACDM measures the distance between activation statistics and task-specific reference centers, and SelectQ employs hierarchical activation statistics and dynamic clustering to improve activation distribution coverage. While these methods improve the representativeness of calibration data, we argue that activation representativeness alone does not fully capture the information required for effective PTQ. A good calibration set should also provide information directly relevant to quantization error estimation and optimization. This motivates our design of CARS and LSDS from two complementary perspectives.

CARS is motivated by the fact that the output reconstruction error introduced by weight quantization depends directly on input activations. For a linear layer y=Wxy = Wx, the output reconstruction error induced by weight quantization can be written as $\mathcal{L}_{S}(\Delta W) =\underset{x\sim S}{\mathbb{E}} \left[\|\Delta W x\|_2^2\right] = \operatorname{tr}(\Delta WG_S \Delta W^\top)$, where GS=Ex∼S[xx⊤]G_S=\underset{x\sim S}{\mathbb{E}} \left[xx^\top\right] is the second-moment matrix of the calibration activations. This expression shows that second-order activation information directly affects the estimation of quantization reconstruction error. Calibration data therefore should not only be representative in the activation space but also provide sufficient coverage of activation statistics relevant to quantization error. To this end, CARS combines complementary statistics, including RMS, Mean Absolute, and high-order quantiles, to characterize overall activation energy, average magnitude, and high-magnitude responses. Unlike methods that primarily rely on hidden-state representations, activation distribution matching, or clustering-based representativeness, CARS explicitly evaluates activation coverage from the perspective of its relevance to quantization reconstruction error.

LSDS is motivated by the inherent differences in quantization sensitivity across Transformer layers. Quantization errors in different layers may affect the model’s overall performance to different degrees. If calibration data fail to reveal these differences, they may inadequately characterize the impact of layer-wise quantization perturbations. Revealing the layer differences may support subsequent quantization algorithms in reducing quantization errors in sensitive layers or adopting layer-wise and mixed-precision quantization strategies. LSDS therefore assesses the degree to which the calibration data discriminate among layer-wise sensitivity responses and the stability of these responses across samples. Unlike methods that select samples primarily based on activation distribution characteristics, LSDS assesses calibration data in terms of their ability to reveal the performance impact of quantization perturbations across different layers.

Taken together, CARS and LSDS evaluate calibration data from two complementary perspectives: activation coverage and inter-layer quantization sensitivity. CARS focuses on the input activation information relevant to quantization error estimation, whereas LSDS captures how quantization perturbations affect different layers. By incorporating both aspects into a unified calibration data selection objective, CalibSelect selects samples that provide more comprehensive activation statistics while better revealing layer-wise sensitivity differences. We believe this is a key reason why CalibSelect achieves better quantization performance than existing selection methods.

Continuation of Comment (1): We further conducted experiments to validate these motivations. First, the original experiments with AWQ and AutoRound show that CalibSelect achieves lower PPL than Random, CoLA, and ACDM in the vast majority of settings. Our ablation studies also demonstrate that jointly optimizing CARS and LSDS yields better quantization performance than using either metric alone, confirming their complementary contributions. Furthermore, following the suggestion, we added comparisons with SelectQ, extended our evaluation to GPTQ (4-bit) and SpQR (2-bit), and conducted additional experiments on vision-language models. These results further demonstrate the effectiveness and applicability of CalibSelect across different calibration data selection baselines, quantization algorithms, bit-widths, and model architectures.

In summary, although CARS and LSDS do not provide formal theoretical guarantees of final quantization performance, their designs are grounded in the underlying mechanisms of PTQ. Their effectiveness is further supported by ablation studies, comparisons with existing baselines, and evaluations across different quantization algorithms and model architectures.

Response to Comment (2): Thank you for the suggestion. In addition to the original AWQ and AutoRound experiments, we have included evaluations with SpQR (2-bit) and Hessian-based GPTQ (4-bit) to examine the applicability of CalibSelect across different PTQ algorithms. Using Qwen2.5-7B, we test four candidate pools (Mixed, Code, Math, and Common) and four calibration budgets (16, 32, 64, and 128). Detailed results have been added to Appendix E. With SpQR, CalibSelect achieves the lowest PPL across all four candidate pools and an overall average PPL of 15.4509, representing a 2.79% reduction relative to the strongest baseline, ACDM (15.8940). With GPTQ, it likewise achieves the lowest PPL across the four pools and an overall average PPL of 12.6215, a 1.69% reduction relative to the strongest baseline, CoLA (12.8383). These improvements can be attributed to the calibration data requirements of the two methods. SpQR uses Hessian-weighted quantization errors to identify outlier weights that require higher-precision representation, while GPTQ constructs a Hessian approximation from calibration activations and mitigates quantization errors through layer-wise quantization and second-order error compensation. Since both methods rely on calibration activations to estimate second-order information, CARS provides more comprehensive activation statistics for quantization error estimation. LSDS complements this by revealing layer-wise differences in quantization sensitivity, providing additional information for sensitivity assessment and error mitigation. Together, the two metrics can improve quantization performance. By jointly capturing these complementary aspects, CalibSelect provides more informative calibration data, which improves quantization performance.

Response to Comment (3): Thank you for pointing this out. We have added SelectQ as a baseline across all evaluated quantization methods. On Qwen2.5-7B, CalibSelect achieves average PPL values of 12.4765 with AWQ and 12.8320 with AutoRound, compared with 12.6693 and 13.2183 for SelectQ. These correspond to PPL reductions of 1.52% and 2.92%, respectively. On LLaMA3.1-8B, the corresponding average PPL values for CalibSelect are 11.0299 (AWQ) and 12.0803 (AutoRound), improving upon SelectQ by 1.90% and 1.21%, respectively. We have also evaluated SelectQ with GPTQ and SpQR. On Qwen2.5-7B, CalibSelect achieves average PPL values of 12.6215 and 15.4509, compared with 12.9560 and 16.2501 for SelectQ, representing reductions of 2.58% and 4.92%, respectively. All averages are computed across the four calibration budgets (16, 32, 64, and 128) and the four candidate pools (Mixed, Code, Math, and Common). CalibSelect achieves a lower average PPL than SelectQ for every evaluated model-quantization-method combination. We attribute the performance gains to CalibSelect's ability to identify calibration samples that better capture the information needed for quantization. Unlike SelectQ, which relies on layer-wise activation statistics and dynamic clustering, CalibSelect also accounts for the coverage of activation responses and variations in quantization sensitivity across layers. This allows it to select more informative calibration data, leading to better quantization performance. The revised manuscript includes the detailed SelectQ results in Appendices D and E.

Response to Comment (4): Thank you for the suggestion. We agree that zero-shot and calibration-free quantization methods are important research directions. However, our work specifically addresses calibration data selection for PTQ methods that require calibration samples. The objective is to improve quantization performance by selecting more suitable calibration data without modifying the underlying quantization algorithm or its optimization procedure. We therefore use Random, CoLA, ACDM, and SelectQ as direct calibration data selection baselines. In contrast, zero-shot and calibration-free methods aim to reduce or eliminate dependence on calibration data. They are consequently not direct baselines for isolating the contribution of our selection strategy.

Nevertheless, following your suggestion, we have also compared against two representative calibration-free quantization methods, HQQ and ReRound, to assess the competitiveness of the resulting quantized models. We compare PPL on the same models and evaluation datasets under 4-bit weight quantization. On Qwen2.5-7B, AWQ with CalibSelect achieves an average PPL of 12.4765, representing reductions of 3.20% and 1.23% relative to HQQ (12.8883) and ReRound (12.6313), respectively. AutoRound with CalibSelect achieves 12.8320, outperforming HQQ but not ReRound. On LLaMA3.1-8B, AWQ and AutoRound with CalibSelect achieve average PPL values of 11.0299 and 12.0803, both lower than HQQ (12.2489) and ReRound (12.3364). These findings indicate that CalibSelect, when combined with existing PTQ algorithms, can achieve competitive quantization performance.

Importantly, the added comparison evaluates the final performance of different quantization approaches, whereas the original experiments isolate the effect of calibration-data selection by fixing the downstream quantization algorithm. The two evaluations therefore provide complementary evidence. The calibration-free results have been added to Appendix J.

Thanks for your responses to my comments. I will consider your answers when discussing with other reviewers and the AC.

Reviewer_3bfrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a method to find a calibration dataset for post-training quantization. They propose two metrics: Calibration Activation Range Score (CARS) and Layer Sensitivity Discrimination Score (LSDS) to characterize layer behavior and guide the selection of the calibration dataset. The CARS metric measures the variation produced by the input calibration set. The LSDS metric measures the difference in sensitivity of the layers to quantization. Both of the metrics are optimized together to select the desired calibration set.

Critical strengths
  1. Two proposed metrics (CARS and LSDS) that seem to work and provide good empirical results.
Critical weaknesses
  1. It is not clear why both metrics work or what the intuition behind them is.
  2. CARS assumes that a broader activation range should get a better calibration set. Not sure how true that is; there should be an ablation study showing how broad the dispersion should be. Is there a sweet spot for this range, or can it be as broad as we can go, in which case it does not make a lot of sense?
  3. Similarly, the intuition behind the LSDS metric is not clear; why does increasing the difference in quantization sensitivity work? How big or small should the differences be?
Questions for authors

Overall, I like the paper. Please address the above weaknesses and provide ablation results for points 2 and 3. I will change my rating once I get the appropriate experimental results.

3 comments

Response to Comment (1): Thank you for your comment. CARS aims to assess whether calibration data can adequately cover the activation states commonly encountered during inference. LSDS evaluates whether calibration data adequately reveal the layer-wise differences in quantization sensitivity inherent to the model. We elaborate below on the motivation for these two objectives and their relevance to PTQ.

The motivation behind CARS is to select calibration data whose induced activation responses approximate the activation states commonly encountered during actual inference. This should help ensure that quantization parameters estimated or optimized using calibration data generalize to inference-time inputs. However, the activation distribution during actual inference is difficult to characterize accurately in advance [1]. We therefore use the statistics of activation responses induced by calibration samples, together with the coverage of these statistics, as a proxy for how well the calibration data set represents activation states during inference. We hypothesize that more comprehensive coverage of activation statistics increases the likelihood of including common activation states under actual inference. Importantly, we do not assume that a wider activation range invariably implies better calibration quality. Overly narrow coverage may reflect only a limited subset of activation states, whereas an apparent range increase driven by a few extreme samples may contribute little to representing common activation states under actual inference. Our objective is therefore to obtain sufficiently informative activation coverage without allowing rare or extreme samples to dominate, rather than to maximize the range without constraint. Computing high-dimensional activation information directly can be computationally and memory intensive. Accordingly, we employ several complementary, low-cost statistics as approximate descriptors of activation responses. Specifically, squared RMS reflects the overall second-order energy of activations; mean absolute activation captures their average absolute magnitude; and the 95th and 99th percentiles (Q95 and Q99) characterize the upper tail, providing additional information about high-magnitude responses. CARS then measures the relative variation of these statistics across calibration samples as a proxy. A higher CARS indicates more pronounced differences in these statistical characteristics across samples, which may help identify calibration sets containing richer activation information and reduce the risk that quantization optimization is tailored to only a narrow set of activation states. Further experiments examining how CARS relates to quantization performance, including whether an optimal level of activation coverage exists, are presented in Response (2).

For LSDS, beyond representing common activation states, we argue that calibration data should reflect the layer-wise differences already present in the model. Some Transformer layers are more consequential for quantization than others. Revealing these differences may support subsequent quantization algorithms in reducing reconstruction errors in sensitive layers or adopting layer-wise and mixed-precision quantization strategies. Accordingly, LSDS examines whether a calibration set reveals inherent layer-wise quantization sensitivity differences clearly and consistently. Such information may be useful for downstream PTQ calibration and reconstruction. To characterize sensitivity differences in the quantization process, we independently apply fake quantization to each Transformer layer and measure the change in negative log-likelihood (NLL) relative to the full-precision model. LSDS evaluates the discrimination between layers based on these NLL changes, together with the stability of the estimates across calibration samples. A higher LSDS suggests that calibration data reveal the differential impact of quantizing individual layers more clearly and may therefore provide more informative signals for subsequent PTQ. Response (3) provides additional experiments to evaluate the relationship between LSDS and quantization performance and to assess whether the layer-wise sensitivity differences captured by LSDS are representative.

In summary, CARS assesses how well calibration data cover the activation states commonly encountered during inference, while LSDS evaluates whether calibration data can clearly and stably reveal the model's inherent differences in layer-wise quantization sensitivity. These two metrics complement each other, providing more informative calibration data for subsequent quantization error estimation and parameter optimization in PTQ.

Continuation of Comment (1): Given that CARS and LSDS are empirically motivated proxy metrics, we conducted additional experiments to further validate their effectiveness and generalizability. Specifically, we extended the baseline comparisons to include SelectQ and evaluated two additional PTQ methods: Hessian-based GPTQ (4-bit) and SpQR (2-bit). CalibSelect achieves lower average PPL than SelectQ across all evaluated model-quantization method combinations. On Qwen2.5-7B, it achieves average PPLs of 12.6215 with GPTQ and 15.4509 with SpQR, outperforming the best baselines by 1.69% and 2.79%, respectively. We also evaluated CalibSelect on the vision-language model Qwen2-VL-7B-Instruct, where it achieves the lowest PPL at three of the four calibration budgets. Finally, comparisons with two representative calibration-free methods, HQQ and ReRound, demonstrate that CalibSelect combined with existing PTQ algorithms achieves competitive quantization performance.

[1] Miles Williams, George Chrysostomou, Nikolaos Aletras. Self-calibration for Language Model Quantization and Pruning, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, 2025. [2] Razvan-Gabriel Dumitru, Vikas Yadav, Rishabh Maheshwary, Paul Ioan Clotan, Sathwik Tejaswi Madhusudhan, Mihai Surdeanu. Variable Layerwise Quantization: A Simple and Effective Approach to Quantize LLMs, Findings of ACL 2025.

Response to Comment (2): Thank you for your comment. As discussed in Response (1), CARS seeks to improve the coverage of activation states relevant to inference. This does not imply that all possible states must be covered. Extending coverage to states that are rarely encountered during inference may offer only limited additional quantization benefit. To further examine the useful range of CARS, we compare the PPL obtained after AWQ quantization using calibration data sets with different CARS scores. As shown in Fig. 7(a) of Appendix G in the manuscript, PPL initially decreases as CARS increases, although the improvement gradually diminishes and some fluctuations remain. This observation suggests a possible saturation effect in the benefits of increasing activation range.

In response to your specific questions: (i) Is there an optimal activation range? A range beyond which additional coverage yields diminishing benefits may exist. However, identifying a precise and universally applicable optimum is difficult because activation distributions vary with the model, task, and data distribution. (ii) Is a wider range always better? No. Maximizing CARS within a finite candidate space is intended to identify calibration sets with richer activation responses, not to expand the activation range without limit. Moreover, the ablation results in Table 2 show that maximizing CARS alone does not yield the best quantization performance; combining it with LSDS produces better results.

Response to Comment (3): Thank you for your comment. We have further clarified the motivation for LSDS in Response (1). We conduct two additional analyses to examine (i) whether revealing these differences is associated with improved quantization performance and (ii) whether the layer-wise differences revealed by LSDS are representative First, we examine the PPL obtained after AWQ quantization using calibration sets with different LSDS scores. As shown in Fig. 7(a) of Appendix G in the manuscript, PPL generally decreases with increasing LSDS among the displayed calibration sets, while the improvement diminishes and fluctuations remain at higher LSDS scores.
Second, we compare the calibration sets of 128 samples selected by CoLA, ACDM, SelectQ, and LSDS against a reference set of 2,000 samples drawn from a pretraining corpus. As shown in Fig. 8 of Appendix G in the manuscript, the resulting layer-wise sensitivity profiles vary across selection methods. CoLA and SelectQ capture some of the layer-wise trends but exhibit discrepancies at certain sensitive layers, while ACDM shows more pronounced fluctuations across several layers. In contrast, the LSDS calibration set better captures the major sensitivity trends and the locations of several sensitive layers observed in the reference set. Although some discrepancies remain at individual layers, the two profiles exhibit broadly consistent overall trends. This result suggests that LSDS can identify a small calibration set that captures key layer-wise sensitivity characteristics revealed by a much larger reference set.

Continuation of Comment (3): In response to your specific questions: (i) Why might clearer layer-wise sensitivity differences improve quantization? Calibration data that more effectively expose the model’s inherent layer-wise differences may provide more representative quantization-sensitivity information, thereby informing subsequent PTQ calibration and reconstruction. (ii) How large should these differences be? There is no universal numerical threshold. Weight distributions, error-propagation characteristics, and relative layer importance vary across models. LSDS therefore does not require sensitivity differences to reach a predefined value; rather, it aims to identify calibration data that adequately reveal the differences already present under the model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.