- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces the calibration activation range score (CARS) and the layer sensitivity discrimination score (LSDS)., CARS assesses activation response coverage by analyzing the mean absolute activation, root mean square, and high-percentile activation values of calibration data across Transformer layers. LSDS evaluates how clearly and stably calibration data capture layer-wise differences in quantization sensitivity. Also proposes a calibration data assessment and selection framework called CalibSelect. Which integrates CARS and LSDS to guide data selection for PTQ.
Critical strengths
This paper addresses a practical and important problem. and introduced a quantization aware data quality assessment framework. From experiments side, it compatible with existing PTQ methods.
Critical weaknesses
I see some results on 4-bit weight quantization but not other number of bits. have test on other number of bits? The novelty of the CARS metric is limited. It's a combination of existing methods like Mean absolute activation, RMS, percentile and quantile range. What's the motivation to combine these pieces together as a metric.
Questions for authors
-
All the experimental results are tested on LLMs, doesn't mean this method is designed for LLM architecture.
-
Can this model used on other VLMs, diffusion models or other types of model?
-
How about the computational cost and efficiency compare with other methods?
-
From table 7, on the right columns "Ours", the performance for LSDS seems increase when budget from 16-->128 while CARS decreases. I am confused if a higher score is needed or lower score is needed for the 2 metrics.
4 comments
Response to Question (4): Thank you for your comment. We clarify that the proposed method aims to select calibration data by maximizing a joint score combining CARS and LSDS. Importantly, the metrics should be compared under the same calibration budget, rather than across different budgets. CalibSelect aims to select a calibration data set that achieves favorable CARS and LSDS scores under a fixed calibration budget. Thus, the relevant comparison in Table 7 is between Ours and Random at each individual budget. For Table 7, the key comparison is therefore between Ours and Random at the same calibration budget. As shown, across all four calibration budgets (16, 32, 64, and 128), our method consistently achieves higher CARS and LSDS scores than random sampling, while yielding lower PPL after quantization. These results indicate that, under a given calibration budget, our method can select calibration data that exhibit more diverse activation responses and more clearly distinguishable differences in layer-wise quantization sensitivity.
feedback
Thank you for the detailed responses and additional experiments. however the substantially higher computational overhead of CalibSelect relative to existing methods remains a concern. I will keep my original score.
Tank you for carefully considering our response and for the continued attention to the computational overhead of CalibSelect. As calibration data selection is performed offline, our initial implementation did not prioritize execution efficiency. To address this concern, we further optimized CalibSelect by introducing multi-GPU parallelization while preserving the definitions of CARS and LSDS and the original two-stage selection procedure. Specifically, we distribute layer-wise quantization sensitivity evaluations across multiple GPUs for concurrent execution. This optimization reduces the overall wall-clock runtime from 1540.73s to 387.47s. This parallelization is enabled by the independence of the layer-wise sensitivity evaluations used in LSDS. For each evaluation, only the target Transformer layer is fake-quantized, while all other layers remain in full precision. The resulting change in NLL is measured through a full-model forward pass. Since each evaluation is independent of the results for other layers, these evaluations can be performed concurrently, and their results are subsequently aggregated to compute LSDS. Thus, parallelization changes only the execution strategy without altering the metric definitions or calibration data selection procedure. We hope these additional results help address your concerns regarding the computational overhead of CalibSelect. Thank you again for your valuable feedback.
Response to Critical Weaknesses: Thank you for your comment. We have addressed the two concerns as follows. (i) Quantization bit-width. We have added experiments with 2-bit weight quantization. Detailed results are reported in Appendix E. CalibSelect achieves an overall average PPL of 15.4509, a 2.79% reduction relative to the strongest baseline, ACDM (15.8940), and a 4.92% reduction relative to SelectQ (16.2501). These additional experiments evaluate the applicability of CalibSelect under a lower quantization bit-width. (ii) Motivation for CARS. We agree that RMS, mean absolute activation, and quantiles are established elementary statistics. Their combination in CARS, however, is motivated by the relationship between PTQ reconstruction error and second-order activation information. For a linear layer y=Wx, the output reconstruction error induced by weight quantization can be written as $\mathcal{L}_{S}(\Delta W) =\underset{x\sim S}{\mathbb{E}} \left[\|\Delta W x\|_2^2\right] = \operatorname{tr}(\Delta WG_S \Delta W^\top)$, where GS=x∼SE[xx⊤] is the second-moment matrix of the calibration activations. This expression shows that second-order activation information directly affects the estimation of quantization reconstruction error. However, using high-dimensional second-moment matrices to compare activation responses across calibration samples incurs substantial computational and storage costs. We therefore employ several complementary low-cost statistics as approximate descriptors of activation characteristics. Squared RMS corresponds to the normalized trace of the second-moment matrix and reflects overall second-order activation energy. Mean absolute activation characterizes average magnitude, while Q95 and Q99 supplement information about higher-magnitude responses. Since no single statistic fully describes the activation-magnitude distribution, we combine these descriptors to assess activation states from the perspectives of overall energy, average magnitude, and upper-tail behavior.
Response to Questions (1) and (2): Thank you for your comments. To address whether CalibSelect is restricted to LLMs architectures and whether it can be extended to other model types, we have further examined its applicability and added a VLMs experiment. The design of CalibSelect is not tailored to a particular LLM architecture, so it may extend to other models. Following your suggestion, we evaluate CalibSelect on the Qwen2-VL-7B-Instruct using AWQ. We construct the candidate calibration pool by combining the GQA and TextVQA training sets and form a single evaluation set by combining their validation sets. Under calibration budgets of 16, 32, 64, and 128, we compare CalibSelect with Random, CoLA, ACDM, and SelectQ. As shown in Fig. 9 of Appendix I in the manuscript, CalibSelect achieves the lowest PPL at three of the four budgets (16, 64, and 128). Although it is not the best method at a budget of 32, the results indicate promising quantization performance on the VLMs. These findings provide additional empirical evidence that the approach can extend beyond text-only LLMs to other models. The corresponding results have been added to Appendix I.
Response to Question (3): Thank you for your comment. We have added a runtime comparison with other calibration data selection methods. Under the same GPU configuration, the average runtimes of CoLA, ACDM, and SelectQ are approximately 258s, 127s, and 99s, respectively, whereas the complete CalibSelect procedure takes approximately 1541s. Although our method incurs higher computational overhead, its primary advantage is improved quantization performance rather than faster calibration data selection. In the Qwen2.5-7B AWQ experiments, CalibSelect achieves an average PPL of 12.4765, outperforming CoLA (12.7154), ACDM (12.7615), and SelectQ (12.7057). Thus, CalibSelect trades additional offline calibration data selection cost for better quantization performance. Because calibration data selection is performed offline before model deployment, it does not add latency to online inference, and we consider this overhead acceptable in practical applications. All methods were evaluated on the same single-GPU configuration, so elapsed runtime provides a direct comparison of GPU time consumption; accordingly, we do not separately report the equivalent values in GPU-hours. The corresponding results have been added to Appendix H.