- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CRCSS, a conformal calibration framework for selecting LLM pruning sparsity under a user-specified performance-degradation constraint. It converts each sample’s non-monotonic degradation trajectory into a cumulative maximum, identifies its largest safe sparsity prefix, and calibrates a global sparsity threshold using a corrected empirical quantile. Under exchangeability and a fixed candidate pruning path, the method provides a finite-sample marginal guarantee that a future sample’s degradation remains within tolerance throughout the selected candidate prefix. Evaluation covers five LLMs, three pruning methods, and three benchmarks. Using answer-attempt counts and correct-answer confidence as performance measures, the experiments report average violation rates within the target levels.
Critical strengths
- A practically relevant formulation of sparsity selection. The paper addresses how to choose a pruning level under an explicit tolerance for sample-level performance degradation. The formulation applies to existing pruning methods and makes the trade-off between sparsity and reliability explicit. This is a useful application of statistical calibration to a concrete model-compression decision.
- Empirical evidence supporting the formulation. The evaluation spans five LLMs, three pruning methods, and three benchmarks. Table 2 reports that removing the prefix construction causes the mean prefix-wide violation rate to exceed its target, whereas CRCSS meets the targets throughout. This supports the practical relevance of accounting for non-monotonic degradation and strengthens confidence in the method’s applicability across the tested settings.
Critical weaknesses
- Limited methodological novelty beyond existing conformal risk control. The central construction is directly recoverable from established CRC results. Corollary 1 in Section 2.3 of Angelopoulos et al. (Conformal Risk Control, ICLR 2024) already handles non-monotonic losses through a worst-case envelope; Section 4.2 controls threshold-exceedance probabilities, and Appendix A establishes the corresponding quantile calibration rule. Applying these results to a binary indicator of prefix violations, with the parameter transformation, recovers both the proposed sparsity selection rule and its guarantee. The paper does not clearly identify a substantial algorithmic or theoretical contribution beyond this specialization to pruning. This limits the methodological contribution central to my assessment.
- Insufficient motivation and practical significance for the application. The paper does not convincingly establish why this formulation constitutes a sufficiently significant contribution for ICLR. A statistical guarantee for unseen samples from the same distribution can be useful, but this guarantee follows from existing CRC machinery; its availability alone does not establish the importance of the pruning application. The paper needs to demonstrate where calibration materially improves upon conventional validation-based sparsity selection, and why certifying every lower candidate sparsity is useful when deploying a selected model. These benefits remain insufficiently demonstrated. The empirical gains are also often modest: weighting the plotted frequencies in Figure 10 yields an average sparsity increase of approximately 2.1 percentage points across 24 configurations. No corresponding deployment savings or calibration costs are reported, and the differing guarantees further limit the interpretation of the LTT comparison. Overall, the application’s demonstrated significance is insufficient to compensate for the limited methodological novelty.
Questions for authors
-
Clarification of the contribution beyond existing CRC. Does CRCSS differ mathematically from applying CRC’s non-monotonic loss construction and quantile calibration to a binary loss indicating whether any candidate sparsity up to exceeds the degradation tolerance? The relevant results appear in Section 2.3, Section 4.2, and Appendix A of Angelopoulos et al. (Conformal Risk Control, ICLR 2024). If there is a difference, which specific step prevents this reduction? If the methods are equivalent, what independent pruning-specific insight constitutes the paper’s main contribution?
-
Clarification of the motivation and demonstrated practical value. What concrete deployment scenario requires certifying the entire candidate sparsity prefix, rather than only the selected deployment model? Which results in the current study demonstrate that this requirement provides a meaningful benefit over conventional validation-based sparsity selection, considering its potential conservatism and calibration cost? In particular, what practical advantage can be concluded from the reported sparsity gains over LTT, given the different statistical guarantees acknowledged?
7 comments
Response to question 1
-
1-1 Yes. CRCSS is mathematically equivalent to applying CRC’s monotone envelope for non-monotonic losses and quantile calibration to a binary loss that indicates whether the degradation tolerance is exceeded at any candidate sparsity up to . In the revised manuscript, we also show that the marginal guarantee implies an upper bound on the mean prefix risk. If the per-example degradation at every candidate sparsity is bounded above by , then the expected maximum degradation over the selected prefix is at most . The proof is provided in Appendix A.2.
-
1-2 Our pruning-specific contribution is to select sparsity while explicitly controlling the risk of sample-level performance degradation. In Section 4.2 of the revised manuscript, we added experiments on Llava-3.1-8B-Instruct to examine how different sparsity levels affect aggregate performance and per-example answering behavior. We found that stable aggregate performance can mask substantial degradation on individual examples, with a subset of examples bearing a disproportionate share of the performance loss. This finding is consistent with As Easy as PIE: Understanding When Pruning Causes Language Models to Disagree (Findings of NAACL 2025) and Selective Brain Damage: Measuring the Disparate Impact of Model Pruning (Hooker et al., Google Research, 2019). The table below reports how the answering behavior of Llava-3.1-8B-Instruct at each sparsity level differs from that of its dense version.
| Sparsity | Improved | Unchanged | +1 attempt | +2 or more attempts (solved) | Unsolved within 5 attempts |
|---|---|---|---|---|---|
| 0% | 0.00% | 100.00% | 0.00% | 0.00% | 0.00% |
| 10% | 2.80% | 88.52% | 4.41% | 2.20% | 2.07% |
| 20% | 4.43% | 87.28% | 2.41% | 2.00% | 3.88% |
| 30% | 5.18% | 84.17% | 3.07% | 2.34% | 5.25% |
| 40% | 5.58% | 79.79% | 4.04% | 2.88% | 7.71% |
| 50% | 4.54% | 65.98% | 10.72% | 9.19% | 9.57% |
| 60% | 3.56% | 46.22% | 13.96% | 15.92% | 20.33% |
| 70% | 1.77% | 9.29% | 5.93% | 10.82% | 72.19% |
Response to question 2
-
2-1 A concrete use case is deploying the same QA service across devices with different GPU memory or latency budgets. Resource-constrained devices can use a higher-sparsity model, and devices with more resources can use a lower-sparsity model from the same candidate prefix. Certifying the entire prefix allows the deployment team to choose a model for each device while retaining the same prefix risk guarantee. Prior work illustrates this deployment pattern. ZipLM: Inference-Aware Structured Pruning of Language Models (NeurIPS 2023) constructs a family of pruned models for different target speedups. Flextron: Many-in-One Flexible Large Language Model (ICML 2024) and MatFormer: Nested Transformer for Elastic Inference (NeurIPS 2024) provide selectable language-model subnetworks for different inference budgets. For a single-model deployment that prioritizes inference efficiency, the team can also deploy the model at the highest sparsity selected by CRCSS.
-
2-2 In Sections 4.2–4.3 and Appendix C.2 of the revised manuscript, we compare CRCSS with MAS, a baseline that selects sparsity using average validation accuracy. In the single-attempt experiments on CommonsenseQA, we set and use the marginal accuracy lower bound implied by CRCSS, , as the selection threshold for MAS. Tables 1, 2, 5, and 6 report the endpoint empirical violation rate (EVR), test accuracy, prefix EVR, and mean prefix risk, respectively, along with the number of runs out of 100 in which each metric met its criterion. The table below summarizes the results under the most stringent criterion by counting the configurations that met the corresponding criterion in all 100 runs. The revised manuscript reports the exact number of successful runs for each configuration. Although MAS generally selects sparsity more aggressively, CRCSS provides more reliable risk control and, for each of the four metrics, meets the corresponding criterion in all 100 runs for substantially more configurations.
| Test metric | CRCSS configurations meeting the target on all 100 splits | MAS configurations meeting the target on all 100 splits |
|---|---|---|
| Endpoint EVR (Table 1) | 29/30 | 11/30 |
| Prefix EVR (Table 5) | 19/30 | 3/30 |
| Mean prefix risk (Table 6) | 19/30 | 3/30 |
| Test accuracy (Table 2) | 30/30 | 21/30 |
- 2-3. The comparison with LTT in the main text focuses on the sparsity each method selects when the marginal risk bounds are similar. Given the calibration set , let denote the probability that a new sample's performance degradation exceeds the tolerance at one or more candidate sparsity levels up to and including the selected sparsity . CRCSS directly bounds the marginal risk by . LTT requires , which implies a marginal risk bound of . At , the bounds differ by less than one percentage point, and CRCSS tends to select higher sparsity. Appendix C.5 of the revised manuscript examines LTT’s sensitivity to . As increased from 0.01 to 0.10 and then to 0.30, the average gap in selected sparsity between CRCSS and LTT narrowed from 3.99 to 3.02 and then to 1.32 percentage points. This quantifies how much sparsity LTT gives up at higher confidence levels in this setting. The choice between guarantees depends on the deployment objective. If the objective is to control the probability of a prefix violation jointly over the calibration set and a future sample, CRCSS directly provides that marginal guarantee. If the objective is for the population prefix violation risk at the sparsity selected in a single calibration run to be at most with probability at least over draws of the calibration set, LTT provides that guarantee.
Manuscript Revisions
We are very grateful to the reviewers for their valuable feedback and for pointing out potential issues in the original manuscript.
We revised the manuscript in response to the reviewers’ comments. The changes are marked in blue.