acceptodds
Under review as a conference paper at ICLR 2027

Conformal Risk Control for Sparsity Selection in Large Language Models

Abstract

Model pruning can effectively reduce the deployment cost of large language models (LLMs), while selecting an appropriate sparsity level is critical to balancing compression gains against performance preservation. Existing methods typically assess a prespecified sparsity level on the basis of its empirical performance on held-out evaluation data. However, this strategy lacks statistical reliability and may lead to uncontrolled performance degradation in real-world deployment. Therefore, we extend split conformal prediction, which can provide statistically valid guarantees for unseen samples under the exchangeability assumption, to pruning sparsity selection. Meanwhile, we propose CRCSS, a performance-degradation risk control framework for sparsity selection in LLM pruning. Specifically, we transform non-monotonic performance degradation trajectories into nondecreasing degradation prefix to determine sample-specific maximum safe sparsity boundaries, thereby formulating pruning sparsity selection as a conformal calibration problem. Extensive experiments across multiple LLMs, pruning methods and language understanding benchmarks demonstrate that CRCSS achieves empirical guarantee rates consistent with the prespecified reliability targets, while enabling a controllable trade-off between reliability and compression.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Large Language Models, Model Pruning, Sparsity Selection, Conformal Risk Control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_g7dhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CRCSS, a conformal calibration framework for selecting LLM pruning sparsity under a user-specified performance-degradation constraint. It converts each sample’s non-monotonic degradation trajectory into a cumulative maximum, identifies its largest safe sparsity prefix, and calibrates a global sparsity threshold using a corrected empirical quantile. Under exchangeability and a fixed candidate pruning path, the method provides a finite-sample marginal guarantee that a future sample’s degradation remains within tolerance throughout the selected candidate prefix. Evaluation covers five LLMs, three pruning methods, and three benchmarks. Using answer-attempt counts and correct-answer confidence as performance measures, the experiments report average violation rates within the target levels.

Critical strengths
  1. A practically relevant formulation of sparsity selection. The paper addresses how to choose a pruning level under an explicit tolerance for sample-level performance degradation. The formulation applies to existing pruning methods and makes the trade-off between sparsity and reliability explicit. This is a useful application of statistical calibration to a concrete model-compression decision.
  2. Empirical evidence supporting the formulation. The evaluation spans five LLMs, three pruning methods, and three benchmarks. Table 2 reports that removing the prefix construction causes the mean prefix-wide violation rate to exceed its target, whereas CRCSS meets the targets throughout. This supports the practical relevance of accounting for non-monotonic degradation and strengthens confidence in the method’s applicability across the tested settings.
Critical weaknesses
  1. Limited methodological novelty beyond existing conformal risk control. The central construction is directly recoverable from established CRC results. Corollary 1 in Section 2.3 of Angelopoulos et al. (Conformal Risk Control, ICLR 2024) already handles non-monotonic losses through a worst-case envelope; Section 4.2 controls threshold-exceedance probabilities, and Appendix A establishes the corresponding quantile calibration rule. Applying these results to a binary indicator of prefix violations, with the parameter transformation, recovers both the proposed sparsity selection rule and its guarantee. The paper does not clearly identify a substantial algorithmic or theoretical contribution beyond this specialization to pruning. This limits the methodological contribution central to my assessment.
  2. Insufficient motivation and practical significance for the application. The paper does not convincingly establish why this formulation constitutes a sufficiently significant contribution for ICLR. A statistical guarantee for unseen samples from the same distribution can be useful, but this guarantee follows from existing CRC machinery; its availability alone does not establish the importance of the pruning application. The paper needs to demonstrate where calibration materially improves upon conventional validation-based sparsity selection, and why certifying every lower candidate sparsity is useful when deploying a selected model. These benefits remain insufficiently demonstrated. The empirical gains are also often modest: weighting the plotted frequencies in Figure 10 yields an average sparsity increase of approximately 2.1 percentage points across 24 configurations. No corresponding deployment savings or calibration costs are reported, and the differing guarantees further limit the interpretation of the LTT comparison. Overall, the application’s demonstrated significance is insufficient to compensate for the limited methodological novelty.
Questions for authors
  1. Clarification of the contribution beyond existing CRC. Does CRCSS differ mathematically from applying CRC’s non-monotonic loss construction and quantile calibration to a binary loss indicating whether any candidate sparsity up to ss exceeds the degradation tolerance? The relevant results appear in Section 2.3, Section 4.2, and Appendix A of Angelopoulos et al. (Conformal Risk Control, ICLR 2024). If there is a difference, which specific step prevents this reduction? If the methods are equivalent, what independent pruning-specific insight constitutes the paper’s main contribution?

  2. Clarification of the motivation and demonstrated practical value. What concrete deployment scenario requires certifying the entire candidate sparsity prefix, rather than only the selected deployment model? Which results in the current study demonstrate that this requirement provides a meaningful benefit over conventional validation-based sparsity selection, considering its potential conservatism and calibration cost? In particular, what practical advantage can be concluded from the reported sparsity gains over LTT, given the different statistical guarantees acknowledged?

7 comments
AuthorsOpenReview ↗

Manuscript Revisions

We are very grateful to the reviewers for their valuable feedback and for pointing out potential issues in the original manuscript.

We revised the manuscript in response to the reviewers’ comments. The changes are marked in blue.

  • Section 4.2 compares the endpoint EVRs of CRCSS and MAS and shows that nearly unchanged aggregate accuracy can mask degradation on individual examples.
  • Section 4.3 compares the selected sparsity and test accuracy of CRCSS and MAS under the same accuracy target.
  • Appendix A.2 derives an upper bound on mean prefix risk from the marginal guarantee.
  • Appendix C.2 compares the prefix EVR and mean prefix risk of CRCSS and MAS and reports the number of successful runs out of 100.
  • Appendix C.3 reports prefix EVR in the multiple-attempt setting.
  • Appendix C.4 reports the sparsity levels selected by CRCSS at different risk levels on CommonsenseQA with Wanda and SparseGPT.
  • Appendix C.5 clarifies the statistical guarantees of CRCSS and LTT and analyzes LTT’s sensitivity to δ\delta.
AuthorsOpenReview ↗

Response to question 1

  • 1-1 Yes. CRCSS is mathematically equivalent to applying CRC’s monotone envelope for non-monotonic losses and quantile calibration to a binary loss that indicates whether the degradation tolerance is exceeded at any candidate sparsity up to s^\hat{s}. In the revised manuscript, we also show that the marginal guarantee implies an upper bound on the mean prefix risk. If the per-example degradation at every candidate sparsity is bounded above by B≥ϵB\geq\epsilon, then the expected maximum degradation over the selected prefix is at most ϵ+(B−ϵ)α\epsilon+(B-\epsilon)\alpha. The proof is provided in Appendix A.2.

  • 1-2 Our pruning-specific contribution is to select sparsity while explicitly controlling the risk of sample-level performance degradation. In Section 4.2 of the revised manuscript, we added experiments on Llava-3.1-8B-Instruct to examine how different sparsity levels affect aggregate performance and per-example answering behavior. We found that stable aggregate performance can mask substantial degradation on individual examples, with a subset of examples bearing a disproportionate share of the performance loss. This finding is consistent with As Easy as PIE: Understanding When Pruning Causes Language Models to Disagree (Findings of NAACL 2025) and Selective Brain Damage: Measuring the Disparate Impact of Model Pruning (Hooker et al., Google Research, 2019). The table below reports how the answering behavior of Llava-3.1-8B-Instruct at each sparsity level differs from that of its dense version.

SparsityImprovedUnchanged+1 attempt+2 or more attempts (solved)Unsolved within 5 attempts
0%0.00%100.00%0.00%0.00%0.00%
10%2.80%88.52%4.41%2.20%2.07%
20%4.43%87.28%2.41%2.00%3.88%
30%5.18%84.17%3.07%2.34%5.25%
40%5.58%79.79%4.04%2.88%7.71%
50%4.54%65.98%10.72%9.19%9.57%
60%3.56%46.22%13.96%15.92%20.33%
70%1.77%9.29%5.93%10.82%72.19%
AuthorsOpenReview ↗

Response to question 2

  • 2-1 A concrete use case is deploying the same QA service across devices with different GPU memory or latency budgets. Resource-constrained devices can use a higher-sparsity model, and devices with more resources can use a lower-sparsity model from the same candidate prefix. Certifying the entire prefix allows the deployment team to choose a model for each device while retaining the same prefix risk guarantee. Prior work illustrates this deployment pattern. ZipLM: Inference-Aware Structured Pruning of Language Models (NeurIPS 2023) constructs a family of pruned models for different target speedups. Flextron: Many-in-One Flexible Large Language Model (ICML 2024) and MatFormer: Nested Transformer for Elastic Inference (NeurIPS 2024) provide selectable language-model subnetworks for different inference budgets. For a single-model deployment that prioritizes inference efficiency, the team can also deploy the model at the highest sparsity selected by CRCSS.

  • 2-2 In Sections 4.2–4.3 and Appendix C.2 of the revised manuscript, we compare CRCSS with MAS, a baseline that selects sparsity using average validation accuracy. In the single-attempt experiments on CommonsenseQA, we set ϵ=0\epsilon=0 and use the marginal accuracy lower bound implied by CRCSS, ACC(0)−α\mathrm{ACC}(0)-\alpha, as the selection threshold for MAS. Tables 1, 2, 5, and 6 report the endpoint empirical violation rate (EVR), test accuracy, prefix EVR, and mean prefix risk, respectively, along with the number of runs out of 100 in which each metric met its criterion. The table below summarizes the results under the most stringent criterion by counting the configurations that met the corresponding criterion in all 100 runs. The revised manuscript reports the exact number of successful runs for each configuration. Although MAS generally selects sparsity more aggressively, CRCSS provides more reliable risk control and, for each of the four metrics, meets the corresponding criterion in all 100 runs for substantially more configurations.

Test metricCRCSS configurations meeting the target on all 100 splitsMAS configurations meeting the target on all 100 splits
Endpoint EVR (Table 1)29/3011/30
Prefix EVR (Table 5)19/303/30
Mean prefix risk (Table 6)19/303/30
Test accuracy (Table 2)30/3021/30
  • 2-3. The comparison with LTT in the main text focuses on the sparsity each method selects when the marginal risk bounds are similar. Given the calibration set Dcal\mathcal D_{\mathrm{cal}}, let r(Dcal)r(\mathcal D_{\mathrm{cal}}) denote the probability that a new sample's performance degradation exceeds the tolerance at one or more candidate sparsity levels up to and including the selected sparsity s^\hat{s}. CRCSS directly bounds the marginal risk by EDcal[r(Dcal)]≤α\mathbb E_{\mathcal D_{\mathrm{cal}}}[r(\mathcal D_{\mathrm{cal}})]\leq\alpha. LTT requires Pr⁡Dcal(r(Dcal)>α)≤δ\Pr_{\mathcal D_{\mathrm{cal}}}(r(\mathcal D_{\mathrm{cal}})>\alpha)\leq\delta, which implies a marginal risk bound of α+(1−α)δ\alpha+(1-\alpha)\delta. At δ=0.01\delta=0.01, the bounds differ by less than one percentage point, and CRCSS tends to select higher sparsity. Appendix C.5 of the revised manuscript examines LTT’s sensitivity to δ\delta. As δ\delta increased from 0.01 to 0.10 and then to 0.30, the average gap in selected sparsity between CRCSS and LTT narrowed from 3.99 to 3.02 and then to 1.32 percentage points. This quantifies how much sparsity LTT gives up at higher confidence levels in this setting. The choice between guarantees depends on the deployment objective. If the objective is to control the probability of a prefix violation jointly over the calibration set and a future sample, CRCSS directly provides that marginal guarantee. If the objective is for the population prefix violation risk at the sparsity selected in a single calibration run to be at most α\alpha with probability at least 1−δ1-\delta over draws of the calibration set, LTT provides that guarantee.
Reviewer_YqdVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CRCSS, a conformal risk control framework for selecting the sparsity level of pruned large language models under predefined performance degradation constraints. The method consists of three stages: constructing a monotonic degradation prefix from potentially non-monotonic performance degradation trajectories, identifying sample-specific safe sparsity boundaries, and applying split conformal calibration to select a global sparsity threshold. Under the exchangeability assumption, the framework provides a finite-sample marginal guarantee for unseen samples. Experiments across five LLMs, three pruning methods, and three benchmarks demonstrate empirical risk control under different risk levels. The authors also compare CRCSS with Learn-then-Test (LTT) and report higher selected sparsity levels in most evaluated configurations.

Critical strengths
  1. Well-motivated problem formulation. The paper addresses a meaningful problem in LLM compression: selecting an appropriate pruning sparsity level under explicit performance degradation constraints. The proposed formulation provides a useful perspective on sparsity selection beyond conventional empirical performance evaluation.
  2. Clear theoretical formulation and relatively comprehensive evaluation. The proposed framework is mathematically straightforward and provides a finite-sample marginal guarantee under the exchangeability assumption. The experiments cover multiple LLM families, pruning methods, and benchmarks. The repeated calibration experiments and sensitivity analysis also provide useful evidence for the statistical validity of the proposed approach.
Critical weaknesses
  1. Insufficient evidence for a fair and conclusive comparison with LTT. My main concern is the comparison between CRCSS and LTT in Figure 3. Although both methods are evaluated under the same risk level (\alpha), they provide different forms of statistical guarantees. CRCSS provides a marginal guarantee for sample-level degradation, whereas LTT controls population risk with an additional confidence parameter (\delta), which is set to 0.01 in the main experiments. Consequently, the higher sparsity levels selected by CRCSS may partly result from the different strengths of their statistical guarantees, rather than an inherent advantage of the proposed method. While Appendix C.3 includes additional results with (\delta=0.1), the comparison still does not fully establish the advantage of CRCSS under comparable risk-control conditions. In particular, the paper does not sufficiently clarify whether the two methods use exactly the same risk definition, nor does it provide a direct comparison of their empirical violation rates at the selected sparsity levels. Therefore, I find the claim of superior compression efficiency over LTT insufficiently supported by the current experiments.
  2. Limited evidence of advantages over alternative sparsity-selection strategies. The proposed method mainly combines a degradation-prefix construction with standard split conformal calibration. While this is a reasonable formulation, the methodological novelty appears relatively limited. Furthermore, the experimental evaluation focuses primarily on validating the statistical guarantees of CRCSS. A comparison with conventional validation-based sparsity selection would help quantify the practical trade-off between selected sparsity and reliability, and better establish the additional value of the proposed framework.
Questions for authors

Q1. Clarification of the comparison with LTT. Could the authors clarify how the risk definition and statistical guarantees are aligned between CRCSS and LTT in Figure 3? In particular, does LTT use the same prefix-wide violation criterion as CRCSS? Given that LTT introduces an additional confidence parameter (\delta), why is comparing the selected sparsity levels at the same (\alpha) sufficient to demonstrate the superiority of CRCSS?

Q2. Additional evidence for the compression efficiency comparison. Could the authors provide a more controlled comparison with LTT using identical candidate sparsity levels, calibration/test splits, and risk definitions? It would be particularly helpful to report both the selected sparsity levels and their corresponding empirical violation rates, along with the sensitivity of LTT to different values of (\delta). Such results would help determine whether the observed sparsity improvements arise from the proposed calibration strategy or from differences in statistical guarantees.

Q3. Comparison with conventional sparsity selection. Could the authors additionally compare CRCSS with a conventional validation-based sparsity selection strategy? Reporting the selected sparsity and empirical violation rate under the same experimental settings would help clarify the practical benefits of introducing conformal calibration. A convincing clarification or targeted experimental comparison addressing the first two questions would substantially strengthen my assessment of the paper.

4 comments
AuthorsOpenReview ↗

Manuscript Revisions

We are very grateful to the reviewers for their valuable feedback and for pointing out potential issues in the original manuscript.

We revised the manuscript in response to the reviewers’ comments. The changes are marked in blue.

  • Section 4.2 compares the endpoint EVRs of CRCSS and MAS and shows that nearly unchanged aggregate accuracy can mask degradation on individual examples.
  • Section 4.3 compares the selected sparsity and test accuracy of CRCSS and MAS under the same accuracy target.
  • Appendix A.2 derives an upper bound on mean prefix risk from the marginal guarantee.
  • Appendix C.2 compares the prefix EVR and mean prefix risk of CRCSS and MAS and reports the number of successful runs out of 100.
  • Appendix C.3 reports prefix EVR in the multiple-attempt setting.
  • Appendix C.4 reports the sparsity levels selected by CRCSS at different risk levels on CommonsenseQA with Wanda and SparseGPT.
  • Appendix C.5 clarifies the statistical guarantees of CRCSS and LTT and analyzes LTT’s sensitivity to δ\delta.
AuthorsOpenReview ↗

Response to question 1

  • 1-1 Figure 3 (Figure 4 in the revised manuscript) compares the sparsity each method selects under its own statistical guarantee when the marginal risk bounds are similar. Given the calibration set Dcal\mathcal D_{\mathrm{cal}}, let r(Dcal)r(\mathcal D_{\mathrm{cal}}) denote the probability that a new sample's performance degradation exceeds the tolerance at one or more candidate sparsity levels in the selected prefix. CRCSS directly bounds the marginal risk by EDcal[r(Dcal)]≤α\mathbb E_{\mathcal D_{\mathrm{cal}}}[r(\mathcal D_{\mathrm{cal}})]\leq\alpha. LTT requires Pr⁡Dcal(r(Dcal)>α)≤δ\Pr_{\mathcal D_{\mathrm{cal}}}(r(\mathcal D_{\mathrm{cal}})>\alpha)\leq\delta, which implies a marginal risk bound of α+(1−α)δ\alpha+(1-\alpha)\delta. At δ=0.01\delta=0.01, the two marginal bounds differ by less than one percentage point, and LTT additionally requires the conditional risk to be at most α\alpha with high probability over draws of the calibration set. Under these conditions, CRCSS selected higher sparsity in 40 of 60 configurations on CommonsenseQA and in all 36 configurations on TriviaQA.

  • 1-2 CRCSS and LTT use exactly the same prefix-wide violation criterion. For a given sample and candidate sparsity s^\hat{s}, the sample counts as a prefix violation at s^\hat{s} if it exceeds the degradation tolerance at any candidate sparsity from the dense model through s^\hat{s}, inclusive. CRCSS uses this event to determine each sample’s safe sparsity boundary and calibrates a quantile of these boundaries. LTT uses fixed-sequence testing to assess the population probability of the same event.

  • 1-3 We limit our claim to the observation that CRCSS tends to select higher sparsity when the marginal risk bounds are similar. Appendix C.5 of the revised manuscript further examines the sensitivity of CRCSS and LTT. As δ\delta increased from 0.01 to 0.10 and then to 0.30, the average gap in selected sparsity between CRCSS and LTT narrowed from 3.99 to 3.02 and then to 1.32 percentage points. This quantifies how much sparsity LTT gives up at higher confidence levels in this setting. The choice between the two guarantees depends on the deployment objective. If the objective is to control the probability of a prefix violation jointly over the calibration set and a future sample, CRCSS directly provides that marginal guarantee. If the objective is for the population prefix violation risk at the sparsity selected in a single calibration run to be at most α\alpha with probability at least 1−δ1-\delta over draws of the calibration set, LTT provides that guarantee.

AuthorsOpenReview ↗

Response to question 2

  • 2-1 In all experiments comparing CRCSS and LTT, both methods use the same candidate sparsity grid, calibration and test splits, and sample-level prefix violation criterion. For each split, the two methods independently select a sparsity level using the same calibration set. When the marginal risk bounds are similar, CRCSS tends to select higher sparsity.

  • 2-2 Appendix C.5 of the revised manuscript presents a sensitivity analysis of the comparison between CRCSS and LTT. As δ\delta increased from 0.01 to 0.10 and then to 0.30, the average gap in selected sparsity between CRCSS and LTT narrowed from 3.99 to 3.02 and then to 1.32 percentage points. These results directly quantify how much sparsity LTT sacrifices at higher confidence levels in this setting.

  • 2-3 When the marginal risk bounds are similar, LTT selects sparsity more conservatively, so we generally expect its test empirical violation rate (EVR) to stay within the target. The main text therefore focuses the comparison on compression efficiency. We report LTT’s test EVRs in the table below. We attribute CRCSS’s sparsity advantage to the different statistical guarantees.

ModelRisk threshold α\alphaCRCSS EVRLTT EVR (δ=0.10\delta=0.10)LTT EVR (δ=0.30\delta=0.30)
Vicuna-13B5%3.54%0.28%2.30%
Vicuna-13B10%7.38%4.64%6.10%
Vicuna-13B15%13.19%11.69%12.79%
Vicuna-13B20%18.09%14.72%17.26%
Vicuna-13B25%22.49%20.81%22.29%
Vicuna-13B30%26.62%22.89%24.96%
Qwen2.5-3B5%2.19%0.00%0.45%
Qwen2.5-3B10%5.21%5.21%5.21%
Qwen2.5-3B15%11.36%5.84%8.87%
Qwen2.5-3B20%15.29%14.28%14.44%
Qwen2.5-3B25%20.99%19.59%20.88%
Qwen2.5-3B30%25.93%21.36%23.37%
Llama-3.1-8B5%4.09%2.55%3.89%
Llama-3.1-8B10%7.43%4.28%6.02%
Llama-3.1-8B15%13.08%9.98%11.76%
Llama-3.1-8B20%18.16%15.13%16.65%
Llama-3.1-8B25%22.16%19.64%20.50%
Llama-3.1-8B30%25.55%25.10%25.23%
AuthorsOpenReview ↗

Response to question 3

  • 3_1 We revised the manuscript to include a comparison with MAS, a conventional validation-based method that selects sparsity using average accuracy. To align the comparison, we use the accuracy lower bound derived from CRCSS, ACC(0)−α\mathrm{ACC}(0)-\alpha, as the selection threshold for MAS. The comparison is presented in Sections 4.2–4.3 and Appendix C.2 of the revised manuscript. Tables 1, 2, 5, and 6 report the endpoint empirical violation rate (EVR), test accuracy, prefix EVR, and mean prefix risk, respectively, along with the number of runs out of 100 in which each metric met its corresponding criterion. Across all four metrics, CRCSS met the evaluation criteria substantially more consistently than MAS, showing that CRCSS provides more reliable control of the risk of sample-level performance degradation when selecting a pruning sparsity.
Reviewer_ufiWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CRCSS, a conformal calibration framework for choosing pruning sparsity under a performance-degradation constraint. It converts non-monotonic degradation trajectories into running maxima, then calibrates sample-level safe boundaries to obtain a finite-sample marginal guarantee. Experiments cover five LLMs, three pruning methods, and several evaluation settings.

Critical strengths
  1. The prefix construction addresses a real issue: a higher sparsity can look acceptable even when an earlier point is unsafe. The resulting guarantee covers the whole candidate prefix, and the ablation shows that this construction matters for risk control.
  2. The framework is reasonably general and does not require changing the pruning algorithm. Results across different models, pruning methods, and degradation metrics support its usefulness as a calibration layer on top of existing methods.
Critical weaknesses
  1. Once the running maximum is introduced, the statistical argument is a fairly direct application of split conformal calibration. This makes the practical advantage particularly important, but the LTT comparison uses different statistical guarantees. Appendix C.3 acknowledges this, so higher selected sparsity alone does not establish better efficiency under equivalent protection.
  2. The practical meaning of “safe” needs more discussion. Capped answer-attempt counts can obscure differences on difficult questions, and reporting mean violation rates across calibration splits leaves the variability of a single deployment decision unclear.
Questions for authors

In the LTT comparison, how much of the sparsity advantage comes from the different guarantees rather than the calibration procedure itself? Could you separate these effects, or state more narrowly what the comparison establishes?

How are questions with no correct answer within ten attempts scored, and how common are they for the dense and pruned models? Could censoring at this cap materially affect the selected sparsity or make degradation appear smaller?

Across the 100 calibration splits, how often does the test violation rate exceed the target, and by how much? I understand that the theorem is marginal; this would help readers understand the variability they might encounter when deploying one calibrated model.

5 comments
AuthorsOpenReview ↗

Manuscript Revisions

We are very grateful to the reviewers for their valuable feedback and for pointing out potential issues in the original manuscript.

We revised the manuscript in response to the reviewers’ comments. The changes are marked in blue.

  • Section 4.2 compares the endpoint EVRs of CRCSS and MAS and shows that nearly unchanged aggregate accuracy can mask degradation on individual examples.
  • Section 4.3 compares the selected sparsity and test accuracy of CRCSS and MAS under the same accuracy target.
  • Appendix A.2 derives an upper bound on mean prefix risk from the marginal guarantee.
  • Appendix C.2 compares the prefix EVR and mean prefix risk of CRCSS and MAS and reports the number of successful runs out of 100.
  • Appendix C.3 reports prefix EVR in the multiple-attempt setting.
  • Appendix C.4 reports the sparsity levels selected by CRCSS at different risk levels on CommonsenseQA with Wanda and SparseGPT.
  • Appendix C.5 clarifies the statistical guarantees of CRCSS and LTT and analyzes LTT’s sensitivity to δ\delta.
AuthorsOpenReview ↗

Response to question 1 and weakness 1

  • 1-1 We attribute CRCSS’s sparsity advantage to the different statistical guarantees. Our claim in the main text is limited to how much sparsity each method selects under its own guarantee when their marginal risk bounds are closely aligned. Given the calibration set Dcal\mathcal D_{\mathrm{cal}}, let r(Dcal)r(\mathcal D_{\mathrm{cal}}) denote the probability that a new example's performance degradation exceeds the tolerance at one or more sparsity levels in the selected prefix. CRCSS directly bounds the marginal risk by EDcal[r(Dcal)]≤α\mathbb E_{\mathcal D_{\mathrm{cal}}}[r(\mathcal D_{\mathrm{cal}})]\le\alpha. LTT requires Pr⁡Dcal(r(Dcal)>α)≤δ\Pr_{\mathcal D_{\mathrm{cal}}}(r(\mathcal D_{\mathrm{cal}}) > \alpha)\le\delta, which implies a marginal risk bound of α+(1−α)δ\alpha+(1-\alpha)\delta. At δ=0.01\delta=0.01, the two marginal bounds differ by less than one percentage point, while LTT additionally requires the conditional risk to be at most α\alpha with high probability over calibration sets. Under these conditions, CRCSS selected higher sparsity in 40 of 60 configurations on CommonsenseQA and in all 36 configurations on TriviaQA.

  • 1-2 The comparison establishes that CRCSS can select higher sparsity when the marginal risk bounds are closely aligned. To examine how LTT’s confidence requirement affects this gap, we added a sensitivity analysis on TriviaQA with Wanda pruning in Appendix C.5. As δ\delta increased from 0.01 to 0.10 and then to 0.30, the mean sparsity gap between CRCSS and LTT narrowed from 3.99 to 3.02 and then to 1.32 percentage points. This quantifies how much sparsity LTT gives up at higher confidence in this setting. The choice between the two guarantees depends on the deployment objective. If the goal is to control the probability of a prefix violation jointly over the calibration set and a future example, CRCSS directly provides that marginal guarantee. If the goal is to ensure that the population prefix violation risk at the sparsity selected in a single calibration run is at most α\alpha with probability at least 1−δ1-\delta over the draw of the calibration set, LTT provides that guarantee.

AuthorsOpenReview ↗

Response to question 2

  • 2-1 We cap the number of attempts at 10 and set the degradation tolerance to ϵ=1\epsilon=1. At each sparsity level, we record the index of the first correct attempt. If none of the ten attempts is correct, we record 11 and flag the question as unsolved at that sparsity. Before splitting the data into calibration and test sets, we exclude questions that the dense model did not solve within the same budget. Risk control therefore applies to questions solved by the dense model, and a pruned model’s failure to solve any of these questions within ten attempts counts as a violation. The table reports the percentage of all CommonsenseQA questions that remained unsolved after ten attempts at each sparsity level.
SparsityQwen2.5-3BLlama-3.1-8B
0%21.11%15.59%
10%20.85%15.50%
20%21.17%16.27%
30%22.55%15.44%
40%23.71%15.99%
50%27.74%10.13%
60%33.59%13.81%
70%80.64%63.07%
  • 2-2 Including questions that the dense model fails to solve within ten attempts may materially affect the selected sparsity. However, it will not cause the selected sparsity to be smaller; on the contrary, it will be larger. We keep the ten-attempt cap and $\epsilon=1$, and repeated calibration on the full CommonsenseQA question set for Vicuna-13B, Qwen2.5-3B, and Llama-3.1-8B. The results are shown in the table below: across 18 model–risk configurations, full-question-set calibration produced a higher average selected sparsity in 16 configurations and the same average in the other two configurations. Intuitively, performing risk control on correct questions is a stricter requirement than performing risk control on the overall data, and the current experiments also demonstrate this. | Model | Question set | α = 0.05 | α = 0.10 | α = 0.15 | α = 0.20 | α = 0.25 | α = 0.30 | |---|---|---|---|---|---|---|---| | Vicuna-13B | Dense-solved questions | 0%: 100 | 0%: 100 | 10%: 22, 20%: 78 | 25%: 100 | 30%: 1, 35%: 99 | 40%: 84, 45%: 16 | | Vicuna-13B | All questions | 0%: 100 | 10%: 100 | 20%: 10, 25%: 90 | 35%: 100 | 40%: 1, 45%: 99 | 50%: 55, 55%: 45 | | Qwen2.5-3B | Dense-solved questions | 20%: 100 | 25%: 10, 30%: 90 | 35%: 99, 40%: 1 | 40%: 100 | 45%: 100 | 45%: 100 | | Qwen2.5-3B | All questions | 20%: 91, 25%: 9 | 30%: 14, 35%: 86 | 40%: 100 | 45%: 100 | 45%: 100 | 50%: 100 | | Llama-3.1-8B | Dense-solved questions | 0%: 36, 10%: 64 | 20%: 100 | 30%: 99, 35%: 1 | 40%: 100 | 45%: 100 | 45%: 15, 50%: 85 | | Llama-3.1-8B | All questions | 10%: 100 | 20%: 3, 25%: 97 | 35%: 99, 40%: 1 | 40%: 3, 45%: 97 | 45%: 47, 50%: 53 | 50%: 100 |

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.