acceptodds
Under review as a conference paper at ICLR 2027

Split Conformal Prediction with Tunable Prediction Set Size

Abstract

Conformal Prediction (CP) is a model-agnostic framework for constructing prediction sets with finite-sample coverage guarantees and has become a key tool for reliable decision-making in high-risk applications. However, a limitation of CP is that the size of the prediction set is implicitly determined by the target coverage level, which can result in sets that are too large to be actionable. This is particularly problematic in applications such as medical diagnosis, where there are many possible conditions and decisions must be made under time constraints. To address this limitation, recent work studies CP procedures in the Tunable Prediction Set Size (TPSS) setting, wherein the user specifies a maximum allowable prediction set size, and the goal is to provide valid coverage guarantees under this constraint. While Split CP (sCP) is the most widely used variant of CP, enforcing such size control within this framework is challenging due to its quantile-based calibration procedure. As a result, existing approaches for TPSS rely on the more flexible Conformal e-Prediction (eCP) framework, which enables data-dependent size control but inherits limitations such as weaker structural properties and less accurate coverage approximation. Motivated by this gap, we develop Tunable-sCP (TsCP), a method to address the TPSS problem within the sCP framework, enabling explicit control over prediction set size while preserving the key structural properties of sCP. We further show that our leave-one-out construction yields a substantially more accurate coverage estimate than eCP for the same calibration size, reducing conservativeness and making it more reliable in practice. Empirically, we demonstrate on synthetic and real-world datasets that TsCP consistently improves the empirical coverage–efficiency trade-off compared to prior approaches.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Conformal Prediction, Prediction Set Size, Adaptive Calibration, Leave-One-Out Estimation, Coverage Guarantees, Uncertainty Quantification, Post-hoc Tuning, Distribution-Free Inference.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aopgOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the Tunable Prediction Set Size (TPSS) setting and proposes Tunable-sCP (TsCP) to obtain size-controlled prediction sets within split conformal prediction (sCP). The main idea is to split calibration data so that one part selects a data-dependent miscoverage level and the other performs quantile calibration, together with leave-one-out coverage estimation and related theory.

Critical strengths
  1. The paper gives a reasonably clear diagnosis of why directly choosing a data-dependent level in sCP can break the classical rank argument, and proposes a split-calibration remedy.
  2. It attempts to discuss coverage estimation and size control in one framework, with some theory-experiment alignment.
Critical weaknesses
  1. There appears to be a gap between the size constraint in the problem definition and what is actually proved/guaranteed: the theory looks more asymptotic/soft, and in that respect seems not fundamentally different from the eCP route. If the central pitch is “solving TPSS within sCP,” this gap weakens the case.
  2. The main-text experiments are incomplete: comparisons are narrow and focus mainly on one recent baseline; classification results, additional settings, and ablations are largely deferred to the appendix; more standard size-control or truncation controls are also missing.
  3. The overall presentation could be improved and polished a lot: the abstract mainly introduces the relavant background, but not their own work TsCP; the main text is not "self-contained" and repeatedly refer to appendix; three figures in the main text are hard to read in printed paper.
Questions for authors
  1. Why were Conformal Risk Control, standard sCP followed by truncation, and common set-optimization methods not compared? Would TsCP’s advantages still hold after these controls?
  2. How should the size constraint be accurately described to readers? Are theoretically suggested δ\delta ranges consistent with empirically best δ\delta, and how does ϵ\epsilon affect the tightness of the guarantees?
  3. In practice, which metrics should be reported and how should they be interpreted when budget violations occur? What is the fallback behavior and frequency when the budget is infeasible?
5 comments
AuthorsOpenReview ↗

Reply [1/4]

Thank you for your thoughtful review and constructive feedback. We have revised our manuscript in response to your suggestions, with all changes highlighted in blue. Below, we address each of your concerns in detail.


W2. The main-text experiments are incomplete: comparisons are narrow and focus mainly on one recent baseline; classification results, additional settings, and ablations are largely deferred to the appendix; more standard size-control or truncation controls are also missing.

Q1. How should the size constraint be accurately described to readers? Are theoretically suggested δ\delta ranges consistent with empirically best δ\delta, and how does ϵ\epsilon affect the tightness of the guarantees?

Answer: We thank the reviewer for raising this important concern. We would like to clarify that our experimental comparison focuses on Backward Conformal Prediction (BCP) [1] because, to the best of our knowledge, it is the most directly comparable existing method specifically designed to address the TPSS problem. In particular, TPSS requires simultaneously (i) controlling the prediction set size according to a user-specified budget S(X)S(X) and (ii) providing a corresponding marginal coverage guarantee. While several related methods address prediction set efficiency or risk control, they do not directly satisfy both requirements under their standard formulations.

  1. Conformal Risk Control (CRC). CRC [2] provides finite-sample control of the expected value of a monotone loss function. In particular, choosing the miscoverage indicator as the loss recovers a marginal coverage guarantee. However, this does not simultaneously guarantee that the prediction set satisfies a prescribed instance-wise size budget ∣C(X)∣≤S(X)|\mathcal{C}(X)|\leq S(X). Similarly, controlling a size-related expected loss does not imply an instance-wise hard size constraint. Therefore, CRC does not directly solve TPSS without additional methodological modifications.

  2. Standard sCP followed by truncation. A natural alternative is to first construct a standard conformal prediction set and subsequently truncate it to satisfy the prescribed size budget. While this approach guarantees size control by construction, truncation generally invalidates the original coverage guarantee, since removing candidate labels may also remove the true label. Thus, this approach satisfies the first TPSS requirement but does not necessarily retain the second.

  3. Standard set-optimization methods. Existing approaches to optimizing conformal prediction sets, such as [3], primarily seek to minimize prediction set size or volume while maintaining a coverage guarantee. However, optimizing prediction set efficiency is fundamentally different from enforcing a user-specified instance-wise maximum size. Consequently, these methods do not directly address the TPSS formulation considered in our work. Furthermore, while regression comparisons are presented in the main text due to space constraints, our experimental evaluation is not limited to these settings. Classification comparisons are provided in Appendix D.2, while additional ablations on the slack parameter, size constraint, budget function, underlying model, and computational efficiency are presented in Appendices D.3–D.7. Nevertheless, we acknowledge that further discussion of the aforementioned alternative approaches, their relationship to TPSS, and their potential adaptation to this setting would strengthen the paper. We will seek to incorporate such a discussion in the revised manuscript, alongside a clearer presentation of our existing experimental results.

We have added comparisons with Conformal Risk Control, post-hoc truncation, and prediction-set optimization at the end of Appendix C.1.2.

References

[1] Gauthier et al. Backward Conformal Prediction. NeurIPS 2025.

[2] Angelopoulos et al. Conformal Risk Control. ICLR 2024.

[3] Gao et al. Volume Optimality in Conformal Prediction with Structured Prediction Sets. ICML 2025.

AuthorsOpenReview ↗

Reply [2/4]

Q1. There appears to be a gap between the size constraint in the problem definition and what is actually proved/guaranteed: the theory looks more asymptotic/soft, and in that respect seems not fundamentally different from the eCP route. If the central pitch is "solving TPSS within sCP," this gap weakens the case.

Answer: We thank the reviewer for raising this important concern. We acknowledge that, unlike Backward Conformal Prediction (BCP) [1], which satisfies the prescribed size constraint by construction, our theoretical analysis guarantees only an asymptotic version of the size constraint, as formally stated in Equation (4) and established in Theorem 4.4. Nevertheless, we would like to emphasize that TsCP achieves near-hard size control in practice, even with relatively small calibration datasets, as demonstrated by our empirical evaluation in Appendix D.4.

Specifically, we evaluate the proportion of test points satisfying the prescribed hard size constraint, i.e., ∣Cδ(X)∣≤S(X)|\mathcal{C}_\delta(X)|\leq S(X), across six regression and classification datasets and varying calibration sizes. The results, reported in Table 1 (Appendix D.4), are reproduced below for the reviewer's convenience.

Table 1: Hard-constraint satisfaction rate (%) across different total calibration set sizes.

Dataset1,0002,0003,0004,0005,0006,000
Synthetic Regression99.199.799.299.1100100
California Housing95.710095.496.197.8100
Superconductivity10099.5100100100100
Synthetic Classification10096.610010099.699.6
MNIST94.999.910098.895.9100
Covertype99.910010099.910099.9

As shown, even with only 1,000 calibration samples, TsCP satisfies the hard size constraint for at least 94.9% of test points across all six datasets, with four datasets exceeding 99%. At a calibration size of 6,000, four datasets achieve 100% satisfaction, while the remaining two achieve 99.6% and 99.9%, respectively. These results demonstrate that, although our theoretical guarantee is asymptotic, TsCP achieves behavior very close to hard size control in finite-sample settings.

Furthermore, we would like to clarify that the distinction between TsCP and the eCP-based approach extends beyond the type of size constraint guaranteed. As discussed in Appendix C.1, BCP leverages the flexibility of e-values to enforce hard size constraints, whereas TsCP addresses TPSS within the quantile-based sCP framework, preserving its structural properties. This difference enables us to establish an exact marginal coverage characterization with an explicit correction term (Theorem 4.2), together with consistency and O(n−1)O(n^{-1}) variance guarantees for the corresponding LOO estimator (Theorems 5.2–5.3). As demonstrated in Figures 3, 8, and 9, these structural distinctions also translate into empirical advantages, including smaller prediction sets at comparable coverage and substantially more accurate LOO coverage estimation. Therefore, our contribution is not solely to enforce prediction set size constraints, but to establish a fundamentally different sCP-based approach to TPSS with its own theoretical and empirical advantages.

Nevertheless, we fully acknowledge that establishing a finite-sample hard size guarantee remains an important theoretical advantage of BCP. While our results demonstrate that TsCP closely approximates this behavior empirically, we recognize that this does not replace a formal hard-constraint guarantee. We will clarify the distinction between the original TPSS formulation and our asymptotic relaxation in the revised manuscript. Extending TsCP to provide finite-sample hard size guarantees while preserving the structural properties of sCP is an important direction for future research.

[1] Gauthier et al. Backward Conformal Prediction. NeurIPS 2025.

AuthorsOpenReview ↗

Reply [3/4]

Q2. How should the size constraint be accurately described to readers? Are theoretically suggested ranges consistent with empirically best, and how does affect the tightness of the guarantees?

Answer: We thank the reviewer for this important question. We would like to clarify the precise interpretation of our size constraint, the relationship between the theoretical and empirical choices of δ\delta, and the role of the truncation parameter ε\varepsilon.

  1. Interpretation of the size constraint. As formally stated in Equation (4), our method guarantees a soft size constraint, rather than a finite-sample hard constraint. Specifically, Theorem 4.4 establishes that P(∣Cδ(X0)∣≤S(X0))→1P(|\mathcal{C}_\delta(X_0)|\leq S(X_0))\to 1 as n→∞n\to\infty, under Assumptions 4.1 and 4.3, continuity of the population quantile function, and feasibility of the adaptive-level selection. We acknowledge that this distinction should be emphasized more clearly and will clarify it in the revised manuscript.

  2. Theoretical and empirical choices of δ\delta. As discussed in Section 4.2 and Appendix D.3, the slack parameter δ\delta controls the trade-off between coverage and size-constraint satisfaction. In particular, Theorem 4.4 provides a sufficient theoretical range L≤δ≤ζL\leq\delta\leq\zeta, where LL is the constant in Assumption 4.3 controlling local changes in prediction set size. This choice ensures that the slack is sufficient to absorb the discrepancy between the two independently estimated quantiles. On the other hand, smaller values of δ\delta generally favor higher coverage, while larger values favor stricter size control. Consequently, the empirically preferred δ\delta depends on the desired trade-off and need not coincide with the theoretically sufficient range. In particular, the asymptotic guarantee of Theorem 4.4 should not be interpreted as applying automatically to choices δ<L\delta<L. This behavior is further supported by our ablation studies in Appendix D.3 (Figures 10 and 11), where we evaluate different values of δ\delta across six regression and classification datasets. Consistently, we observe that increasing δ\delta decreases empirical coverage and its theoretical counterpart, while improving the probability of satisfying the prescribed size constraint. These results support the interpretation of δ\delta as a tunable hyperparameter governing the practical coverage–size trade-off.

  3. Role of the truncation parameter ε\varepsilon. As discussed in detail in Appendix C.2, the primary purpose of ε\varepsilon is to ensure theoretical stability by restricting the adaptive miscoverage level to [ε,1−ε][\varepsilon,1-\varepsilon]. In particular, the proof of Theorem 4.4 requires uniform convergence of the empirical quantiles over this compact interval. Without truncation, extreme order statistics near the endpoints may become unstable, preventing the same uniform convergence argument. Importantly, ε\varepsilon is not introduced as an additional approximation error or coverage slack. Its influence on the coverage characterization in Theorem 4.2 is indirect, through the selected adaptive level and its correction term. As explained in Appendix C.2, empirical conformal quantiles have an effective resolution of order 1/n1/n. Therefore, choosing ε≪1/n\varepsilon\ll 1/n, such as ε=10−10\varepsilon=10^{-10} in our experiments, has negligible practical impact on the resulting prediction sets. Thus, ε\varepsilon primarily serves a theoretical purpose, rather than introducing a meaningful additional source of conservativeness in our experimental setting.

Reviewer_FAReOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Tunable-sCP (TsCP) for constructing prediction sets under a user-specified, potentially input-dependent size budget. One calibration split selects an adaptive quantile level using a margin-adjusted budget, and an independent split determines the final threshold. The paper establishes an exact marginal coverage identity, asymptotic size compliance, and consistency and variance bounds for LOO coverage estimation under additional assumptions. Experiments on six regression and classification datasets compare coverage–size trade-offs and coverage estimates with eCP-based methods.

Critical strengths

The independent selection and calibration splits make it possible to express achieved coverage through a binomial calculation and analyze the discrepancy between the two empirical quantiles. The LOO analysis accounts for dependence arising from overlapping samples and shared calibration data. This construction and its analysis constitute a useful methodological contribution in settings satisfying the assumptions.

Critical weaknesses
  1. The practical guarantee claims exceed the established theoretical scope. For the deterministic classification score in Eq. (243), Assumptions 4.1 and 5.1 are incompatible: no ties makes the marginal score law nonatomic, whereas the conditional law given X=xX=x has finite support. Thus Theorems 5.2–5.3 do not justify the reported classification coverage estimator. Theorem 4.4 also requires L≤δ≤inf⁡xS(x)L\le\delta\le\inf_x S(x); the automatic choice L=∣Y∣L=|\mathcal Y| only covers trivial budgets, and a useful smaller LL is unverified. Finally, Theorem 4.2 provides a coverage identity with an unknown correction. Consistency of its LOO estimate does not make that estimate a finite-sample coverage lower bound.

  2. The claimed advance is not isolated from changes in statistical targets and size constraints. BCP already provides the size-first formulation and LOO convergence/variance results; its arXiv extension also derives a non-Taylor marginal bound under additional assumptions. The incremental contribution is therefore the split-quantile construction, whose advantage remains insufficiently established against the omitted ST-BCP method and direct size-constrained prediction with independent coverage validation.

  3. The central comparisons do not resolve this gap. Task (b) compares TsCP’s achieved-coverage estimate with a different eCP population quantity and measures error after averaging 50 trials. It therefore does not isolate superior estimation accuracy or stronger guarantees. The efficiency comparison likewise permits different levels of budget compliance: at the main classification setting δ=1\delta=1, Fig. 11 shows approximately 24% and 28% violations for Synthetic Classification and MNIST, while Table 1’s high compliance uses δ=2\delta=2. Consequently, the results do not establish improved efficiency and reliable size control at the same operating point, or superiority under a common statistical target and size requirement.

Questions for authors
  1. Do the classification experiments use fresh score randomization beyond Eq. (243)? If so, what construction verifies Assumption 5.1 after conditioning on the fitted model? Separately, what uniform local multiplicity bound LL justifies applying Theorem 4.4 with the tested classification budgets and slack values?
  2. How are the unknown score CDF and correction Δδ,n\Delta_{\delta,n} evaluated for the “theoretical” curves in Eqs. (233) and (246) on real datasets? If the correction is computed as observed miscoverage minus the adaptive level, the comparison becomes two estimates of coverage rather than an independent evaluation of the binomial expression. Which calculation was implemented, and what does that validation establish?
  3. Which sample supplies the calibration-scale quantiles defining S(x)S(x) in Eqs. (264)–(265)? If CcC_c contributes, how is the independence argument in Appendix B.1 retained? If only CaC_a contributes, how is budget estimation handled in each LOO calculation and in the stability analysis?
  4. Which common (c,δ,2n)(c,\delta,2n) configuration supports both the claimed efficiency advantage and near-hard budget compliance? Please also reconcile the approximately 0.91 compliance in Fig. 4 with 100% in Table 1 for Superconductivity at c=60,δ=8,2n=6000c=60,\delta=8,2n=6000. Do these results average over the same calibration randomness, or estimate different conditional and marginal quantities?
11 comments
AuthorsOpenReview ↗

Reply [1/7]

Thank you for your thoughtful review and constructive feedback. We have revised our manuscript in response to your suggestions, with all changes highlighted in blue. Below, we address each of your concerns in detail.


W2. The claimed advance is not isolated from changes in statistical targets and size constraints. BCP already provides the size-first formulation and LOO convergence/variance results; its arXiv extension also derives a non-Taylor marginal bound under additional assumptions. The incremental contribution is therefore the split-quantile construction, whose advantage remains insufficiently established against the omitted ST-BCP method and direct size-constrained prediction with independent coverage validation.

Answer: We thank the reviewer for this insightful observation. We acknowledge that BCP [1] has already established the size-first formulation of TPSS and developed LOO estimators with convergence and variance guarantees. We also recognize that its extended arXiv version derives a non-Taylor marginal coverage bound under additional assumptions. Nevertheless, we would like to emphasize that our contribution is not the introduction of TPSS or LOO estimation, but rather establishing a fundamentally different approach to these problems within the sCP framework, with distinct theoretical properties and practical implications.

  1. Fundamental methodological and theoretical distinctions.* Unlike BCP, which relies on the flexibility of e-values and post-hoc validity, TsCP preserves the quantile-based structure of sCP by introducing two independent calibration subsets: Ca\mathcal{C}^a for selecting the adaptive miscoverage level and Cc\mathcal{C}^c for constructing the final prediction set. This construction enables us to derive an exact marginal coverage identity with an explicit correction term (Theorem 4.2), rather than relying on e-value inequalities. Furthermore, exploiting the rank locality of sCP, we establish consistency and O(n−1)O(n^{-1}) variance guarantees for our corrected LOO estimator (Theorems 5.2–5.3). These results rely on a different set of assumptions and proof techniques from those of BCP, as discussed in Appendices C.1.2 and C.3.

  2. Empirical advantages under comparable settings. Our experimental results demonstrate that these structural differences have practical implications beyond the construction itself. Specifically, Figures 3 and 8 show that TsCP produces smaller prediction sets at comparable empirical coverage across all six regression and classification datasets. Furthermore, Figures 3 and 9 demonstrate that our corrected LOO estimator tracks empirical coverage substantially more closely than the original eCP-based approach.

  3. Comparison with ST-BCP and independent coverage validation. We appreciate the reviewer for highlighting ST-BCP [2], which improves BCP through data-dependent transformations of nonconformity scores to obtain tighter coverage estimates. However, we would like to emphasize that ST-BCP remains within the eCP framework, retaining its e-value-based construction and associated structural properties. In particular, ST-BCP focuses on improving the coverage estimation of BCP through score transformations, whereas our work takes a fundamentally different direction by addressing TPSS within the classical sCP framework. By preserving the quantile-based structure and rank locality of sCP, TsCP enables an exact marginal coverage characterization (Theorem 4.2) and a distinct LOO analysis with consistency and variance guarantees (Theorems 5.2–5.3). Thus, while both methods aim to improve coverage estimation under size constraints, they differ fundamentally in their underlying constructions, theoretical analyses, and methodological objectives.

Regarding direct size-constrained prediction followed by independent coverage validation, we acknowledge that this represents a straightforward alternative. However, enforcing the size constraint by construction and subsequently evaluating empirical coverage on an independent validation set does not, by itself, establish a finite-sample conformal coverage guarantee. While such an approach satisfies the prescribed size constraint, empirical validation alone does not ensure the required coverage for future test points. Consequently, without an additional procedure establishing coverage validity, this approach does not satisfy both requirements of the TPSS problem, which demands simultaneous size control and a corresponding coverage guarantee.

[1] Gauthier et al. Backward Conformal Prediction. NeurIPS 2025.

[2] Liu et al. Improving Backward Conformal Prediction via Non-Conformity Score Transformation. ICML 2026.

AuthorsOpenReview ↗

Reply [2/7]

W1. The practical guarantee claims exceed the established theoretical scope. For the deterministic classification score in Eq. (243), Assumptions 4.1 and 5.1 are incompatible: no ties makes the marginal score law nonatomic, whereas the conditional law given X=xX=x has finite support. Thus Theorems 5.2–5.3 do not justify the reported classification coverage estimator. Theorem 4.4 also requires L≤δ<inf⁡xS(x)L\leq\delta<\inf_x S(x); the automatic choice L=∣Y∣L=|\mathcal Y| only covers trivial budgets, and a useful smaller LL is unverified. Finally, Theorem 4.2 provides a coverage identity with an unknown correction. Consistency of its LOO estimate does not make that estimate a finite-sample coverage lower bound.

Answer: We thank the reviewer for this careful observation regarding the scope of our theoretical guarantees. We acknowledge that the assumptions introduced in Sections 4 and 5 are sufficient conditions for establishing our theoretical results and are not necessarily satisfied in every practical setting. In particular, verifying such distributional and regularity assumptions can be challenging for real-world datasets. Nevertheless, we would like to emphasize that our empirical evaluation also investigates the behavior of TsCP in settings where these assumptions may not hold, providing evidence of its practical effectiveness beyond the settings directly covered by our theoretical analysis.

  1. Assumptions 4.1 and 5.1 in classification. We acknowledge the reviewer's observation that, for deterministic classification scores with a finite label space, the conditional score distribution given X=xX=x has finite support. When the marginal score distribution is nonatomic, as required by the no-ties assumption, this creates an incompatibility with the absolute continuity condition in Assumption 5.1. Consequently, Theorems 5.2–5.3 do not directly establish the LOO convergence and variance guarantees for this particular classification setting. However, as demonstrated in Appendix D.2 (Figure 7), our empirical results show that the LOO estimator closely tracks the independently evaluated coverage, with decreasing variability as the calibration size increases. These results suggest that the estimator can exhibit favorable practical behavior even outside the assumptions used in our theoretical analysis, although they do not constitute a formal extension of the theorems.

  2. Assumption 4.3 and the size-control guarantee. We also acknowledge that the sufficient condition L≤δ≤ζL\leq\delta\leq\zeta in Theorem 4.4 may be restrictive, particularly in classification settings where the coarse choice L=∣Y∣L=|\mathcal{Y}| can exceed the prescribed size budget. As discussed in Section 4.2, smaller values of LL are possible under additional conditions on the local multiplicity or separation of candidate-label scores, although these conditions are not established for every dataset considered. Nevertheless, our experiments in Appendix D.2 and D.4 demonstrate strong empirical size control without requiring verification of these sufficient conditions. In particular, Table 1 shows that TsCP satisfies the prescribed hard size constraint for at least 94.9% of test points across all six datasets with only 1,000 calibration samples, increasing to at least 99.6% with 6,000 samples.

  3. Interpretation of the coverage characterization and LOO estimator. We agree that Theorem 4.2 establishes an exact marginal coverage identity involving an unknown correction term, rather than a directly computable finite-sample coverage lower bound. The purpose of our LOO construction is to approximate this coverage quantity, with consistency and variance guarantees under the assumptions of Theorems 5.2–5.3. Therefore, the resulting estimate should be interpreted as a coverage estimator rather than a certified finite-sample lower confidence bound. Nevertheless, Figures 2, 3, 7, and 9 demonstrate that the proposed estimator closely approximates empirical coverage and exhibits favorable stability across the evaluated regression and classification settings.

We have clarified the interpretation and scope of the coverage and size-control results in Appendix D.1.1, and the limitations of the LOO and size-control theory for the classification experiments in Appendix D.2.

AuthorsOpenReview ↗

Reply [3/7]

W3. The central comparisons do not resolve this gap. Task (b) compares TsCP's achieved-coverage estimate with a different eCP population quantity and measures error after averaging 50 trials. It therefore does not isolate superior estimation accuracy or stronger guarantees. The efficiency comparison likewise permits different levels of budget compliance: at the main classification setting δ=1\delta=1, Fig. 11 shows approximately 24% and 28% violations for Synthetic Classification and MNIST, while Table 1's high compliance uses δ=2\delta=2. Consequently, the results do not establish improved efficiency and reliable size control at the same operating point, or superiority under a common statistical target and size requirement.

Answer: We thank the reviewer for this careful observation. We would like to clarify the objectives of our experiments and the interpretation of their results.

  1. Coverage estimation in Task (b). As described in Appendices D.1.2 and D.2.2, Task (b) compares the LOO-based coverage estimate of each method against its corresponding empirical test coverage, under the same total calibration data budget. Specifically, TsCP uses the corrected estimator 1−α^δLOO−Δ^δ,nLOO1-\hat{\alpha}_\delta^{\mathrm{LOO}} -\hat{\Delta} _{\delta,n}^{\mathrm{LOO}}, whereas eCP uses 1−α^LOO1-\hat{\alpha}^{\mathrm{LOO}}. We acknowledge that these quantities have different theoretical interpretations and that measuring discrepancies after averaging over 50 trials does not fully characterize trial-level estimation accuracy. Nevertheless, Figures 2, 5, and 7 additionally examine the distribution of TsCP's LOO estimator across repeated trials, demonstrating close agreement with independent-test coverage and decreasing variability as the calibration size increases. Thus, our results provide empirical evidence of accurate and stable coverage estimation, although they do not establish a universally stronger coverage guarantee than eCP.

  2. Prediction efficiency and size-control violations. We acknowledge the reviewer's observation that Figure 11 indicates approximately 24% and 28% budget violations for Synthetic Classification and MNIST at δ=1\delta=1, respectively. As discussed in Appendix D.3, these observations reflect the coverage–size trade-off governed by δ\delta: smaller values favor higher coverage, whereas larger values provide a greater margin for satisfying the prescribed size constraint. Importantly, these finite-sample violations do not contradict Theorem 4.4, which establishes asymptotic size control only under its stated sufficient conditions. Furthermore, Table 1 demonstrates that TsCP can achieve near-hard size control at alternative operating points. In particular, with δ=2\delta=2 and a total calibration size of 2,000, the hard-constraint satisfaction rates reach 96.6% for Synthetic Classification and 99.9% for MNIST. At 6,000 calibration samples, these rates reach 99.6% and 100%, respectively. These results are consistent with the trade-off discussed in Appendix D.3, although the experimental protocols differ, so the improvement cannot be attributed solely to changing δ\delta.

  3. Interpretation of the comparisons. Task (a) in Figures 3 and 8 demonstrates smaller average prediction sets at comparable empirical coverage, while Table 1 evaluates hard-constraint satisfaction under different configurations. We acknowledge that these results do not establish simultaneous superiority in coverage, efficiency, and size control at a single matched operating point. Nevertheless, they demonstrate complementary practical advantages of TsCP, particularly its coverage-estimation accuracy and adjustable coverage–size trade-off.

Reviewer_NwH1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Tunable Split Conformal Prediction (TsCP), which adaptively selects the miscoverage level to control prediction set size while retaining the split conformal framework. The method uses two calibration subsets: one to determine the adaptive miscoverage level and the other to construct the final prediction set. The paper also develops leave-one-out estimators for analyzing marginal coverage.

Critical strengths
  1. The paper addresses an important problem in conformal prediction, namely controlling prediction set size while maintaining coverage.
  2. The proposed method provides a relatively simple extension of split conformal prediction, with a clear calibration procedure.
  3. The paper includes theoretical analysis and experiments on multiple regression and classification tasks.
Critical weaknesses
  1. The novelty is limited. BCP (NeurIPS 2025) has already studied adaptive miscoverage selection for prediction set size control, together with leave-one-out coverage estimation. The authors should better clarify what fundamentally distinguishes TsCP from BCP.

  2. TsCP splits the calibration data into two subsets, potentially reducing calibration efficiency. More importantly, its prediction set size guarantee is asymptotic, while BCP constructs sets satisfying the prescribed size constraint by design.

  3. The experimental evaluation does not sufficiently demonstrate the practical advantage of TsCP over BCP. A comparison under the same calibration data budget is particularly important.

Questions for authors
  1. What is the fundamental contribution of TsCP beyond BCP (NeurIPS 2025)?

  2. How does TsCP justify its asymptotic size guarantee compared with BCP's direct size constraint?

  3. Can the authors demonstrate a clear advantage over BCP under the same calibration data budget?

4 comments
AuthorsOpenReview ↗

Reply [1/3]

Thank you for your thoughtful review and constructive feedback. We have revised our manuscript in response to your suggestions, with all changes highlighted in blue. Below, we address each of your concerns in detail.


W1. The novelty is limited. BCP (NeurIPS 2025) has already studied adaptive miscoverage selection for prediction set size control, together with leave-one-out coverage estimation. The authors should better clarify what fundamentally distinguishes TsCP from BCP.

Q1. What is the fundamental contribution of TsCP beyond BCP (NeurIPS 2025)?

Answer: We thank the reviewer for this important observation. We acknowledge that our work is closely related to Backward Conformal Prediction (BCP) [1], particularly in addressing the Tunable Prediction Set Size (TPSS) problem and employing leave-one-out (LOO) coverage estimation. Nevertheless, we would like to emphasize several fundamental differences in the underlying framework, methodology, and theoretical analysis.

  1. Different underlying frameworks and objectives. BCP addresses TPSS through conformal e-prediction (eCP), leveraging the flexibility of e-values and their post-hoc validity to accommodate data-dependent miscoverage levels. However, this flexibility comes at the cost of departing from the classical rank-based structure of split conformal prediction (sCP). In contrast, our primary objective is to establish a framework for TPSS that retains the structural properties of sCP, particularly its quantile-based construction and rank locality. This distinction is important because preserving these properties enables a different characterization of coverage and a more direct analysis of the corresponding LOO estimator.

  2. Fundamentally different methodology and theoretical guarantees. Unlike BCP, which directly adapts the miscoverage level through an e-value inclusion threshold, TsCP introduces two independent calibration subsets: an approximation set Ca\mathcal C^a for selecting the size-aware miscoverage level and a calibration set Cc\mathcal C^c for constructing the final prediction set using standard conformal quantiles. This separation is essential for addressing the dependence introduced by adaptive miscoverage selection within sCP (Section 3 and Appendix C.1.1).

We have added a comparison with Backward Conformal Prediction at the end of Appendix C.1.2, clarifying the framework, calibration construction, and size-control trade-off.

[1] Gauthier et al. Backward Conformal Prediction. NeurIPS 2025.

AuthorsOpenReview ↗

Reply [2/3]

W2. sCP splits the calibration data into two subsets, potentially reducing calibration efficiency. More importantly, its prediction set size guarantee is asymptotic, while BCP constructs sets satisfying the prescribed size constraint by design.

Q2. More importantly, its prediction set size guarantee is asymptotic, while BCP constructs sets satisfying the prescribed size constraint by design.

Answer: We thank the reviewer for raising this important concern. We acknowledge that, unlike Backward Conformal Prediction (BCP) [1], which satisfies the prescribed size constraint by construction, our theoretical analysis guarantees only an asymptotic version of the size constraint (Theorem 4.4). Nevertheless, we would like to emphasize that TsCP achieves near-hard size control in practice, even with relatively small calibration datasets, as demonstrated by our extensive empirical evaluation in Appendix D.4.

Specifically, we evaluate the proportion of test points satisfying the prescribed hard size constraint, i.e., ∣Cδ(X)∣≤S(X)|C_\delta(X)|\leq S(X), across six regression and classification datasets and varying calibration sizes. The results, reported in Table 1 (Appendix D.4), are reproduced below for the reviewer's convenience.

Table 1: Hard-constraint satisfaction rate (%) across different total calibration set sizes.

Dataset1,0002,0003,0004,0005,0006,000
Synthetic Regression99.199.799.299.1100100
California Housing95.710095.496.197.8100
Superconductivity10099.5100100100100
Synthetic Classification10096.610010099.699.6
MNIST94.999.910098.895.9100
Covertype99.910010099.910099.9

As shown, even with only 1,000 calibration samples, TsCP satisfies the hard size constraint for at least 94.9% of test points across all six datasets, with four datasets exceeding 99%. At a calibration size of 6,000, four datasets achieve 100% satisfaction, while the remaining two achieve 99.6% and 99.9%, respectively. These results demonstrate that, although our theoretical guarantee is asymptotic, TsCP achieves behavior very close to hard size control in finite-sample settings.

Nevertheless, we fully acknowledge that establishing a finite-sample hard size guarantee remains an important theoretical advantage of BCP. Extending our framework to provide such a guarantee while preserving the structural properties of sCP is an important direction for future research.

[1] Gauthier et al. Backward Conformal Prediction. NeurIPS 2025.

W2. TsCP splits the calibration data into two subsets, potentially reducing calibration efficiency.

Answer: We thank the reviewer for raising this concern. We acknowledge that TsCP requires an additional approximation set Ca\mathcal C^a beyond the standard calibration set Cc\mathcal C^c, potentially increasing the amount of calibration data required. However, as discussed in Appendix D.4, this increase is relatively modest in practice.

In particular, Table 1 demonstrates that TsCP achieves near-hard size control even with small to moderate calibration set sizes. With only 1,000 total calibration samples, TsCP satisfies the prescribed hard size constraint for at least 94.9% of test points across all six datasets, with four datasets exceeding 99%. At 6,000 samples, four datasets achieve 100% satisfaction, while the remaining two achieve 99.6% and 99.9%, respectively. These results suggest that the additional approximation set Ca\mathcal C^a need not be prohibitively large to achieve effective size control. Thus, the additional calibration requirement represents a constant-factor increase, rather than a qualitatively larger demand for calibration data.

Furthermore, as discussed in Theorem D.1, the conditional coverage induced by classical conformal calibration follows a Beta distribution, whose variance decreases at a rate of O(n−1)\mathcal O(n^{-1}). Standard order-statistic concentration results also imply exponentially decreasing probabilities of fixed deviations from the target coverage. Consequently, a constant-factor increase in calibration data requirements does not fundamentally alter the asymptotic concentration behavior of conformal calibration. In TsCP, Cc\mathcal C^c retains the standard calibration role of sCP, while Ca\mathcal C^a is used exclusively to determine the size-aware miscoverage level.

AuthorsOpenReview ↗

Reply [3/3]

W3. The experimental evaluation does not sufficiently demonstrate the practical advantage of TsCP over BCP. A comparison under the same calibration data budget is particularly important. Q3. Can the authors demonstrate a clear advantage over BCP under the same calibration data budget?

Answer: We thank the reviewer for emphasizing the importance of a fair comparison under the same calibration data budget. However, we would like to clarify that all comparisons between TsCP and BCP [1] in our experiments are already conducted under the same total calibration data budget, as explicitly described in Appendices D.1.2 and D.2.2.

Specifically, for a given total calibration size Ncal=2nN_{\mathrm{cal}}=2n, TsCP partitions these samples into two subsets Ca\mathcal{C}^a and Cc\mathcal{C}^c, each of size nn, while the eCP-based baseline is evaluated under the same total calibration budget. Furthermore, both methods are evaluated using the same randomly sampled calibration data and shared test points to ensure a fair comparison. For Task (a), we fix Ncal=2000N_{\mathrm{cal}}=2000, while for Task (b), we vary the total calibration size up to 6,000 samples.

Despite splitting its calibration data into two subsets, TsCP consistently demonstrates empirical advantages over BCP under this equal-budget setting. In particular, Figures 3 and 8 show that TsCP achieves smaller average prediction sets at comparable empirical coverage across all six regression and classification datasets. Furthermore, Figures 3 and 9 demonstrate that the LOO-based coverage estimate of TsCP remains substantially closer to empirical coverage, whereas the eCP-based estimate exhibits a noticeably larger and more persistent gap.

[1] Gauthier et al. Backward Conformal Prediction. NeurIPS 2025.


We hope we have cleared your concerns about our work. We have also revised our manuscript according to your comments, and we would appreciate it if we can get your further feedback at your earliest convenience.

AuthorsOpenReview ↗

Any Questions from Reviewer NwH1 on Our Rebuttal?

We would like to thank the reviewer again for your thoughtful review and valuable feedback.

We would appreciate it if you could let us know if our responses have addressed your concerns and whether you still have any other questions about our rebuttal.

We would be happy to do any follow-up discussion or address any additional comments.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Global Response

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their thoughtful evaluations, constructive comments, and valuable suggestions, which have helped us improve our manuscript substantially. We appreciate the reviewers' recognition of the importance of tunable prediction set size control, the simplicity of our split-calibration construction, and the theoretical contributions of our framework.

Several common concerns emerged regarding (1) the novelty of TsCP relative to BCP and related approaches, (2) the scope of our theoretical guarantees, particularly the distinction between asymptotic and hard size control, and (3) the fairness and completeness of our empirical comparisons.

To address these concerns, we have expanded our theoretical discussions, clarified the underlying assumptions and limitations, and conducted additional experiments evaluating coverage, prediction efficiency, and budget satisfaction jointly under matched configurations. We have also investigated discrepancies between experimental protocols and clarified important implementation details. Importantly, we have further revised the manuscript to more clearly distinguish our work from prior approaches and emphasize our main contributions. In particular, our contribution lies in addressing TPSS within the classical split conformal prediction (sCP) framework, preserving its quantile-based structure while establishing a distinct calibration methodology, an exact marginal coverage characterization, and theoretical guarantees for the corresponding LOO coverage estimator. We have also clarified the differences in theoretical guarantees and practical trade-offs between our method and existing approaches, particularly BCP.

We believe these revisions strengthen the theoretical positioning, empirical evidence, and overall clarity of our work. We thank the reviewers again for their time and consideration, and we welcome any further feedback.

AuthorsOpenReview ↗

Summary of Revisions

Incorporating the reviewers' valuable comments and suggestions, we have revised our manuscript and conducted additional empirical studies. We summarize the main revisions below:

  1. We have expanded our comparison with existing approaches to TPSS (Appendix C.1.2), including Backward Conformal Prediction (BCP), Conformal Risk Control, post-hoc truncation, and prediction-set optimization. We clarify the methodological distinctions, theoretical guarantees, and practical trade-offs, emphasizing that TsCP preserves the quantile-based structure of classical sCP while addressing TPSS through independent calibration splits.

  2. We have clarified the scope and limitations of our theoretical guarantees (Appendix D.1.1). Specifically, Theorem 4.2 establishes an exact marginal coverage identity involving an unknown correction term, rather than a directly computable finite-sample coverage lower bound. Similarly, Theorem 4.4 establishes asymptotic size compliance under stated assumptions, rather than finite-sample hard size control.

  3. We have clarified the experimental evaluation of our coverage characterization (Appendices D.1.1 and D.2.1). In particular, we explain that the reference coverage curves are computed using independent Monte Carlo calibration–test sampling, rather than direct numerical evaluation of the theoretical binomial expression. Their agreement with empirical coverage therefore assesses consistency between independent estimates of achieved marginal coverage.

  4. We have clarified the applicability of our theoretical assumptions to classification experiments (Appendix D.2). In particular, we acknowledge that our deterministic classification scores do not jointly satisfy Assumptions 4.1 and 5.1, and that the sufficient conditions of Theorem 4.4 have not been verified for the evaluated classification settings. Accordingly, the reported coverage-estimation and budget-satisfaction results are interpreted as empirical evidence beyond the scope of the current theoretical guarantees.

  5. We have conducted an additional joint evaluation of empirical coverage, average prediction-set size, and hard-budget satisfaction under matched operating configurations (Appendix D.4). Across three classification datasets, TsCP achieves smaller prediction sets than BCP at comparable coverage, with budget-satisfaction rates ranging from 94.45% to 95.59%.

  6. We have clarified the construction of input-dependent budget functions and their independence from conformal calibration (Appendix D.5). Specifically, the normalization quantiles are computed exclusively from training data and remain fixed during calibration and LOO estimation, preserving the independence assumptions required by our theoretical analysis.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer aopg,

Thank you for taking the time to read our responses and provide further feedback.

Regarding the additional baselines, as discussed in our previous response and Appendix C.1.2, these methods do not directly address both requirements of TPSS under their standard formulations. For example, post-hoc truncation can enforce the prescribed size constraint but may invalidate the original coverage guarantee, while Conformal Risk Control and prediction-set optimization do not directly enforce an instance-wise size budget. We agree, however, that adapting these methods and evaluating their coverage and size-constraint satisfaction experimentally would provide a more comprehensive comparison and help illustrate these differences in practice. Your suggestion will be helpful in guiding our future revisions.

Regarding the presentation, we agree that the organization of the paper could be further improved. We will carefully reconsider which results and discussions should appear in the main text and which should remain in the appendix, with the goal of making the paper more self-contained and easier to follow.

Thank you again for your thoughtful comments and constructive suggestions. We appreciate the time and effort you have put into reviewing our work, and your feedback will be valuable as we continue improving the manuscript.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer NwH1,

Thank you for taking the time to read our responses and carefully consider our revisions. We appreciate your feedback and understand your remaining concerns. Your comments have helped us better clarify the contributions and limitations of our work and will also be valuable for future revisions.

Thank you again for your time and thoughtful review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.