acceptodds
Withdrawn from ICLR 2027

Quantile Merging for Bootstrapped Transductive Conformal Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Transductive conformal prediction constructs joint prediction sets for groups of test samples, but practical implementations face a fundamental challenge: calibration data are typically available as i.i.d. individual samples, whereas inference is performed on groups whose size may vary at deployment time. We study how to construct transductive calibration sets from a fixed pool of calibration samples while preserving coverage. We consider bootstrap sampling from the empirical calibration distribution and show that valid marginal coverage requires an adjustment of the calibration quantile. To address the variance increase from bootstrap sampling, we introduce a -fold transductive conformal framework that repeats bootstrap calibration and aggregates the resulting predictors using p-value and e-value merging techniques. We show that all considered aggregation schemes admit an equivalent quantile-level implementation, reducing inference to standard split conformal prediction with a merged calibration quantile. We further theoretically characterize the effects of calibration size, group size, and the number of folds on coverage and efficiency. Empirical results validate the theory and demonstrate substantial improvements in stability over single fold bootstrap calibration.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
transductive conformal prediction, p-value merging, e-value merging, conditional coverage

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KGhdOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies joint prediction for a batch of test observations when calibration data are available as individual samples. It forms calibration groups by bootstrap sampling and aggregates repeated calibrations through p-value or e-value merging. With a common score function, the merged decision reduces to a scalar threshold. The paper also analyzes bootstrap quantiles and conditional coverage for the single-test case, and evaluates the methods on image-classification tasks with perturbed labels.

Critical strengths
  1. The mismatch between individual calibration data and batch prediction is a relevant problem. Reducing a merged decision to a common score threshold is useful computationally and permits reuse of the underlying transductive inference procedure.
  2. The distinction between variability from the original calibration sample and additional Monte Carlo variability is informative. The single-test bootstrap analysis provides a useful starting point for understanding why repeated resampling can stabilize thresholds without supplying new calibration information.
Critical weaknesses
  1. The validity argument does not cover the central transductive procedure. Section 4 explicitly restricts its analysis to a single test point, and Proposition 4.2 corrects a quantile obtained by resampling existing scalar scores. Algorithms 1–2 instead resample groups of observations from a shared empirical distribution. Their group scores are not generally exchangeable with the score of a fresh test group, so Equation (2) cannot simply be treated as a valid p-value. Valid p-value or e-value merging does not repair invalid inputs. A guarantee addressing this distinction, with explicit dependence on group size, is essential to the paper’s main claim.
  2. Proposition 4.3 omits a centering error that can survive under its stated joint-growth condition. Remark G.2 acknowledges an error of order one over the number of bootstrap groups when replacing the exact Beta quantile by its Gaussian approximation. This error need not vanish after multiplication by the square root of the calibration size. For example, take both quantile levels to be one half, an even number N of bootstrap groups, calibration size N², and N folds. The relevant Beta(N/2 + 1, N/2) median differs from one half by approximately 1/(2N), leaving a limiting shift of one half. This contradicts the zero-centered limit and the claim that median aggregation removes the bias for every allowed scaling.
Questions for authors
  1. What finite-sample result justifies the fold-level p-values or e-values for groups of more than one observation? Please state the actual correction used in Algorithms 1–2 and how it depends on the original calibration size and the group size.
  2. Can Proposition 4.3 be corrected by centering at the exact Beta quantile or imposing an additional growth condition? Please address the median example above and specify which variance conclusions remain valid.
  3. At a fixed total number of sampled groups, how do multiple merged folds compare with one larger bootstrap calibration? This comparison is needed to distinguish an advantage of merging from the effect of additional Monte Carlo computation.
Reviewer_eGHEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Quantile Merging, a K-fold framework to construct valid transductive conformal prediction sets for variable-sized groups. The authors use bootstrap calibration with quantile-level p-value and e-value merging, which guarantees valid marginal coverage while improving efficiency and stability. The experiments validate the theoretical claims and demonstrate substantial improvements over single-fold calibration.

Critical strengths
  1. The effectiveness of the proposed method is validated through both theoretical analysis and empirical evaluation.
  2. The experimental settings are clearly explained and easy to follow.
  3. The authors conduct extensive experiments to systematically study the properties of the proposed framework.
Critical weaknesses
  1. The empirical evaluation is somewhat limited in scope, focusing exclusively on image classification tasks.
  2. The study would benefit from evaluating a broader variety of (non-)conformity scores.
  3. The code is currently unavailable, which hinders reproducibility.
Questions for authors

While I am not a core expert in this specific sub-field, I offer the following questions and suggestions:

  1. Could the proposed framework be extended to other data modalities (e.g., text, graphs) and tasks, such as LLM generation and regression? Adding a brief discussion or preliminary results on these would strengthen the paper's generalizability.
  2. Could the authors incorporate more advanced (non-)conformity scores into the experiments, such as RAPS [1] and SAPS [2]?
  3. The overall structure of the paper feels incomplete. It would be highly beneficial to include a dedicated Conclusion and Discussion section to summarize the findings and outline potential future directions.

[1] Uncertainty Sets for Image Classifiers Using Conformal Prediction
[2] Conformal Prediction for Deep Classifier via Label Ranking

Reviewer_ZfnUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a new method that adapts the Split Conformal Prediction (usually of inductive nature, reusing the fixed pretrained model) to Transductive Conformal Prediction (usually requires retraining/adaptation), which is used for simultaneous prediction for a batch of test points. One of the previous works (TSCP, Behboodi et al., 2025) addressed computation inefficiencies of TCP-based technique with hard calibration sample splitting and efficient dynamic-programming based final prediction set construction. In the presented work, hard splitting of the calibration sample is replaced by a K-random-fold / Bootstrap construction, which provides better sample efficiency. The central technical result is that the proposed sample splitting-aggregation scheme is equivalent to a substantially more computationally efficient aggregation of the calibration quantiles. Current work also provides theoretical characterization of how calibration size m, batch size n, and number of folds K affect both marginal and conditional coverage and efficiency. Experiments on MNIST, Fashion-MNIST, CIFAR-10 and CIFAR-100 assess the the proposed method.

Critical strengths
  1. Motivation is clear and practical: inefficiency of previous TSCP approach result in non-informative prediction sets in some relevant cases.
  2. The n = 1 bootstrap analysis is clean and instructive.
Critical weaknesses
  1. All theoretical analysis, including validity, is performed for the case when the test batch size n=1n=1, which is not what is advertised throughout the paper (including the abstract). Consequently, TCCP's claim that it "preserves coverage" in the target regime of n>1 is not backed by the supplied proofs.
  2. The usefulness of resampling is not supported: when sampling with replacement to form groups, a single fold of Ns′=K×NsN_s' = K \times N_s uses the same amount of score computations and yields a smaller or equal asymptotic variance. However the experiments never vary NsN_s and never compare to this natural baseline. Thus the variance reduction that is attributed to the KK-fold construction can be achieved by taking a large single bootstrap sample.
Questions for authors
  1. (Clarification) In the paragraph "p-value based aggregation" (page 3), the notation sometimes computes p-values for labels ym+1m+ny_{m+1}^{m+n} and other times for the scores t. Also, the merging function M is defined right before the Proposition 3.1, before M was first used. I may have missed something, but this paragraph is very confusing and hard to read. Can you please clarify this notation and the paragraph overall?
  2. Why use KK folds of NsN_s groups instead of one fold of K×NsK \times N_s groups? Please compare the two at the same total number of groups. Additionally, please also consider variability of calibration sets.
  3. Why do you consider other batch prediction methods "complementary rather than competing" to the proposed approach (page 16)? The work of Lee et al., 2024 and Gazin et al., 2025 aim for the same goal, avoid the disjoint partitions and thus seem highly relevant. Can you compare against some methods from this line of work?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.