- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates why full conformal prediction can produce larger prediction sets than split conformal prediction. It decomposes prediction margins into calibration-data adaptation, candidate-label “formability,” and calibration-threshold movement. Under exact Bayesian updating, it connects formability to the posterior variance of class probabilities and derives covariance-based margin bounds. It then proposes probability-weighted deterministic refits as a practical uncertainty measure.
Evaluations are on controlled synthetic experiments, exact Bayesian regression, and benchmarks on graphs, images, and multiple-choice language tasks. Results are competitive on image and graph error detection, while language results are weaker.
Critical strengths
- The interpretable theoretical connection.
Lemma 3.1 establishes an exact relationship between candidate-label adaptation and posterior predictive disagreement. Theorem 3.2 additionally accounts for calibration-threshold movement. Together, these provide a useful explanation of how refitting affects full CP beyond the usual argument about improved data efficiency.
- Complementary theoretical and empirical validation.
The conjugate Bayesian experiment checks the identities and margin bounds directly, while comparisons against pNML, Laplace approximations, ensembles, and other uncertainty measures assess practical usefulness. Competitive image and graph results suggest that the continuous formability score contains useful information beyond the discretized set-size gap.
Critical weaknesses
-
The main identity does not directly justify the implemented estimator. The theory concerns posterior expectations under exact Bayesian updating, whereas the practical method substitutes deterministic/MAP refits. Without a quantitative connection or direct validation against reference posterior variance, the central epistemic-uncertainty interpretation remains insufficiently supported for the implemented method.
-
Some CP-efficiency interpretations exceed the established results. The paper describes the calibration-trained reference set as an “adaptation floor” or lower bound. However, the decomposition permits negative net formability when threshold movement exceeds the candidate’s probability increase, and Theorem 3.2 does not generally establish the claimed ordering. Because this reference also lacks coverage validity, its smaller size alone does not establish improved conformal efficiency.
Questions for authors
-
What quantitative evidence justify interpreting the deterministic score as an approximation to Bayesian posterior variance? Can you provide a direct comparison on a tractable model where both quantities can be computed, rather than relying on error-detection performance?
-
Under what conditions is the adaptation reference a lower bound on full-CP set size? If it is intended only as an empirical reference, which directional or convergence claims about the full–split gap remain theoretically established?
We thank the reviewer for pointing out their concerns; in light of the mentioned points we add the following results and discussions, hopefully to address their concerns.
Stability across seeds. On CIFAR-10 (a 10-class image dataset), for 3 data splits and 10 seeds each, we retrained the base model and all 2,000 refits with a new seed and compared the per-image scores between seeds.
Refits. For one test point and one candidate class, a refit trains a new network from freshly initialised weights on all labelled data plus that one pseudo-labelled point. The base model and every refit of the same data split start from the same initial weights and use the same dropout, so they differ only by the one added label. On images the network is a two-layer classifier with 64 hidden units on top of fixed image features, trained for 300 passes over the data with the Adam optimiser.
HMC. We evaluated over 3 splits of 200 points for CIFAR-10 and CoraML datasets and the AUROC pooled over the 600 points; the mean of the three per-split AUROCs is the same to within 0.003.
Comparison with softmax entropy. We note that on the real-data the amount of epistemic uncertainty is not clear; specially in current baselines where the models are already performing well. The main comparison of softmax entropy and formability is where we can see each of the uncertainties separately. Fig 4. specifically shows that when blobs get closer formability reacts to the decrease of the epistemic (at the center) uncertainty while softmax remains steady.
We added this experiment which further highlights this effect. We synthesize a dataset where the input is a one-hot indicating either of the 3000 groups, and each group is assigned with a n, and \epsilon, which controls the number of observation, and the true posterior distribution which is a permutation of (1−ϵ,ϵ/2,ϵ/2). The first controls the epistemic and the second the aleatoric uncertainty. The model is a linear model so that Bayesian posterior can be computed analytically. Rank agreement with the exact posterior variance is 1.00 (1 is perfect) for formability, and 0.48 for softmax entropy. This is the breakdown in different regions of high/low alea/epis uncertainties.
Computational cost. While we agree the method is costly, we can reduce this cost by only applying the refit to a small part of the network, like the last layers, or only the weights with higher magnitude. The following compare different refit regimes in comparison with a 5 member ensemble (CIFAR-10, 200 images):
We discuss the points on full cp in the next comment.