- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates why transductive conformal prediction (specifically Conf-OT) fails for vision-language models under label shift. The authors show that forcing calibration class proportions onto the pooled data creates logit biases that contradict with the query prior. To fix this, the authors propose a split-sample prior-aware conformal predictor that separates transport and temperature fitting from query-prior estimation and calibration. The authors conducted experiments on six classification datasets , and various CLIP backbones and show the method handles synthetic label shift well (91% coverage using 3.30 labels per set), and more efficient than the standard baseline.
Critical strengths
- The theoretical analysis of the failure scenario of transductive conformal prediction during label shift is insightful.
- The experiments cover a wide range of datasets, backbones, and shifts. Also reporting of negative results (like behavior under class-conditional shifts) gives a transparent view of the method's limits.
Critical weaknesses
-
I think the main weakness is the lack of theoretical guarantees. Conformal prediction is about reliable coverage, but the proposed method lacks a finite-sample guarantee under label shift. The theorems provided by authors rely on population-level calibration or assumptions such as class-symmetry. Empirically, the method still suffers from undercoverage in specific configurations e.g., dropping below 85% despite a 90% target. Without guarantees, the whole point of conformal prediction is questionable!
-
The majority of the performance gain comes from a simple fix of removing query samples from the transport-fitting stage (I mean the Per-split baseline). That step accounts for roughly 84% of the efficiency improvement over standard split conformal prediction. The rest of the method pipeline e.g., prior estimation, shift detection, stability checks adds heavy complexity for very little marginal benefit. Particularly on long-tailed ImageNet tasks, the method somehow ignores its own correction and reverts to the baseline.
Questions for authors
-
Please see my comments above regarding coverage guarantee, it will b great if author provide this possibly factoring in the uncertainty of the estimated priors? Also, could you report worst case configuration level undercoverage instead of just aggregate means?
-
How does the full method compare against a simple calibration-only transport followed by standard Black-box shift estimation (BBSE) or EM prior correction? Since the simple baseline is already doing most of the work, you need to prove the full pipeline is truly necessary.
-
The stability thresholds are selected using coverage behavior on ViT-B/16 across the evaluation dataset, particularly the method falls back to the baseline on ImageNet, does the active prior correction actually work on a completely independent dataset without just reverting? How would it perform with larger query batches or better prior estimators?
Thank you for your careful review. We derive a finite-sample error bound and add matched BBSE/EM comparisons. The comparisons identify complementary roles: active correction gives compact sets on independent CIFAR-100, whereas fallback avoids large coverage losses in the prespecified Aircraft test.
1. Finite-sample coverage and estimated-prior uncertainty.
For an independently fixed score map/prior, our bound separates calibration error from prior uncertainty. To evaluate the adaptive steps as well, we report full-pipeline coverage below. Under label shift, with positive prior p and classwise i.i.d. calibration scores, the maximum-weight-atom quantile gives
P(Y in C(X)) >= 0.9 - 0.9/(n_min+1) - R TV(p,pi_q).
Here n_min is the smallest class count, pi_q the query prior, and R the range of classwise coverages averaged over calibration sampling. The map and p are fixed independently of the calibration features and test image. Within-class exchangeability gives the calibration term, and a total-variation bound gives the prior-error term. Thus larger calibration samples and better prior estimates tighten distinct terms.
At a population threshold calibrated under p, coverage error relative to the target equals the negative inner product of p-pi_q with classwise coverage. Consequently, errors on poorly covered classes matter differently from equal-sized errors elsewhere, motivating selection by the coverage effect of correction.
2. Worst configuration-level coverage.
The added distributional comparison shows improved lower-tail coverage across 123 shifted dataset/backbone/strength configurations, each averaged over 20 seeds. Minimum/5th-percentile/median coverage is 85.3%/87.5%/90.1% for Ours versus 83.8%/84.9%/89.4% for Per-split. Counts below 85%/88%/90% are 0/8/56 versus 9/20/91. Separately, the leave-one-dataset-out minimum is 84.96%. Across individual runs, the 5th percentile rises from 80.6% to 83.2%. Both methods' minimum, 57.3%, occurs on a Caltech101 draw where SCP reaches 61.2% and Ours falls back.
3. Independent correction versus Per-split, BBSE and EM.
On independent CIFAR-100, Ours improves coverage and reduces size relative to Per-split, with higher coverage and larger sets than EM-BCTS. To test transfer without retuning, we keep CLIP ViT-B/32 and the original thresholds, using 20 seeds, 50 calibration images/class split into A/B, and 1,000 distinct queries from the remaining 550/class at concentration 0.1. BBSE and EM share the base map and calibration half, with BCTS fitted only on A. Entries are mean coverage (%) and size.
As the table shows, Ours improves coverage by 2.00 points [95% CI 0.99,3.24] and reduces size by 55%, or 1.85 labels [1.75,1.96], over Per-split. Relative to EM-BCTS, it gains 1.89 coverage points [1.05,2.70] with 0.17 additional labels.
Both corrected methods retain smaller sets than Per-split at matched observed coverage. Descriptive interpolation at 90% gives 1.35/1.34/3.45 labels for Ours/EM-BCTS/Per-split. We use five fixed alpha values from 0.05 to 0.15, recomputing the cap at each value. This curve comparison uses no test-label selection of deployment parameters.
4. Independent threshold transfer, larger queries and fallback.
Correction transfers without retuning: it activates in all 20 distinct-query CIFAR runs and all 60 shifted runs in the earlier independent test-only experiment. In that earlier experiment, strongest-shift coverage/size is 91.99%/1.43 versus Per-split's 91.03%/3.03. At concentration 1, coverage is 89.02% versus 89.58%.
With 5,000 CIFAR-100 queries from the enlarged pool, Ours activates in 19/20 runs and reaches 90.91%/1.54, versus Per-split's 89.62%/3.33 and EM-BCTS's 90.32%/1.34. This arm has 83.5% distinct images, changing both batch size and diversity. In contrast, long-tailed ImageNet has only 12–13 calibration examples per class for its 1,000-class confusion matrix. Fallback there is the prescribed response to unstable estimation.
Fallback avoids large coverage losses on Aircraft/ViT-B/32 under the fixed estimator budget. With the original 20 seeds and four shift settings, Ours falls back in all 80 runs. Under unchanged baseline settings, three-shift mean coverage (%)/size is 90.18/20.01 for Ours/Per-split, 52.89/7.22 for BBSE, 40.11/5.28 for EM-TS and 42.32/5.14 for EM-BCTS. The paired Ours-minus-EM-BCTS coverage gap is 47.86 points [45.12,50.77]. Ours' shift-level mean coverage ranges from 89.54% to 90.91%. Both EM variants reach their fixed 1,000-step cap in 79/80 runs. These results complement active correction on CIFAR.