acceptodds
Under review as a conference paper at ICLR 2027

Prior-Aware Transductive Conformal Prediction for Vision–Language Models under Label Shift

Abstract

Conformal prediction gives zero-shot vision–language models prediction sets with a guaranteed probability of containing the true label. Transductive methods shrink these sets by adapting scores to an unlabeled batch of query images, preserving coverage when calibration and query points are exchangeable and treated symmetrically. Conf-OT jointly fits optimal transport to calibration and query scores while fixing the pooled average class probabilities to the calibration label proportions. Compared with split conformal prediction, which calibrates fixed scores, Conf-OT reports smaller sets when both subsets share a distribution. We examine its coverage under label shift, where query class proportions change while the image distribution within each class stays fixed. We show that enforcing the calibration proportions drives the fitted class bias against classes that become more frequent in the query batch. Standard prior correction upweights these same classes. In a class-symmetric population model, we prove that the resulting coverage shortfall grows quadratically with small mismatches in class proportions. Under the strongest tested shift, Conf-OT averages 69.3% coverage versus 89.7% for split conformal prediction at a 90% target. To address this mechanism, we fit transport on half of the calibration set alone and estimate query class proportions from the other half and the query predictions. Holding the map fixed removes most of the shortfall, and prior correction reduces the remaining shortfall when class proportions can be reliably estimated from predictions. Under the same shift, our predictor attains 91.0% mean coverage with 3.30 labels per set versus 4.35 for split conformal prediction. These results link the coverage loss to pooled transport fitting and support prior-aware correction for compact prediction sets under label shift.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal prediction, uncertainty quantification, label shift, vision-language models, zero-shot classification, optimal transport, class-prior estimation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_feRcOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why transductive conformal prediction (specifically Conf-OT) fails for vision-language models under label shift. The authors show that forcing calibration class proportions onto the pooled data creates logit biases that contradict with the query prior. To fix this, the authors propose a split-sample prior-aware conformal predictor that separates transport and temperature fitting from query-prior estimation and calibration. The authors conducted experiments on six classification datasets , and various CLIP backbones and show the method handles synthetic label shift well (91% coverage using 3.30 labels per set), and more efficient than the standard baseline.

Critical strengths
  • The theoretical analysis of the failure scenario of transductive conformal prediction during label shift is insightful.
  • The experiments cover a wide range of datasets, backbones, and shifts. Also reporting of negative results (like behavior under class-conditional shifts) gives a transparent view of the method's limits.
Critical weaknesses
  • I think the main weakness is the lack of theoretical guarantees. Conformal prediction is about reliable coverage, but the proposed method lacks a finite-sample guarantee under label shift. The theorems provided by authors rely on population-level calibration or assumptions such as class-symmetry. Empirically, the method still suffers from undercoverage in specific configurations e.g., dropping below 85% despite a 90% target. Without guarantees, the whole point of conformal prediction is questionable!

  • The majority of the performance gain comes from a simple fix of removing query samples from the transport-fitting stage (I mean the Per-split baseline). That step accounts for roughly 84% of the efficiency improvement over standard split conformal prediction. The rest of the method pipeline e.g., prior estimation, shift detection, stability checks adds heavy complexity for very little marginal benefit. Particularly on long-tailed ImageNet tasks, the method somehow ignores its own correction and reverts to the baseline.

Questions for authors
  • Please see my comments above regarding coverage guarantee, it will b great if author provide this possibly factoring in the uncertainty of the estimated priors? Also, could you report worst case configuration level undercoverage instead of just aggregate means?

  • How does the full method compare against a simple calibration-only transport followed by standard Black-box shift estimation (BBSE) or EM prior correction? Since the simple baseline is already doing most of the work, you need to prove the full pipeline is truly necessary.

  • The stability thresholds are selected using coverage behavior on ViT-B/16 across the evaluation dataset, particularly the method falls back to the baseline on ImageNet, does the active prior correction actually work on a completely independent dataset without just reverting? How would it perform with larger query batches or better prior estimators?

1 comment

Thank you for your careful review. We derive a finite-sample error bound and add matched BBSE/EM comparisons. The comparisons identify complementary roles: active correction gives compact sets on independent CIFAR-100, whereas fallback avoids large coverage losses in the prespecified Aircraft test.

1. Finite-sample coverage and estimated-prior uncertainty.

For an independently fixed score map/prior, our bound separates calibration error from prior uncertainty. To evaluate the adaptive steps as well, we report full-pipeline coverage below. Under label shift, with positive prior p and classwise i.i.d. calibration scores, the maximum-weight-atom quantile gives

P(Y in C(X)) >= 0.9 - 0.9/(n_min+1) - R TV(p,pi_q).

Here n_min is the smallest class count, pi_q the query prior, and R the range of classwise coverages averaged over calibration sampling. The map and p are fixed independently of the calibration features and test image. Within-class exchangeability gives the calibration term, and a total-variation bound gives the prior-error term. Thus larger calibration samples and better prior estimates tighten distinct terms.

At a population threshold calibrated under p, coverage error relative to the target equals the negative inner product of p-pi_q with classwise coverage. Consequently, errors on poorly covered classes matter differently from equal-sized errors elsewhere, motivating selection by the coverage effect of correction.

2. Worst configuration-level coverage.

The added distributional comparison shows improved lower-tail coverage across 123 shifted dataset/backbone/strength configurations, each averaged over 20 seeds. Minimum/5th-percentile/median coverage is 85.3%/87.5%/90.1% for Ours versus 83.8%/84.9%/89.4% for Per-split. Counts below 85%/88%/90% are 0/8/56 versus 9/20/91. Separately, the leave-one-dataset-out minimum is 84.96%. Across individual runs, the 5th percentile rises from 80.6% to 83.2%. Both methods' minimum, 57.3%, occurs on a Caltech101 draw where SCP reaches 61.2% and Ours falls back.

3. Independent correction versus Per-split, BBSE and EM.

On independent CIFAR-100, Ours improves coverage and reduces size relative to Per-split, with higher coverage and larger sets than EM-BCTS. To test transfer without retuning, we keep CLIP ViT-B/32 and the original thresholds, using 20 seeds, 50 calibration images/class split into A/B, and 1,000 distinct queries from the remaining 550/class at concentration 0.1. BBSE and EM share the base map and calibration half, with BCTS fitted only on A. Entries are mean coverage (%) and size.

MethodCoverageSize
SCP89.914.25
Conf-OT82.413.96
Cal-only transport89.463.35
Per-split89.763.35
BBSE89.191.29
EM-TS88.331.29
EM-BCTS89.871.33
Ours91.751.50

As the table shows, Ours improves coverage by 2.00 points [95% CI 0.99,3.24] and reduces size by 55%, or 1.85 labels [1.75,1.96], over Per-split. Relative to EM-BCTS, it gains 1.89 coverage points [1.05,2.70] with 0.17 additional labels.

Both corrected methods retain smaller sets than Per-split at matched observed coverage. Descriptive interpolation at 90% gives 1.35/1.34/3.45 labels for Ours/EM-BCTS/Per-split. We use five fixed alpha values from 0.05 to 0.15, recomputing the cap at each value. This curve comparison uses no test-label selection of deployment parameters.

4. Independent threshold transfer, larger queries and fallback.

Correction transfers without retuning: it activates in all 20 distinct-query CIFAR runs and all 60 shifted runs in the earlier independent test-only experiment. In that earlier experiment, strongest-shift coverage/size is 91.99%/1.43 versus Per-split's 91.03%/3.03. At concentration 1, coverage is 89.02% versus 89.58%.

With 5,000 CIFAR-100 queries from the enlarged pool, Ours activates in 19/20 runs and reaches 90.91%/1.54, versus Per-split's 89.62%/3.33 and EM-BCTS's 90.32%/1.34. This arm has 83.5% distinct images, changing both batch size and diversity. In contrast, long-tailed ImageNet has only 12–13 calibration examples per class for its 1,000-class confusion matrix. Fallback there is the prescribed response to unstable estimation.

Fallback avoids large coverage losses on Aircraft/ViT-B/32 under the fixed estimator budget. With the original 20 seeds and four shift settings, Ours falls back in all 80 runs. Under unchanged baseline settings, three-shift mean coverage (%)/size is 90.18/20.01 for Ours/Per-split, 52.89/7.22 for BBSE, 40.11/5.28 for EM-TS and 42.32/5.14 for EM-BCTS. The paired Ours-minus-EM-BCTS coverage gap is 47.86 points [45.12,50.77]. Ours' shift-level mean coverage ranges from 89.54% to 90.91%. Both EM variants reach their fixed 1,000-step cap in 79/80 runs. These results complement active correction on CIFAR.

Reviewer_bc8sOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies conformal prediction for vision–language models under label shift. It identifies a failure mode of Conf-OT: fitting transport on pooled calibration and query images while constraining their average class probabilities to match the calibration prior induces a shared class-logit bias that moves against changes in the query prior. Under class-symmetry assumptions, the authors derive a locally quadratic coverage shortfall. They propose fitting the score map on one calibration half, estimating the query prior from the other half and unlabeled queries, and applying selective prior correction with weighted calibration.

Critical strengths

The main result is a failure mechanism paired with an ablation that isolates it, rather than just a performance gap. The shared-bias argument predicts that the calibration-prior constraint should suppress classes that are more frequent in the query, and the paper then removes exactly the suspected cause: with the temperature and calibration set fixed, leaving query images out of the transport fit takes coverage at β=0.1 from 0.693 to 0.886. Tables 2 and 18 report this comparison. I found it a convincing causal check, and it does not rely on the theory being right.

The evaluation is broad and, more importantly, honest about its failures. Besides the 41 main configurations, the paper covers adapted scores, class counts up to 1000, long-tailed queries, and natural shifts. On long-tailed ImageNet queries the stability gate never opens and the predictor simply returns Per-split, as shown in Table 29. Table 30 shows that on ImageNet-V2 even the oracle prior reaches only 0.847 coverage, because P(X|Y) has changed. Section 6 also discloses that the class-symmetric coefficient overstates measured loss by a factor of three to four, and Appendix B.14 reports the fixed-point residual at the published T=3 setting.

Critical weaknesses
  1. The guarantees do not cover the predictor that is evaluated. The quadratic result is a population theorem under class symmetry and converged Sinkhorn iterations, while the experiments run T=3, and under label shift the full predictor with its pretest, stability gate, shrinkage and cap has no finite-sample coverage guarantee. The only finite-sample bound, about 0.845, applies under no shift with an independent split, whereas the main evaluation uses stratified splits. Section 6 acknowledges this, but the abstract and title still present a conformal predictor with 91.0% coverage; the distinction should be made up front.
  2. Most of the headline efficiency gain comes from the fixed map, not the prior-aware component. Set size drops from 4.35 to 3.47 just by fitting the map on calibration alone, and only to 3.30 after correction. The gate rejects essentially all repetitions on FGVCAircraft and Flowers102, where the predictor is then the fixed-map version. Correction is useful where it fires, and coverage also rises from 0.889 to 0.910, but it is a conditional gain assembled from standard pieces such as BBSE, weighted conformal calibration and pretesting.
  3. There is no figure that shows the central claim. The mechanism is described only in equations, and the coverage-versus-shift comparison for the main methods lives in tables; a single figure combining a schematic of the pooled bias with the coverage curves would make the paper much more readable.
Questions for authors
  1. Can you report per-dataset coverage, set size, and the fraction of repetitions with nonzero correction for Conf-OT, the calibration-only fit, Per-split and Ours?
  2. How much do the coverage loss and the fitted quadratic coefficient change when T is increased from 3 to convergence?
  3. Do the strongest-shift results survive a no-replacement query design, and what does the distribution of configuration-level coverage look like?
1 comment

Thank you for your detailed comments. We clarify the coverage conditions, compare BBSE/EM, and test whether pooled failure persists at convergence and with distinct queries. The results confirm its persistence. They also show complementary benefits: active correction gives compact sets on independent CIFAR-100, while fallback protects coverage on Aircraft.

1. Which guarantees apply to the evaluated predictor?

The finite-sample bound covers an independently fixed map and positive prior p, while full-pipeline coverage is empirical. The quadratic theorem concerns converged, class-symmetric population transport. Accordingly, the revised abstract separates these claims from empirical 91.0% mean coverage. Lemma 1(a)'s shared bias is exact at every finite iteration. With classwise i.i.d. calibration, the added bound is 0.9-0.9/(n_min+1)-R TV(p,pi_q), where n_min is the smallest class count, pi_q the query prior and R the range of marginal classwise coverages. The no-shift bound requires independent splits.

2. Per-dataset coverage, set size and correction activity.

Correction adds coverage on Caltech101/Pets and efficiency on EuroSAT to the fixed-map gain. The table gives strongest-shift coverage (%) / size over all 41 configurations and 20 seeds. CalOnly fits transport on all calibration data with fixed temperature. Per-split divides calibration data into A for fitting and B for calibration. Active means nonzero correction after both gates and a positive cap.

DatasetConf-OTCalOnlyPer-splitOursActive
Aircraft84.9/12.7089.5/9.8789.5/9.9289.5/9.920/120
Caltech10137.2/0.4884.1/0.8785.6/0.8891.0/0.94119/140
DTD79.9/8.7790.4/5.4190.3/5.5090.6/5.4315/140
EuroSAT82.3/3.7090.1/2.5390.3/2.5194.0/1.55129/140
Flowers10275.5/4.3588.8/1.9789.0/1.9989.1/1.992/140
Pets58.5/0.9188.7/0.9488.6/0.9491.5/0.93140/140

As the table shows, correction selectively complements the fixed-map gain. At concentrations 1/0.3/0.1, it activates in 388/397/405 of 820 runs and gives 89.7%/90.2%/91.0% coverage. Calibration-only fitting removes query dependence from the map. Selective correction then uses established BBSE and weighted calibration.

Fallback avoids large coverage losses on Aircraft under the fixed estimator budget. We use ViT-B/32 and the original 80 runs (four settings, 20 seeds). Three-shift mean coverage (%)/size is 90.18/20.01 for Ours/Per-split, 52.89/7.22 for BBSE, 40.11/5.28 for EM-TS and 42.32/5.14 for EM-BCTS. The paired Ours-minus-EM-BCTS coverage gap is 47.86 points [95% CI 45.12,50.77]. Both EM variants reach the unchanged 1,000-step cap in 79/80 runs.

3. Does the failure persist from T=3 to convergence?

Convergence strengthens the pooled coverage loss across the 41 configurations, three shifts and 20 seeds. Coverage at concentrations 1/0.3/0.1 changes from 85.5%/79.6%/69.3% at T=3 to 83.4%/76.4%/66.6%. At the strongest shift, the paired change is -2.76 points [-3.22,-2.27]. None of the 123 configuration-level Conf-OT-minus-SCP differences changes sign.

The fitted coverage-loss coefficient increases: across six backbones with K=100/200, the convergence/cap endpoint is 1.32–1.58 times the T=3 value. The median ratio of fitted to theoretical coefficients changes from 0.20 to 0.28, with rank correlation 0.76. The median fitted exponent changes from 1.96 to 1.52, using 1,153/1,195 positive-loss points out of 1,200. Three ladder runs reach 5,000 iterations with residuals at most 4e-7, while all main-grid runs converge. These fits show persistence beyond T=3, while the theorem gives the local class-symmetric leading term.

4. Distinct queries and configuration-level coverage distributions.

The no-replacement test confirms useful correction and pooled undercoverage on independent CIFAR-100. Across 20 seeds with 50 calibration images/class and 1,000 distinct queries at concentration 0.1, Ours activates in 20/20 runs and reaches 91.75%/1.50 versus Per-split's 89.76%/3.35. Paired gains are +2.00 coverage points [0.99,3.24] and -1.85 labels [-1.96,-1.75]. Conf-OT remains 7.50 points below SCP [5.97,9.14].

A fixed five-alpha curve further gives 1.35/1.34/3.45 labels for Ours/EM-BCTS/Per-split at 90% observed coverage by descriptive interpolation. This supports compact sets when correction activates, complementing the Aircraft fallback comparison.

Across the original 123 shifted configuration means, minimum/5th-percentile/median coverage is 85.3%/87.5%/90.1% for Ours versus 83.8%/84.9%/89.4% for Per-split. Counts below 85%/88%/90% are 0/8/56 versus 9/20/91, showing the improvement beyond aggregate means.

5. A figure explaining the mechanism and coverage effect.

Our new schematic links query-frequency change to the pooled constraint, shared bias and coverage loss. Adjacent curves for all five methods show how separate fitting and correction affect coverage across shifts.

Reviewer_rFTXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies conformal prediction, which gives zero-shot vision-language models prediction sets with a guaranteed probability of containing the true label. This paper investigates this under label shift, specifically focusing on the failure mode of methods like Conf-OT. The authors demonstrate that under label shift, the pooled constraint induces a class-specific logic bias that penalizes classes gaining frequency in the query set.

Critical strengths
  • The paper is clearly comprehensive and dense. For a technical audience with prior knowledge of the literature, the paper should offer valuable insight.
  • Both empirical experiments and theoretical components are comprehensive.
Critical weaknesses
  • As a researcher in the vision-language space, this is my first time hearing the phrase "conformal prediction". To me, this paper is hard to understand. There is clearly a lot of hard work in both the main paper and the Appendix, but it is not written to be accessible by a general ML audience. I'll give some examples:
    • The abstract jumps right into Conf-OT. It is not clear whether this is existing work or the authors' own work. A Google search yields this prior work: "Conformal Prediction for Zero-Shot Models". I think the authors are writing under the assumption that the reader has already read that paper.
    • There's some jargon that is not defined that I think is critical to understanding the results. What is "shift strength"? Is this simulated or a natural label shift?
Questions for authors

Elaborating further on the "weaknesses" section, the only reason I rate this paper 3 not 4 is the lack of clarity to a general ML audience. I would suggest the following clarifications in the next draft:

  • The lead in to the introduction could be gentler. Please answer the question, as a general vision-language researcher, why should I care about conformal prediction? You can reference the introduction of "Conformal Prediction for Zero-Shot Models". When I read the first part of introduction, I'm trying to understand what the problem setting is, is it image classification, image captioning, segmentation, etc? The authors go directly into talking about "prediction sets" and "coverage" before establishing what task they're talking about.
  • The intro is cluttered with jargon that is hard to understand, such as "pooled class probabilities", "split conformal prediction".
  • For the method section, the authors dive directly into dense notation without explaining the problem setting. What is the shape of the input data? The notation is not clear, one example is Pc(X∣Y)=Pq(X∣Y)P_c(X|Y) = P_q(X|Y). The authors have yet to define any of these symbols. This issue persists through out the theoretical sections.
  • In general, I think it's a good idea, after you state a theorem , to provide an intuition, or practical impact, or even why do we want to prove this in the first place? I do not think the authors include this info.
  • The algorithm is not precise enough. The point of an algorithm, in my opinion, is to inform the user how to code the process. Therefore, it is essential that every word is precisely defined. Here, it is unclear what "calibration" is, or what "split" means, or what "fit Sinkhorn scaling" means, what is the shape of A and B etc.?
  • Finally, I'm unclear what label shift means here. The datasets used here are widely used image classification datasets that are well balanced. This leads me to infer that the authors are synthetically inducing a label shift. This cannot be representative of real-world label-shift scenarios.
2 comments

Thank you for your helpful suggestions. We clarify the task, notation and algorithm below, starting with image classification and tracing the steps from logits to prediction sets. This order explains the practical lesson: calibration-only fitting removes the query dependence responsible for pooled coverage loss.

1. The task, why prediction sets matter, and the prior Conf-OT paper.

The task is zero-shot image classification: CLIP compares an image with descriptions of K candidate classes. Because a single label hides ambiguity, a prediction set returns plausible alternatives, such as {cat, fox}. Coverage measures how often the set contains the true class. The goal is therefore small sets at a specified coverage level, such as 90%.

To construct such sets, conformal prediction uses labeled calibration examples to select a score threshold. Split conformal prediction (SCP) calibrates a previously fixed score function. For label y, our score is 1-p_y(x), where p_y(x) is its predicted probability. Labels with scores at or below the threshold enter the set. Exchangeable calibration and test scores give the usual finite-sample coverage statement.

Conf-OT, the existing method from Conformal Prediction for Zero-Shot Models, adapts scores using unlabeled queries to obtain smaller sets. We now introduce and cite it before asking how coverage changes when those queries have different class frequencies.

2. What the distributions and “pooled class probabilities” mean.

X is an image, Y its label, and P_c/P_q are the calibration/query distributions. Label shift means P_c(X|Y)=P_q(X|Y): images within each class follow the same distribution, while class frequencies P_c(Y) and P_q(Y) may differ. For example, a product-recognition service may encounter more images of one product during a promotion.

The frozen model supplies logit matrices of shapes N×K for N calibration images and M×K for M queries. Each row contains one image's K scores. A “pooled class probability” averages that class's probability over all N+M images. Conf-OT constrains these averages toward calibration class frequencies. Consequently, changing the query mixture can change every image's fitted scores.

3. What “split,” “calibration,” and “fit Sinkhorn scaling” mean in the algorithm.

The algorithm separates fitting the score map from constructing prediction sets. Split labeled calibration examples within each class into disjoint subsets A and B, with logit matrices N_A×K and N_B×K. A fits the map. B supplies class-conditional prediction averages and final calibration scores. The vision-language model stays frozen.

To fit Sinkhorn scaling, exponentiate A's logits into a K×N_A matrix with classes as rows. Alternate row scaling toward A's class proportions and column scaling toward equal image mass for three iterations. The K class factors r define the bias log r. A also fits a scalar temperature. Applying this fixed map produces N_B×K calibration probabilities and M×K query probabilities.

With the map fixed, match mean query probabilities to B's classwise means to estimate query frequencies. When shift is detected and estimation is stable, apply a shrunk, capped prior correction and use weighted calibration on B. Otherwise, use the ordinary B-calibrated threshold. Thus A fixes the map, while B and unlabeled queries determine the correction and threshold.

4. Why the theoretical results matter.

The theory identifies query-induced logit bias as the mechanism and motivates fitting the map on calibration data alone. The shared-bias lemma shows how pooled fitting alters every image's scores, including at three iterations. At convergence, the population sign theorem explains why enforcing old frequencies suppresses newly common classes. Because these classes receive more weight in query coverage, the loss is locally quadratic under class symmetry. Accordingly, calibration-only fitting removes the map's dependence on query frequencies. The prior-error result then motivates selecting correction according to estimation reliability.

5. Shift strength, simulation and real-world relevance.

The main label shifts are simulated to isolate frequency changes while preserving class-conditional image distributions. Dirichlet concentration beta controls query proportions: smaller beta produces more uneven proportions on average. No shift uses the unresampled query pool. We also record total-variation distance, half the sum of absolute differences between calibration and query proportions.

This design tests frequency changes like the product example. Long-tailed ImageNet tests a different imbalance profile. ImageNet-V2/A/R additionally change appearance within classes, distinguishing natural shifts from label shift. Our schematic and coverage curves connect the assumption, pooled-bias mechanism and measured effect.

Thanks you for the detailed rebuttal.

I think it clarifies the key concepts quite a bit.

I think my assessment is fair based on the following points:

  • The revised manuscript remains difficult to read for a general ML audience.
  • My concern regarding unrealistic synthetic label shift remains unaddressed. As the authors acknowledge, the class-conditional image distribution generally changes in realistic deployment settings alongside class proportions (as seen on ImageNet-V2, ImageNet-R, and ImageNet-A in Sec. 5.6 and Table 30). This is acknowledged as a limitation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.