acceptodds
Under review as a conference paper at ICLR 2027

ReProCP: Refined Prototypical Conformal Prediction

Abstract

Conformal prediction has become a popular framework for distribution-free uncertainty quantification, but its effectiveness in low-data regimes is limited by the data splitting required for model fitting and calibration. Full conformal prediction avoids this split by reusing all labeled examples but typically incurs the prohibitive cost of repeatedly refitting the model. We introduce ReProCP (Refined Prototypical Conformal Prediction), an efficient full conformal method built on pretrained self-supervised vision models. Our key insight is that self-supervised embeddings provide not only a strong foundation for accurate transfer learning, but also an effective representation for uncertainty quantification. Operating directly in this embedding space, ReProCP employs a training-free, prototype-based nonconformity score that can be efficiently evaluated across candidate labels, eliminating repeated model fitting. ReProCP further leverages unlabeled images to refine the embedding geometry through dimensionality reduction, whitening, and neighborhood smoothing, yielding more compact prediction sets without additional annotations. Across diverse image-classification datasets, self-supervised backbones, and labeling budgets, ReProCP consistently produces smaller prediction sets than competing conformal baselines, with increasingly pronounced gains as the labeling budget decreases.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Conformal prediction, self-supervised learning, uncertainty quantification, few-shot learning, prototype-based classification, computer vision

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_E53FOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes the method ReProCP. The method works in two steps: First an embedding space is constructed by embedding unlabelled datapoints using a pre-trained embedding model, then performing smoothing, whitening, and dimensionality reduction. In this enhanced embedding space all labeled datapoints are used to establish prototypical class centers. A query point’s non-conformation score can then be evaluated as the cosine similarity to these prototypical class centers. Because a labeled datapoint can be removed from the prototypical class centers trivially, full conformal prediction can be computed efficiently. Their method is simple (which is a good thing) and well aligned for the task. The authors compare ReProCP to baselines on three datasets and find their method outperforms baselines tested. Ablation testing is performed on the the embedding space construction.

Critical strengths
  1. The author’s proposed conformal prediction method is well motivated and elegant for full conformal prediction.
  2. The proposed method performs well compared to baselines and achieve lower average label sets in label scarce settings.
  3. The paper is well written and easy to understand.
Critical weaknesses
  1. The methodological novelty is in general insufficiently distinguished from existing components, as many components are well tested well tested in split-conformal prediction. Some of the stated novelty claims have already been investigated elsewhere, for example “We develop a label-free representation refinement based on dimensionality reduction, whitening, and neighborhood smoothing, with theoretical analysis addressing its validity and benefit.” These components have already been established in [1] in a labeled setting and [3] in an unlabeled setting. In addition, the usage of pre-trained models from self-supervised embeddings has also been explored by [2, 3, 4]. Due to the substantial overlap with existing methods, the Related Work section should be considerably expanded, and differences to these methods should be contrasted to the proposed methods. These works include: [1]: CPSC combines conformal prediction with class centroids, within-class feature standardization, suppression of uninformative features, and temperature scaling. [2]: Uses unsupervised smoothing through similarity-weighted interpolation of neural embeddings also on unlabeled datapoints. [3]: Performs smoothing of non-conformity scores in embedding space using the same DinoV2 backbone on unlabelled datapoints [4]: Uses distance to class prototypes in an embedding space to perform label transduction on unlabeled data for conformal prediction

  2. The paper investigates their setup on only 3 datasets CIFAR-10, CIFAR-100, and EuroSAT, all datasets that are low dimensional, in the image modality, and that their chosen embedding model DinoV2 presumably are very accurate at separating in embedding space. Testing on a more realistic and challenging dataset such as ImageNet would be insightful, and further strengthen the authors claims of efficient full conformal prediction. Investigation and documentation of performance when the embedding models accuracy is reduced would be beneficial.

References:

[1]: Liu et al., “CPSC: Conformal Prediction With Shrunken Centroids…”, IEEE Transactions on Instrumentation and Measurement, 2022

[2]: Rodríguez et al., “Embedding Propagation: Smoother Manifold for Few-Shot Classification” — ECCV 2020

[3]: Weikop et al. Improving Conformal Prediction Sets Through Semantic Neighborhood Diffusion. In ICML 2026 Workshop on Structured Probabilistic Inference & Generative Modeling.

[4]: Silva-Rodríguez et al. "Conformal prediction for zero-shot models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2025.

Questions for authors
  1. Can the authors be exactly specify which components of their method that are novel?
  2. Could there be problems with representing an entire class with a single representative sample?
  3. What is the motivation / intuition for the temperature selection scheme. What is the reason for the median and the factor 1/4? How are we in general to select the hyperparameters ReProCP while mentioning marginal coverage guarantees?
  4. How are we to interpret for Proposition 2? As far as I understand the proof simply uses the triangle inequality to show that the distance to the reference center is reduced from smoothing, i.e. smoothing is contracting the embedding space. From this it does not mean that the conformal properties improves. A datapoint could just as well be moved even closer to another reference center, resulting in a reduction in distance to the true-class center need not improve the classification margin or reduce expected prediction-set size. Can the authors establish a connection to either quantity?
  5. Could the authors, in addition to mean set size, also report empty-set size frequency before adding datapoints, and frequency of sets with a single datapoints in them? Smoothing predictions is know to change the distribution of set sizes in addition to reducing them.
  6. How was the lambda value for the baselines in line 753 selected?
  7. Could the authors evaluate split CP and CV+ using the same refined embeddings and, where applicable, the same prototype classifier? The current comparison changes the representation, classifier, and conformal procedure simultaneously. Such controls would help identify which changes explain the performance gains.
5 comments
AuthorsOpenReview ↗

Response to Reviewer E53F – Part 1/2

Thank you for your helpful and constructive review.

On the novelty of the method compared to previous works. We agree that certain components of the proposed method are connected to stages that are performed in previous work. However, there are major differences in the settings and the overall methodology, as we clarify below.

CPSC [1]: The paper studies classification of electronic-nose measurements, and does not target pretrained vision representations. In terms of methodology, they use class centroids and shrink class-specific deviations toward the overall mean. This is different from the proposed refinement procedure and the prototype-based score. Also, they do not specifically present a leave-one-out score computation.

Embedding Propagation [2]: This paper studies few-shot recognition of novel classes, using labeled base classes for representation learning. It constructs a similarity graph over embeddings and applies a propagation operator. Our method differs in the particular score construction and in the integration with conformal scoring.

Score diffusion [3]: They consider standard split CP, assuming sufficient data is available for both tuning and calibration. The general idea of score smoothing was studied in other works as well (Zargarbashi et al., 2023; Song et al., 2024). We draw inspiration from these works and perform smoothing over embeddings rather than scores, as part of our refinement procedure.

Conf-OT [4]: It starts with zero-shot class predictions from a pretrained model, plus labeled calibration observations and unlabeled queries. It addresses mismatch between pretrained predictions and the target task through entropy-regularized optimal transport. Conf-OT can operate on class logits, without requiring the full image-feature geometry. Its principal adaptation changes probability assignments, rather than embeddings or labeled visual prototypes. Moreover, its category prototypes originate from the zero-shot classifier, such as text embeddings. In addition, it jointly processes calibration and queries, rather than fitting refinement on a separate pool. To summarize, Conf-OT adjusts assignments to existing semantic categories, while ReProCP estimates visual class prototypes from scarce labels.

Harder datasets and weaker embeddings. We added two datasets (Appendix C.1, Tables 3–4): mini-ImageNet (100 ImageNet classes) and PathMNIST, a colon-pathology dataset from MedMNIST that is far from the backbones’ pretraining data. ReProCP gives the smallest sets at every budget on mini-ImageNet and on PathMNIST with DINOv2.

Mean set size (coverage), α=0.1\alpha = 0.1.

mini-ImageNet (K=100K = 100, DINOv2 ViT-B)

\text{Method} & \text{2 shots} & \text{4 shots} & \text{8 shots} \\\\ \hline \text{Split CP (best)} & 92.59\ (0.93) & 2.73\ (0.94) & 1.10\ (0.94) \\\\ \text{CV+} & 2.28\ (0.94) & 1.12\ (0.93) & 1.06\ (0.94) \\\\ \text{SemiCP (best)} & 76.16\ (0.95) & 1.45\ (0.92) & 1.06\ (0.93) \\\\ \textbf{ReProCP} & \mathbf{1.25\ (0.95)} & \mathbf{1.06\ (0.94)} & \mathbf{1.04\ (0.94)} \end{array}$$ PathMNIST ($K = 9$, DINOv2 ViT-B) $$\begin{array}{lcccc} \text{Method} & \text{2 shots} & \text{4 shots} & \text{6 shots} & \text{8 shots} \\\\ \hline \text{CV+} & 7.70\ (1.00) & 3.76\ (0.93) & 2.95\ (0.94) & 2.46\ (0.95) \\\\ \text{SemiCP (best)} & 9.00\ (1.00) & 5.86\ (0.92) & 4.99\ (0.93) & 3.05\ (0.91) \\\\ \textbf{ReProCP} & \mathbf{5.34\ (0.97)} & \mathbf{2.80\ (0.92)} & \mathbf{2.26\ (0.90)} & \mathbf{2.10\ (0.90)} \end{array}$$ **Single-prototype representation.** A single sample as a representative mean can under-represent a class, affecting only set size efficiency, not coverage, which holds for any nonconformity score. Our refinement aims to make embeddings more suitable for prototype-based scoring by reducing nuisance and local variability (Sections 4.1 and 4.3). Each example is scored against a leave-one-out prototype built from the other $s-1$ examples of its class. At least two labels per class are therefore needed. Using only one label, the leave-one-out prototype is empty and the resulting scores are uninformative. **Temperature selection.** The temperature $T$ controls the sharpness of the softmax score. Since cosine margins vary across datasets and encoders, we estimate their scale using unlabeled pool embeddings and the k-means groups from the whitening stage as class proxies. We set $T=|\operatorname{median}(\text{margin})|/4$: the median is robust to outliers and hub points, while the fixed factor 1/4 maps a typical margin to a logit gap of 4. This factor is shared across datasets and encoders. Figure 3 shows that the selected temperatures lie in relatively flat regions of the sensitivity curves.
AuthorsOpenReview ↗

Response to Reviewer E53F – Part 2/2

Other hyperparameters. The structural hyperparameters were selected during development and then fixed across all datasets, encoders, and label budgets, based on the role of each stage. The same applies to the baselines’ regularization λ=10−2\lambda = 10^{-2}. In a sensitivity check over λ∈{10−3,10−2,10−1}\lambda \in \{10^{-3}, 10^{-2}, 10^{-1}\} on CIFAR-100, the ranking against ReProCP did not change.

On the theoretical connection between geometric refinement and set size. We agree that reducing the distance to the true-class center does not by itself establish an improved classification margin. To connect the geometric result to prediction-set efficiency, we added Proposition 3 (Section 4.3, proof in Appendix A.2). Intuitively, as smoothing moves an embedding closer to its true-class center, it increases the lower bound on its cosine margin over each rival class. This gives a sufficient condition for excluding rival labels and yields a non-increasing upper bound on prediction-set size as the distance decreases. In this way, Proposition 3 connects the geometric contraction in Proposition 2 to a potential reduction in prediction-set size.

Empty and singleton prediction-set frequencies. We now report empty-set and singleton-set frequencies before applying the non-empty-set convention (Appendix C.6, Table 10). Empty sets occur mainly on CIFAR-10 (4.2–8.3%), where the mean set size is close to one. They are rare on CIFAR-100 (≤1.3%) and absent on EuroSAT. Singleton sets account for 92–94% of predictions on CIFAR-10, and their frequency increases with the label budget on CIFAR-100 (47% to 75%) and EuroSAT (23% to 62%).

Split CP and CV+ on refined embeddings. We evaluated split CP and CV+ on the same refined embeddings used by ReProCP (Appendix C.3, Table 6). ReProCP gives the smallest or tied-smallest sets in all 12 dataset–budget cells, including every cell on CIFAR-10 and EuroSAT and every cell at two labels per class (e.g., CIFAR-100: 2.43 versus 3.47 for refined CV+). On CIFAR-100, refinement also reduces the baselines’ sets, and refined CV+ approaches ReProCP from four labels per class onward, which shows that the refinement is beneficial beyond our score and can improve other conformal methods as well. The remaining gap at low budgets reflects the need to split the scarce labels between fitting and calibration, which refinement alone does not remove.

I appreciate the authors’ time in addressing some of the concerns raised especially through new experiments.

Novelty

While I appreciate the more detailed separation from earlier work, I do still see only limited novelty in most of the proposed setup as smoothing over some latent representation has already been widely studiet as also pointed out by reviewer naXC. I, however, do see some novelty in the fast full-conformal prediction setup formulation of the method.

Harder datasets and weaker embeddings

While the investigation to harder datasets such as mini-ImageNet and a comparison with CLIP ViT-B improve the proposed methods usefulness for more difficult problems, half of the initial question largely remains unaddressed. Namely, how does the method scale as the competency of the embedding model changes? CLIP and DINO pertaining is quite different, and is confounding the embedding competency. An ablation that would be quite clear would be the performance of ReProCP on a fixed 2 or 4 shot budget with DINOv2 ViT-S, ViT-B, ViT-L, and ViT-G would show this quite clearly on a fixed dataset. The shared family allows for a direct comparison of conformal performance versus the downstream accuracy of the embedding model.

Single-prototype representation

I’m not questioning if the conformal guarantees are satisfied by ReProCP. My main concern is the potential set size increase that can arise from ReProCP particular combination of prototype clustering on embeddings. If a particular class has two modes in the embedding centered around aa and bb with the fraction pp of labeled points around aa and 1−p1-p around bb, then the leave-one-out mean prototype per class is approximately μc≈pa+(1−p)b\mu_c\approx pa+(1-p)b, which could be outside of the high density area of aa and bb. Local smoothing may tighten each mode separately without resolving this mismatch with the single-prototype score. This is also not a hypothetical problem, as more complex datasets such as ImageNet (full size) can often shows this behavior. An example could be images of dogs inside a building and dogs out in the nature. This problem is also not necessarily fixed by a more competent embedding model when trained in an unsupervised fashion, as the two types of images are indeed semantically different. This concern is slightly addressed by the inclusion of the mini-ImageNet dataset, but remains largely un-addressed.

Hyperparameter selection

I apologize that the reasoning for my initial question was quite indirect. As also raised by Reviewer naXC, I remain unconvinced by the current hyperparameter selection and motivation. My main concern is the paper is introducing many hyperparameters such as C’C’, κ\kappa, and γ\gamma, which to me seems would be selected based on some knowledge of the labels. Can the authors provide a principled alternative way? This is a normal problem in conformal prediction, and is the reason many methods require a tuning set, as to avoid breaking the exchangeability assumption by selecting hyperparameters based on the calibration / test set. This is, however, something ReProCP explicitly described as being novel for not needing. I am curious what the authors suggest for selecting these parameters in a practical setting on a new dataset without giving up on exchangeability by selecting them with label information. ReProCP seems to be sensitive to this choice of hyperparameters based on figure 4, especially as the datasets become challenging for the embedding model. How does the impact of the hyperparameters and temperature vary across the two new datasets tested (mini-ImageNet, and PathMNIST)?

Overall

Again, I appreciate the thorough answer from the authors, especially the theoretical extension, but for me to raise my score I would like to see two remaining parts addressed:

  1. More thorough evaluation of the impact of the backbone model. A single experiment where the dataset is sufficiently hard (CIFAR100) with a varying backbone of same family such as DINOv2 ViT-S, ViT-B, ViT-L, and ViT-G (keeping the projected dimension fixed is fine between models).
  2. Stronger hyperparameter motivation. The transferability of the chosen hyperparameters should be shown on the new datasets (mini-ImageNet and PathMNIST), by reproducing figure 3 and 4. Fewer seeds is acceptable in the interest of time and to save computation. In addition, the way hyperparameters was chosen should be added to the paper.

I recognize the limited remaining discussion time when asking for follow-up and clarification of experiments, but your response was first provided today.

Follow up question

I noticed that an explicit normalization step was added to the manuscript after your initial response. Was this step already implicit the initial experiments, but omitted from the original description, or was it introduced subsequently? If the implementation changed, please clarify which results were recomputed and whether any reported metrics changed.

Reviewer_LUS2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReProCP combines prototype-based full conformal prediction with PCA truncation, within-group whitening, and neighborhood smoothing fitted on an independent unlabeled pool. Cached prototype updates accelerate inference. The paper establishes exchangeability preservation under its assumptions and a center-distance improvement condition, and evaluates prediction-set efficiency on three datasets using DINOv2 and CLIP.

Critical strengths
  1. At two shots on DINOv2/CIFAR-100, mean set size decreases from CV+’s 8.34 to 2.43, with coverage 0.940 at a 0.900 target. Figure 2 and Table 6 provide useful refinement and SNAPS comparisons.
  2. After precomputation, the cached implementation processes 500 CIFAR-100 examples in 0.44–0.63 seconds, substantially faster than its unvectorized full-CP counterpart.
Critical weaknesses
  1. Prototype-based full CP [1], training-free full CP on frozen features [2], and exact-update acceleration [3] already exist. Related work also omits relevant unlabeled adaptation/refinement methods [4,5]. The identifiable increment is the particular independent-pool refinement pipeline and its score-specific integration. Although the ablations are useful, the main comparisons change representations, classification heads, conformal procedures, and unlabeled-data access together. Proposition 2 does not resolve this attribution gap, where center-distance contraction does not imply improved cosine-softmax margins or conformal ranks. Superiority is also not uniform. On CLIP/EuroSAT at eight shots, ReProCP produces larger sets than CV+ (2.36 versus 1.78).

  2. The guarantee and accelerated computation are not reconciled with the evaluated procedure. Fixed-ss-per-class sampling is not jointly exchangeable with an added test example: its label necessarily identifies the unique class with s+1s+1 observations, whereas exchangeability would assign that event conditional probability (s+1)/(sK+1)(s+1)/(sK+1). Proposition 1 therefore does not establish coverage for this protocol. Separately, Eq. (12) generally produces non-unit embeddings, while Eq. (21) requires unit norms. Post-smoothing normalization would change prototype averaging and the representation analyzed by Proposition 2. The paper must specify a consistent algorithm and establish which guarantee applies to its evaluation.

[1] Fisch, Schuster, Jaakkola, and Barzilay. Few-Shot Conformal Prediction with Auxiliary Tasks. ICML, 2021.

[2] Silva-Rodríguez, Fillioux, Cournède, Vakalopoulou, Christodoulidis, Ben Ayed, and Dolz. Full Conformal Adaptation of Medical Vision-Language Models. IPMI, 2025.

[3] Cherubin, Chatzikokolakis, and Jaggi. Exact Optimization of Conformal Predictors via Incremental and Decremental Learning. ICML, 2021.

[4] Silva-Rodríguez, Ben Ayed, and Dolz. Trustworthy Few-Shot Transfer of Medical VLMs through Split Conformal Prediction. MICCAI, 2025.

[5] Bozorgtabar, Mahapatra, Roy, Naseer, Razzak, and Ge. LATA: Laplacian-Assisted Transductive Adaptation for Conformal Uncertainty in Medical VLMs. CVPR, 2026.

Questions for authors
  1. What coverage guarantee applies to the fixed-ss-per-class protocol, beyond Proposition 1’s exchangeability assumption?
  2. Are embeddings normalized after smoothing, and do cached p-values match direct evaluation of Eqs. (13)–(14) under the same convention?
  3. What advantage remains over FCA on CLIP and Split CP/CV+ using the same refined features? How frequently do baseline training splits omit classes?
  4. Why does Figure 5 report mean set size 0.99 despite non-empty-set postprocessing, and how does its single threshold represent candidate-dependent full CP?
6 comments
AuthorsOpenReview ↗

Response to Reviewer LUS2 – Part 1/3

Thank you for your helpful and constructive review.

On the relation to previous works. We agree that there is a strong relation between our method and previous works in the field. However, there are major differences in the considered setting and in the overall methodology, as we clarify below.

Few-shot CP with auxiliary tasks [1]: This paper assumes access to many related labeled auxiliary tasks, in addition to the few examples available for a new target task. It transfers information across tasks to obtain useful prediction sets despite limited target-task data. This is different from our setting where we have only a single task.

FCA [2]: FCA addresses few-shot adaptation of frozen medical vision-language models. It asks how the same limited labels can support both adaptation and conformal prediction. There are several differences compared to ReProCP. FCA incorporates textual class information, while ReProCP can operate with image-only encoders. Accordingly, FCA’s adapter combines textual and visual prototypes, while ReProCP uses visual prototypes in a pool-refined representation. FCA describes fitting a candidate-specific adapter on the augmented sample, while ReProCP explicitly uses leave-one-out prototypes for scoring. Finally, an independent unlabeled refinement pool is central to ReProCP’s representation stage, but is not required by FCA.

Efficient Full CP [3]: This paper addresses the computational expense of full conformal prediction, rather than representation learning or few-shot adaptation. Its proposed optimization procedures inspire our efficient leave-one-out computation for our prototype-based scores.

SCA-T [4]: The paper relies on a pretrained medical VLM, a small labeled calibration set, and unlabeled test inputs. The goal is to adapt the classifier while retaining useful conformal predictions. SCA-T jointly adapts on calibration and test embeddings. It optimizes class-weight vectors, initialized from text prototypes, using a transductive information-based objective. While SCA-T changes the classification head using the current calibration–test batch, ReProCP changes the representation using a separate unlabelled pool, then uses labeled prototypes in full CP. SCA-T does not perform a separate adaptation for each candidate.

LATA [5]: It also studies medical VLMs with a small calibration set and a joint calibration-test pool. It builds an image-similarity graph and refines class-probability vectors, rather than image embeddings. It also incorporates a frozen ViLU module into the conformal score, using estimated prediction difficulty and image–text label plausibility. ViLU is pretrained on separately labeled source data. LATA includes both a label-free adaptation variant and a variant using calibration-label frequencies. While ReProCP’s smoothing changes geometry before prototype construction, LATA changes predictions after the zero-shot classifier has produced probabilities. Its graph spans calibration and test inputs, whereas our refinement uses a separate reference pool.

On the theoretical connection between geometric refinement and set size. We agree that reducing the distance to the true-class center does not by itself establish an improved classification margin. To connect the geometric result to prediction-set efficiency, we added Proposition 3 (Section 4.3, proof in Appendix A.2). Intuitively, as smoothing moves an embedding closer to its true-class center, it increases the lower bound on its cosine margin over each rival class. This gives a sufficient condition for excluding rival labels and yields a non-increasing upper bound on prediction-set size as the distance decreases. In this way, Proposition 3 connects the geometric contraction in Proposition 2 to a potential reduction in prediction-set size.

AuthorsOpenReview ↗

Response to Reviewer LUS2 – Part 2/3

Harder datasets and weaker embeddings. We added two datasets (Appendix C.1, Tables 3–4): mini-ImageNet (100 ImageNet classes) and PathMNIST, a colon-pathology dataset from MedMNIST that is far from the backbones’ pretraining data. ReProCP gives the smallest sets at every budget on mini-ImageNet and on PathMNIST with DINOv2.

Mean set size (coverage), α=0.1\alpha = 0.1.

mini-ImageNet (K=100K = 100, DINOv2 ViT-B)

undefined
AuthorsOpenReview ↗

Response to Reviewer LUS2 – Part 3/3

Advantage over FCA and other baselines. We do not target text-embedding-based adaptation, which is central to FCA. Instead, we view FCA as conceptually related to our unrefined baseline: full conformal prediction with the prototype score on raw embeddings, before feature refinement. This baseline isolates the contribution of refinement. We also include this ablation for CLIP ViT-B in the appendix (Appendix C.7, Figure 6).

As for other baselines, we further evaluated split CP and CV+ on the same refined embeddings used by ReProCP (Appendix C.3, Table 6). Refinement helps the baselines on CIFAR-100, where refined CV+ approaches ReProCP from four labels per class onward, but enlarges their sets on CIFAR-10 and EuroSAT. ReProCP still gives the smallest sets at two labels per class on every dataset (e.g., CIFAR-100: 2.43 versus 3.47 for refined CV+), since refinement does not remove the need to split the scarce labels between fitting and calibration.

Regarding class omission, our main experiments use balanced splits, so every class is represented in the labeled set. This is one reason we adopt this protocol, which is also common in few-shot conformal prediction.

Figure 5 and non-empty prediction sets. We thank the reviewer for catching this inconsistency. Figure 5 (now Figure 7, Appendix C.9) indeed displayed the results without the non-empty-set convention, allowing some empty sets and resulting in a mean set size of 0.9865. We have corrected the figure using the same non-empty-set convention as in the rest of the experiments.

Reviewer_naXCOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ReProCP, a training-free full conformal prediction method for few-shot image classification using frozen self-supervised model embeddings. It scores examples based on their similarity to leave-one-out class prototypes, making full conformal prediction computationally cheap and allowing all labeled examples to be used without a data split. Experiments on CIFAR-10, CIFAR-100, and EuroSAT with DINOv2 and CLIP backbones show smaller prediction sets than the split conformal, CV+, and SemiCP baselines, especially when there are very few labels per class.

Critical strengths
  1. The paper addresses a practical setting in which labels are too scarce to split between training and calibration.
  2. The method is simple to implement and fast at test time, and coverage remains valid because the refinement is fitted only on an independent unlabeled pool.
  3. The gains at very low labeling budgets are large and consistent across datasets and two backbones.
Critical weaknesses
  1. The paper's main insight is that pre-trained embeddings are a good space in which to quantify uncertainty. This was already put forward in [1], which argued that conformal prediction can be carried out in a learned semantic feature space instead of the output space and showed that this can yield smaller prediction sets. The authors do not cite this work. Can the authors discuss it and position their contribution more precisely? As I understand it, the novelty lies in the leave-one-out prototype score that makes full conformal prediction cheap, together with the "pool-fitted representation refinement" on the unlabeled pool, rather than in the idea of performing conformal inference in latent space.
  2. The method is over-engineered. The refinement involves multiple post-processing tricks, each with its own hand-picked setting and an ad-hoc justification, including a temperature rule with an arbitrary scaling factor. None of this follows from a principled objective.
  3. The authors do not compare against few-shot methods that also exploit unlabeled data, such as label propagation or LaplacianShot, or even simply clustering the pool and labeling the clusters, each wrapped in conformal prediction.

References

  1. Teng, Jiaye, et al. "Predictive Inference with Feature Conformal Prediction." In ICLR, 2023.
Questions for authors

In addition to the highlighted weaknesses, I ask the authors to answer the following questions:

  1. Could the authors test their approach with different vision foundation models, e.g., DINOv3, SigLIP 1/2, etc.?
  2. How does the method perform on datasets with many classes, such as ImageNet (1000 classes)?
  3. Full Conformal Adaptation (FCA; [1]) also reuses the same few labels for both adaptation and full conformal inference, using a training-free adapter. Why is it not included as a baseline, at least in the CLIP experiments?

References

  1. Silva-Rodríguez, Julio, et al. "Full conformal adaptation of medical vision-language models." In IPMI, 2025.
3 comments
AuthorsOpenReview ↗

Response to Reviewer naXC Part 1/2

Thank you for your helpful and constructive review.

On the relation to Predictive Inference with Feature Conformal Prediction. FeatureCP studies how a trained neural network's internal representations can improve conformal uncertainty estimates. It is not specifically designed for settings with extremely scarce target labels. Methodologically, it constructs prediction regions in the feature space and maps them back to the output space.

ReProCP also utilizes internal features for conformal inference, but differs in both the setting and the methodology. We consider settings with very few labeled examples, where a fully trained model is unavailable. Instead, we use a pretrained self-supervised model and measure compatibility with class prototypes estimated from the available labels, after applying a refinement procedure fitted on unlabeled data. ReProCP also uses full conformal prediction rather than split conformal prediction.

We have added the missing citation and clarified these distinctions in the revised manuscript (Section 2).

On the complexity and heuristic choices in the refinement pipeline. The refinement is intended as a label-free adaptation of pretrained embeddings to the available image pool, making the representation more suitable for conformal scoring while preserving all labeled examples for full conformal inference (Sections 1 and 4). This design is motivated by prior evidence that post-hoc feature refinement can improve pretrained representations in image retrieval and few-shot classification (Section 2, Representation refinement). Its contribution is evaluated through the ablation study (Section 5.2, Figure 2). Moreover, the configuration is shared across datasets and encoders. Appendix B.1 reports sensitivity analyses, showing that the selected settings lie in relatively stable regions, including the pool-based temperature selection (Figure 3).

On the lack of comparisons with few-shot methods that exploit unlabeled data. Following your comment, we added a comparison to unlabeled-data baselines (Appendix C.2, Table 5): label propagation and two clustering-based approaches. The first assigns each of KK clusters to a class by majority vote from the labeled examples. The stronger variant uses more clusters and soft class distributions. Each baseline is wrapped in split conformal prediction, using half of the labeled examples for fitting and half for calibration. The comparison also includes the split-CP probe baseline, while SemiCP is already included in the main experiments.

The results below report mean set size (marginal coverage) at α=0.1\alpha = 0.1, for 2, 4, and 8 shots per class.

\text{Dataset} & \text{Method} & \text{2 shots} & \text{4 shots} & \text{8 shots} \\\\ \hline \text{CIFAR-100} & \text{Cluster-and-label } (c=K) & 100.00\ (1.00) & 100.00\ (1.00) & 100.00\ (1.00) \\\\ & \text{Label propagation} & 100.00\ (1.00) & 97.13\ (1.00) & 3.57\ (0.93) \\\\ & \text{Smoothed clustering (best } c) & 70.39\ (0.96) & 41.53\ (0.95) & 3.18\ (0.92) \\\\ & \text{Split CP} & 92.92\ (0.93) & 8.76\ (0.95) & 1.93\ (0.92) \\\\ & \textbf{ReProCP} & \mathbf{2.43\ (0.94)} & \mathbf{1.42\ (0.91)} & \mathbf{1.28\ (0.90)} \\\\ \hline \text{mini-ImageNet} & \text{Cluster-and-label } (c=K) & 100.00\ (1.00) & 100.00\ (1.00) & 42.17\ (0.95) \\\\ & \text{Label propagation} & 100.00\ (1.00) & 60.03\ (0.96) & 1.37\ (0.91) \\\\ & \text{Smoothed clustering (best } c) & 51.25\ (0.95) & 17.75\ (0.95) & 1.39\ (0.92) \\\\ & \text{Split CP} & 92.59\ (0.93) & 2.73\ (0.94) & 1.10\ (0.94) \\\\ & \textbf{ReProCP} & \mathbf{1.25\ (0.95)} & \mathbf{1.06\ (0.94)} & \mathbf{1.04\ (0.94)} \end{array}$$ The additional baselines generally produce very large sets at the lowest budgets, reflecting the limited calibration data available to split-based methods. ReProCP achieves the smallest sets at every budget. Unlike these split-based approaches, ReProCP uses the unlabeled pool for representation refinement while retaining all labeled examples for full conformal inference.
AuthorsOpenReview ↗

Response to Reviewer naXC Part 2/2

On experimenting with more backbones. Following your comment, we added DINOv3 ViT-B results to the backbone-transfer section of the appendix (Appendix C.7, Figure 5), alongside the existing CLIP ViT-B comparison (Table 11). The pipeline and configuration remain unchanged. Only the pool statistics are refitted and the temperature is adjusted accordingly for the new encoder. DINOv3 follows the same trend: ReProCP produces the smallest sets at every budget on both CIFAR-10 and CIFAR-100, with the largest margins at 2–4 labels per class.

Mean set size (coverage), DINOv3 ViT-B, α=0.1\alpha = 0.1:

CIFAR-10 (K=10K = 10)

\text{Method} & \text{2 shots} & \text{4 shots} & \text{6 shots} & \text{8 shots} \\\\ \hline \text{CV+} & 2.48\ (0.96) & 1.10\ (0.93) & 1.03\ (0.95) & 1.01\ (0.94) \\\\ \text{SemiCP (best)} & 4.72\ (0.93) & 1.80\ (0.92) & 1.29\ (0.94) & 1.04\ (0.93) \\\\ \textbf{ReProCP} & \mathbf{1.36\ (0.96)} & \mathbf{1.05\ (0.95)} & \mathbf{1.01\ (0.95)} & \mathbf{1.00\ (0.95)} \end{array}$$ CIFAR-100 ($K = 100$) $$\begin{array}{lcccc} \text{Method} & \text{2 shots} & \text{4 shots} & \text{6 shots} & \text{8 shots} \\\\ \hline \text{CV+} & 13.09\ (0.95) & 3.53\ (0.92) & 2.72\ (0.92) & 2.27\ (0.91) \\\\ \text{SemiCP (best)} & 92.40\ (0.92) & 9.06\ (0.93) & 3.93\ (0.91) & 2.99\ (0.92) \\\\ \textbf{ReProCP} & \mathbf{6.78\ (0.95)} & \mathbf{2.75\ (0.91)} & \mathbf{2.29\ (0.90)} & \mathbf{2.11\ (0.90)} \end{array}$$ Split CP is omitted because its sets are larger than those of all three methods in every cell. **On comparison with FCA.** We do not target text-embedding-based adaptation, which is central to FCA. Instead, we view FCA as conceptually related to our unrefined baseline: full conformal prediction with the prototype score on raw embeddings, before feature refinement. This baseline is included in our ablation study and allows us to isolate the contribution of feature refinement. We have also included the same ablation for CLIP ViT-B in the appendix (Appendix C.7, Figure 6), where refinement reduces the mean set size on CIFAR-10 from 5.09 to 1.66 at 2 labels per class.

I would like to thank the authors for their response and for all the additional experiments. My concern about limited methodological novelty remains, although the new results strengthen the empirical contribution. I am willing to raise my score to weak reject.

Reviewer_afYxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an efficient full conformal prediction method for few-shot image classification. It uses a prototype-based nonconformity score on frozen self-supervised embeddings, avoiding repeated model fitting, and refines the embeddings using unlabeled data to improve prediction-set efficiency.

Critical strengths
  1. Generalize conformal prediction to few-shot with unlabeled data is novel setting.
Critical weaknesses
  1. The performance relies on the quality/properties of the representation.

  2. The performance gains diminish as the labeling budget increases.

Questions for authors

NAN

3 comments
AuthorsOpenReview ↗

Response to Reviewer afYx

Thank you for your helpful and constructive review.

On the dependence on representation quality. We agree that representation quality is an important factor. ReProCP is designed to leverage strong pretrained representations rather than improve the backbone itself. To further examine its robustness across representations and datasets, we added experiments with DINOv3 and two additional datasets, MiniImageNet and PathMNIST (Appendix C.1, Tables 3–4; Appendix C.7, Figure 5).

On diminishing gains with larger labeling budgets. We agree that the relative advantage decreases as the labeling budget grows. This is expected, as additional labeled examples improve competing methods and reduce the need for ReProCP's label-efficient approach. Nevertheless, ReProCP remains competitive across the evaluated budgets (Table 1).

Thanks for the reply.

  1. I agree that the main goal of the paper is not to improve the representation quality but rely and build on it. The representation quality affects the performance and guarantees, the limitation exists.

  2. The gap vanishes at 8 shots (in CIFAR-10/EuroSAT). And In larger setup, linear probe might better than FSL methods. It is make sense to focus on the small shot setup.

I share similar concerns with other reviewers on the novelty of the method.

AuthorsOpenReview ↗

Response to Reviewer afYx

Thank you very much for the follow-up. We would like to address your concerns.

1. Representation quality affects efficiency not coverage guarantee.

Under the exchangeability and independent-pool assumptions of Proposition 1, coverage holds regardless of how well the representation separates the classes. A weaker representation can produce larger, less informative prediction sets, but it does not invalidate the conformal guarantee.

To examine this dependence more directly, we have now added a controlled experiment on CIFAR-100 using DINOv2 ViT-S, ViT-B, ViT-L, and ViT-G, with the same configuration and projected dimension. This experiment quantifies how efficiency changes with backbone strength while holding the remaining procedure fixed. At two labels per class, their mean set sizes are 6.15, 2.43, 1.66, and 1.76, respectively, with empirical coverage remaining stable.

2. Gains at small labeling budgets.

We agree that linear probes become stronger competitors as more labels become available. ReProCP specifically addresses the regime in which allocating scarce labels between model fitting and calibration is costly. Its advantage is therefore expected to diminish as this constraint becomes less severe.

On CIFAR-10, prediction-set sizes also approach the minimum of one, leaving little room for further improvement. Thus, convergence at larger budgets should be considered alongside the substantial gains at smaller budgets. We do not claim that a prototype-based approach must outperform a well-trained classifier when sufficient labeled data are available.

3. Novelty.

Our contribution is the use of an independent unlabeled pool to refine representations for few-shot full-conformal scoring, together with an efficient leave-one-out implementation that retains all labeled examples for inference. We recognize that prototype classification, full conformal prediction, and representation refinement are related to ideas explored in other contexts, but their combination and adaptation to conformal prediction with few labels constitute a new approach that also leads to significant performance gains in the low-data regime.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

We thank all reviewers for their careful reading of the manuscript and for their constructive comments. Their feedback helped us clarify the contribution of ReProCP, strengthen the theoretical discussion, and broaden the empirical evaluation. The main revisions are summarized below.

We clarified ReProCP’s contribution and its relationship to prior work. The revised discussion distinguishes our setting and methodology from related approaches to few-shot conformal prediction and adaptation. In particular, it explains how ReProCP uses an unlabeled pool to refine pretrained image representations, then applies full conformal prediction with prototypes estimated from few labeled examples.

We substantially expanded the empirical evaluation. We added comparisons with methods that use unlabeled data, including label propagation and clustering-based approaches, and evaluated additional datasets and backbones, including MiniImageNet, PathMNIST, and DINOv3. We also compare baseline methods using the same refined representations as ReProCP, allowing us to assess whether the gains come from the refinement procedure or from the conformal method built on top of it.

We strengthened the theoretical and coverage evaluation. We added an analysis connecting the geometric effect of refinement to a sufficient condition for improving prediction-set efficiency. We also evaluated coverage under unstratified random sampling, complementing the balanced-sampling results and examining the method under the exchangeability setting assumed by our guarantee.

We have incorporated these changes into the revised manuscript and appendix.

We are grateful to the reviewers for their comments, which helped us improve the paper’s clarity, theoretical presentation, and empirical support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.