- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes the method ReProCP. The method works in two steps: First an embedding space is constructed by embedding unlabelled datapoints using a pre-trained embedding model, then performing smoothing, whitening, and dimensionality reduction. In this enhanced embedding space all labeled datapoints are used to establish prototypical class centers. A query point’s non-conformation score can then be evaluated as the cosine similarity to these prototypical class centers. Because a labeled datapoint can be removed from the prototypical class centers trivially, full conformal prediction can be computed efficiently. Their method is simple (which is a good thing) and well aligned for the task. The authors compare ReProCP to baselines on three datasets and find their method outperforms baselines tested. Ablation testing is performed on the the embedding space construction.
Critical strengths
- The author’s proposed conformal prediction method is well motivated and elegant for full conformal prediction.
- The proposed method performs well compared to baselines and achieve lower average label sets in label scarce settings.
- The paper is well written and easy to understand.
Critical weaknesses
-
The methodological novelty is in general insufficiently distinguished from existing components, as many components are well tested well tested in split-conformal prediction. Some of the stated novelty claims have already been investigated elsewhere, for example “We develop a label-free representation refinement based on dimensionality reduction, whitening, and neighborhood smoothing, with theoretical analysis addressing its validity and benefit.” These components have already been established in [1] in a labeled setting and [3] in an unlabeled setting. In addition, the usage of pre-trained models from self-supervised embeddings has also been explored by [2, 3, 4]. Due to the substantial overlap with existing methods, the Related Work section should be considerably expanded, and differences to these methods should be contrasted to the proposed methods. These works include: [1]: CPSC combines conformal prediction with class centroids, within-class feature standardization, suppression of uninformative features, and temperature scaling. [2]: Uses unsupervised smoothing through similarity-weighted interpolation of neural embeddings also on unlabeled datapoints. [3]: Performs smoothing of non-conformity scores in embedding space using the same DinoV2 backbone on unlabelled datapoints [4]: Uses distance to class prototypes in an embedding space to perform label transduction on unlabeled data for conformal prediction
-
The paper investigates their setup on only 3 datasets CIFAR-10, CIFAR-100, and EuroSAT, all datasets that are low dimensional, in the image modality, and that their chosen embedding model DinoV2 presumably are very accurate at separating in embedding space. Testing on a more realistic and challenging dataset such as ImageNet would be insightful, and further strengthen the authors claims of efficient full conformal prediction. Investigation and documentation of performance when the embedding models accuracy is reduced would be beneficial.
References:
[1]: Liu et al., “CPSC: Conformal Prediction With Shrunken Centroids…”, IEEE Transactions on Instrumentation and Measurement, 2022
[2]: Rodríguez et al., “Embedding Propagation: Smoother Manifold for Few-Shot Classification” — ECCV 2020
[3]: Weikop et al. Improving Conformal Prediction Sets Through Semantic Neighborhood Diffusion. In ICML 2026 Workshop on Structured Probabilistic Inference & Generative Modeling.
[4]: Silva-Rodríguez et al. "Conformal prediction for zero-shot models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2025.
Questions for authors
- Can the authors be exactly specify which components of their method that are novel?
- Could there be problems with representing an entire class with a single representative sample?
- What is the motivation / intuition for the temperature selection scheme. What is the reason for the median and the factor 1/4? How are we in general to select the hyperparameters ReProCP while mentioning marginal coverage guarantees?
- How are we to interpret for Proposition 2? As far as I understand the proof simply uses the triangle inequality to show that the distance to the reference center is reduced from smoothing, i.e. smoothing is contracting the embedding space. From this it does not mean that the conformal properties improves. A datapoint could just as well be moved even closer to another reference center, resulting in a reduction in distance to the true-class center need not improve the classification margin or reduce expected prediction-set size. Can the authors establish a connection to either quantity?
- Could the authors, in addition to mean set size, also report empty-set size frequency before adding datapoints, and frequency of sets with a single datapoints in them? Smoothing predictions is know to change the distribution of set sizes in addition to reducing them.
- How was the lambda value for the baselines in line 753 selected?
- Could the authors evaluate split CP and CV+ using the same refined embeddings and, where applicable, the same prototype classifier? The current comparison changes the representation, classifier, and conformal procedure simultaneously. Such controls would help identify which changes explain the performance gains.
5 comments
Response to Reviewer E53F – Part 2/2
Other hyperparameters. The structural hyperparameters were selected during development and then fixed across all datasets, encoders, and label budgets, based on the role of each stage. The same applies to the baselines’ regularization . In a sensitivity check over on CIFAR-100, the ranking against ReProCP did not change.
On the theoretical connection between geometric refinement and set size. We agree that reducing the distance to the true-class center does not by itself establish an improved classification margin. To connect the geometric result to prediction-set efficiency, we added Proposition 3 (Section 4.3, proof in Appendix A.2). Intuitively, as smoothing moves an embedding closer to its true-class center, it increases the lower bound on its cosine margin over each rival class. This gives a sufficient condition for excluding rival labels and yields a non-increasing upper bound on prediction-set size as the distance decreases. In this way, Proposition 3 connects the geometric contraction in Proposition 2 to a potential reduction in prediction-set size.
Empty and singleton prediction-set frequencies. We now report empty-set and singleton-set frequencies before applying the non-empty-set convention (Appendix C.6, Table 10). Empty sets occur mainly on CIFAR-10 (4.2–8.3%), where the mean set size is close to one. They are rare on CIFAR-100 (≤1.3%) and absent on EuroSAT. Singleton sets account for 92–94% of predictions on CIFAR-10, and their frequency increases with the label budget on CIFAR-100 (47% to 75%) and EuroSAT (23% to 62%).
Split CP and CV+ on refined embeddings. We evaluated split CP and CV+ on the same refined embeddings used by ReProCP (Appendix C.3, Table 6). ReProCP gives the smallest or tied-smallest sets in all 12 dataset–budget cells, including every cell on CIFAR-10 and EuroSAT and every cell at two labels per class (e.g., CIFAR-100: 2.43 versus 3.47 for refined CV+). On CIFAR-100, refinement also reduces the baselines’ sets, and refined CV+ approaches ReProCP from four labels per class onward, which shows that the refinement is beneficial beyond our score and can improve other conformal methods as well. The remaining gap at low budgets reflects the need to split the scarce labels between fitting and calibration, which refinement alone does not remove.
I appreciate the authors’ time in addressing some of the concerns raised especially through new experiments.
Novelty
While I appreciate the more detailed separation from earlier work, I do still see only limited novelty in most of the proposed setup as smoothing over some latent representation has already been widely studiet as also pointed out by reviewer naXC. I, however, do see some novelty in the fast full-conformal prediction setup formulation of the method.
Harder datasets and weaker embeddings
While the investigation to harder datasets such as mini-ImageNet and a comparison with CLIP ViT-B improve the proposed methods usefulness for more difficult problems, half of the initial question largely remains unaddressed. Namely, how does the method scale as the competency of the embedding model changes? CLIP and DINO pertaining is quite different, and is confounding the embedding competency. An ablation that would be quite clear would be the performance of ReProCP on a fixed 2 or 4 shot budget with DINOv2 ViT-S, ViT-B, ViT-L, and ViT-G would show this quite clearly on a fixed dataset. The shared family allows for a direct comparison of conformal performance versus the downstream accuracy of the embedding model.
Single-prototype representation
I’m not questioning if the conformal guarantees are satisfied by ReProCP. My main concern is the potential set size increase that can arise from ReProCP particular combination of prototype clustering on embeddings. If a particular class has two modes in the embedding centered around and with the fraction of labeled points around and around , then the leave-one-out mean prototype per class is approximately , which could be outside of the high density area of and . Local smoothing may tighten each mode separately without resolving this mismatch with the single-prototype score. This is also not a hypothetical problem, as more complex datasets such as ImageNet (full size) can often shows this behavior. An example could be images of dogs inside a building and dogs out in the nature. This problem is also not necessarily fixed by a more competent embedding model when trained in an unsupervised fashion, as the two types of images are indeed semantically different. This concern is slightly addressed by the inclusion of the mini-ImageNet dataset, but remains largely un-addressed.
Hyperparameter selection
I apologize that the reasoning for my initial question was quite indirect. As also raised by Reviewer naXC, I remain unconvinced by the current hyperparameter selection and motivation. My main concern is the paper is introducing many hyperparameters such as , , and , which to me seems would be selected based on some knowledge of the labels. Can the authors provide a principled alternative way? This is a normal problem in conformal prediction, and is the reason many methods require a tuning set, as to avoid breaking the exchangeability assumption by selecting hyperparameters based on the calibration / test set. This is, however, something ReProCP explicitly described as being novel for not needing. I am curious what the authors suggest for selecting these parameters in a practical setting on a new dataset without giving up on exchangeability by selecting them with label information. ReProCP seems to be sensitive to this choice of hyperparameters based on figure 4, especially as the datasets become challenging for the embedding model. How does the impact of the hyperparameters and temperature vary across the two new datasets tested (mini-ImageNet, and PathMNIST)?
Overall
Again, I appreciate the thorough answer from the authors, especially the theoretical extension, but for me to raise my score I would like to see two remaining parts addressed:
- More thorough evaluation of the impact of the backbone model. A single experiment where the dataset is sufficiently hard (CIFAR100) with a varying backbone of same family such as DINOv2 ViT-S, ViT-B, ViT-L, and ViT-G (keeping the projected dimension fixed is fine between models).
- Stronger hyperparameter motivation. The transferability of the chosen hyperparameters should be shown on the new datasets (mini-ImageNet and PathMNIST), by reproducing figure 3 and 4. Fewer seeds is acceptable in the interest of time and to save computation. In addition, the way hyperparameters was chosen should be added to the paper.
I recognize the limited remaining discussion time when asking for follow-up and clarification of experiments, but your response was first provided today.
Follow up question
I noticed that an explicit normalization step was added to the manuscript after your initial response. Was this step already implicit the initial experiments, but omitted from the original description, or was it introduced subsequently? If the implementation changed, please clarify which results were recomputed and whether any reported metrics changed.
Response to Reviewer E53F – Part 1/2
Thank you for your helpful and constructive review.
On the novelty of the method compared to previous works. We agree that certain components of the proposed method are connected to stages that are performed in previous work. However, there are major differences in the settings and the overall methodology, as we clarify below.
CPSC [1]: The paper studies classification of electronic-nose measurements, and does not target pretrained vision representations. In terms of methodology, they use class centroids and shrink class-specific deviations toward the overall mean. This is different from the proposed refinement procedure and the prototype-based score. Also, they do not specifically present a leave-one-out score computation.
Embedding Propagation [2]: This paper studies few-shot recognition of novel classes, using labeled base classes for representation learning. It constructs a similarity graph over embeddings and applies a propagation operator. Our method differs in the particular score construction and in the integration with conformal scoring.
Score diffusion [3]: They consider standard split CP, assuming sufficient data is available for both tuning and calibration. The general idea of score smoothing was studied in other works as well (Zargarbashi et al., 2023; Song et al., 2024). We draw inspiration from these works and perform smoothing over embeddings rather than scores, as part of our refinement procedure.
Conf-OT [4]: It starts with zero-shot class predictions from a pretrained model, plus labeled calibration observations and unlabeled queries. It addresses mismatch between pretrained predictions and the target task through entropy-regularized optimal transport. Conf-OT can operate on class logits, without requiring the full image-feature geometry. Its principal adaptation changes probability assignments, rather than embeddings or labeled visual prototypes. Moreover, its category prototypes originate from the zero-shot classifier, such as text embeddings. In addition, it jointly processes calibration and queries, rather than fitting refinement on a separate pool. To summarize, Conf-OT adjusts assignments to existing semantic categories, while ReProCP estimates visual class prototypes from scarce labels.
Harder datasets and weaker embeddings. We added two datasets (Appendix C.1, Tables 3–4): mini-ImageNet (100 ImageNet classes) and PathMNIST, a colon-pathology dataset from MedMNIST that is far from the backbones’ pretraining data. ReProCP gives the smallest sets at every budget on mini-ImageNet and on PathMNIST with DINOv2.
Mean set size (coverage), α=0.1.
mini-ImageNet (K=100, DINOv2 ViT-B)
\text{Method} & \text{2 shots} & \text{4 shots} & \text{8 shots} \\\\ \hline \text{Split CP (best)} & 92.59\ (0.93) & 2.73\ (0.94) & 1.10\ (0.94) \\\\ \text{CV+} & 2.28\ (0.94) & 1.12\ (0.93) & 1.06\ (0.94) \\\\ \text{SemiCP (best)} & 76.16\ (0.95) & 1.45\ (0.92) & 1.06\ (0.93) \\\\ \textbf{ReProCP} & \mathbf{1.25\ (0.95)} & \mathbf{1.06\ (0.94)} & \mathbf{1.04\ (0.94)} \end{array}$$ PathMNIST ($K = 9$, DINOv2 ViT-B) $$\begin{array}{lcccc} \text{Method} & \text{2 shots} & \text{4 shots} & \text{6 shots} & \text{8 shots} \\\\ \hline \text{CV+} & 7.70\ (1.00) & 3.76\ (0.93) & 2.95\ (0.94) & 2.46\ (0.95) \\\\ \text{SemiCP (best)} & 9.00\ (1.00) & 5.86\ (0.92) & 4.99\ (0.93) & 3.05\ (0.91) \\\\ \textbf{ReProCP} & \mathbf{5.34\ (0.97)} & \mathbf{2.80\ (0.92)} & \mathbf{2.26\ (0.90)} & \mathbf{2.10\ (0.90)} \end{array}$$ **Single-prototype representation.** A single sample as a representative mean can under-represent a class, affecting only set size efficiency, not coverage, which holds for any nonconformity score. Our refinement aims to make embeddings more suitable for prototype-based scoring by reducing nuisance and local variability (Sections 4.1 and 4.3). Each example is scored against a leave-one-out prototype built from the other $s-1$ examples of its class. At least two labels per class are therefore needed. Using only one label, the leave-one-out prototype is empty and the resulting scores are uninformative. **Temperature selection.** The temperature $T$ controls the sharpness of the softmax score. Since cosine margins vary across datasets and encoders, we estimate their scale using unlabeled pool embeddings and the k-means groups from the whitening stage as class proxies. We set $T=|\operatorname{median}(\text{margin})|/4$: the median is robust to outliers and hub points, while the fixed factor 1/4 maps a typical margin to a logit gap of 4. This factor is shared across datasets and encoders. Figure 3 shows that the selected temperatures lie in relatively flat regions of the sensitivity curves.