- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a post-hoc uncertainty quantification framework based on the Linearized Laplace Approximation (LLA) to address the lack of uncertainty estimation in contrastive learning, equipping contrastive representations with statistical confidence without requiring encoder retraining. First, by analyzing the output geometric curvature of loss functions, the authors reveal that distance- and margin-based contrastive objectives acting on relational outputs exhibit negative tangential curvature, which renders the LLA precision matrix indefinite and induces an invalid Gaussian posterior; to address this, they systematically characterize the curvature properties of various contrastive objectives and develop positive-semidefinite (PSD) spectral and radial repair schemes. Second, the authors theoretically demonstrate that the marginal variance of an individual embedding or pair score cannot directly reflect downstream decision uncertainty, and instead propose propagating encoder posterior perturbations directly through downstream decision rules (e.g., via predictive probability averaging for classification and posterior co-assignment matrices for clustering), further introducing a quality-coverage tradeoff mechanism based on "reliable cores". Extensive experiments on metagenomic binning as well as multiple image classification benchmarks (including histopathology datasets) demonstrate that the proposed framework substantially enhances calibration (effectively reducing ECE and NLL) and reliably filters ambiguous boundary instances in clustering tasks, outperforming deterministic baselines and domain-specific approaches.
Critical strengths
1.The paper rigorously reveals the curvature deficiencies within the loss geometry of contrastive objectives and provides mathematically sound positive-semidefinite (PSD) repair mechanisms. Furthermore, it theoretically clarifies and challenges the conventional assumption that representation variance directly equates to downstream decision uncertainty.
2.The ablation studies are thorough and comprehensive, accompanied by objective and multifaceted evaluation dimensions.
Critical weaknesses
1.Disconnected motivation for propagating encoder uncertainty to downstream tasks: When the downstream task is classification, what is the justification for artificially constructing the posterior using a contrastive loss (e.g., InfoNCE)? For standard classification tasks, the conventional cross-entropy loss is inherently convex and positive semidefinite—it suffers from neither negative eigenvalues nor the need for curvature repairs. Why, then, should one impose an auxiliary contrastive objective to fit the LLA in this setting?
2.Deep representation uncertainty is completely overlooked: When adapting foundation models to downstream applications, the primary source of epistemic uncertainty often stems from the domain shift experienced by the deep feature extractor between the upstream pretraining distribution and the downstream target distribution. Completely freezing the backbone and perturbing only the final linear projection layer leaves this uncertainty unmodeled, failing to address the fundamental discrepancy between upstream representation objectives and downstream task constraints.
3.Significantly weak baselines for downstream uncertainty quantification: In the image classification benchmarks (Table 2), the proposed approach is essentially compared only against the deterministic point estimate (MAP). More competitive and standard uncertainty quantification baselines (e.g., standard cross-entropy-based LLA, Deep Ensembles, MC-Dropout, and temperature scaling) are necessary to substantiate the claims of empirical superiority.
Questions for authors
See the analysis of weaknesses above.
6 comments
Question 3.
This is highly related to your first question. First, an important distinction is the source of uncertainty being modeled. Our work focuses on estimating epistemic uncertainty in the pretrained contrastive encoder and studying how this uncertainty propagates to downstream classification and clustering decisions. In contrast, conventional approaches (such as the baselines proposed by the reviewer), when applied to a classifier trained on fixed pretrained embeddings, primarily quantify uncertainty associated with the downstream classifier. Both sources of uncertainty are relevant and complementary, and could potentially be combined within a unified framework. Nevertheless, our main objective is to investigate the contribution of uncertainty originating from the representation itself. We use large amounts of unlabeled data to build the encoder and our paper focus on modeling the uncertainty of the encoder producing the representations, later used for the classifier. We could add uncertainty modeling on the classifier on top, but this outside the scope of our paper.
Regarding uncertainty in the encoder, an important baseline that we missed was LAM, as pointed by reviewer 2Ggh. We added a LAM-style binning baseline. We sample encoder weights from a Laplace posterior, summarize each contig’s normalized embeddings by its vMF mean direction and concentration, run k-means once on the mean directions, and reject low-concentration contigs first at reduced coverage. We compare this with our posterior coassignment pipeline on the four benchmark panels. High-quality bins use the aligned criterion from §6.1.
| Panel | LAM-style at 50% | LLA consensus at 50% | LAM-style at 100% | LLA consensus at 100% |
|---|---|---|---|---|
| Marine 5 | 34.2 ± 3.3 | 111.3 ± 1.5 | 19.8 ± 3.0 | 40.0 ± 1.0 |
| Marine 6 | 38.8 ± 2.2 | 132.3 ± 1.2 | 19.2 ± 1.1 | 48.7 ± 1.5 |
| Plant 5 | 5.8 ± 0.8 | 33.3 ± 1.2 | 3.6 ± 0.9 | 13.0 ± 0.0 |
| Plant 6 | 4.8 ± 0.8 | 36.7 ± 1.2 | 2.6 ± 0.9 | 14.0 ± 1.0 |
The gap at full coverage reflects differences in the partitions before any contigs are rejected. At 50% coverage the comparison also reflects the retention rules. We therefore interpret the table as evidence for the full coassignment pipeline in this binning setting.
Classifier-uncertainty baselines. To further address the reviewer's concern, we have also extended our CIFAR-100 experiments using frozen CLIP representations to include cross-entropy-based LLA, an eight-head ensemble, and MC-Dropout applied to the downstream classifier. Note that these approaches model uncertainty in a tangential component of the pipeline (the classifier, rather than in the encoder) and, thus, do not represent a completely fair comparison. Nevertheless, they provide relevant reference points for evaluating the quality of the resulting predictive distributions.
| Method | Accuracy (%) ↑ | NLL ↓ | ECE ↓ |
|---|---|---|---|
| Paper MAP | 74.00 ± 0.00 | 1.424 ± 0.000 | 0.165 ± 0.000 |
| Paper sampled LLA | 76.32 ± 0.05 | 0.993 ± 0.004 | 0.084 ± 0.000 |
| Paper Gaussian LLA | 77.34 ± 0.21 | 0.766 ± 0.002 | 0.014 ± 0.003 |
| Cross-entropy LLA, 16 draws | 75.79 ± 0.14 | 0.9907 ± 0.0012 | 0.0232 ± 0.0015 |
| Eight-head ensemble | 79.71 ± 0.05 | 0.7630 ± 0.0033 | 0.0444 ± 0.0008 |
| MC dropout, 16 draws | 75.22 ± 0.34 | 0.8344 ± 0.0055 | 0.0531 ± 0.0024 |
As it is shown in the table, an ensemble is a strong baseline to compete with. However, Gaussian LLA gives comparable results in NLL, better in ECE, and worse in ACC. Importantly, we do not interpret these comparisons as demonstrating that encoder uncertainty is inherently more informative than classifier uncertainty. Rather, they illustrate that uncertainty originating from different components of the pipeline can contribute to downstream predictive performance. Investigating their relative contributions and potential combination is a meaningful direction for future work, while the present paper focuses on modeling uncertainty in contrastive representations and propagating it through downstream decisions.
Clarifying the Practical Value and Frozen-Encoder Setting
I acknowledge the value of uncertainty quantification, but would appreciate clarification of the specific downstream reliability problem addressed and the additional value of the estimated uncertainty over existing confidence measures. Improved calibration and reliable-core selection provide initial evidence, but comparisons with simple calibration and selective prediction methods are missing. Moreover, the computational efficiency of last-layer LLA does not fully justify freezing the encoder. If avoiding retraining is a central motivation, please specify the relevant application constraints and compare the proposed approach with lightweight fine-tuning in terms of predictive performance, reliability, and computational cost, thereby establishing its practical value beyond improvements over a frozen MAP baseline.
Thank you for your careful review and for raising these questions about the classification experiments.
Question 1.
First, we emphasize that we are not constructing a posterior over a classifier using a contrastive loss, but rather estimating uncertainty in the pretrained encoder itself. We are sorry that this principal motivation was not clear. We now further develop the motivation of our paper.
In modern machine learning pipelines, a very common practice is to use large pretrained encoders, trained on datasets substantially larger than, and different from those available for the downstream task, to extract representations that are subsequently used for classification, clustering, or other tasks. In this setting, the encoder and the downstream model play distinct roles: the encoder defines the representation space, while the downstream method operates on the extracted embeddings. Our work focuses on quantifying epistemic uncertainty in this representation space by applying LLA to the encoder parameters, using a contrastive objective consistent with the relational structure of the learned representations. We then study how this encoder uncertainty propagates to downstream decisions.
As the reviewer points out, for classification, LLA could instead be applied to the downstream classifier using a cross-entropy objective, which yields positive-semidefinite GGN curvature. However, this would quantify uncertainty in the classifier parameters conditional on the extracted representations, rather than uncertainty in the encoder itself. Similarly, in clustering, post-hoc methods could be used to assess the stability of cluster assignments without explicitly modeling uncertainty in the encoder parameters. These approaches are complementary to ours, as they characterize different sources of uncertainty. Investigating their relative contributions, or combining them within a unified framework, is an interesting direction for future work that we leave outside the scope of the present study.
This is clearly not explained well enough in the paper and we will make use of the extra space to make sure it is.
Question 2.
As the reviewer correctly points out, we model uncertainty in the encoder’s final projection layer and propagate it through downstream decisions, while leaving uncertainty over the preceding feature extractor unmodeled. As discussed in our previous response, our work considers the common setting in which large pretrained encoders are reused as feature extractors for downstream tasks. In this setting, our approach builds on Laplace-based uncertainty estimation for metric-learning encoders, as introduced by Warburg et al. [1], which uses a restricted parameter scope for computational efficiency. Last-layer Laplace is a well-established and computationally efficient approximation, which has been shown to provide competitive uncertainty estimates at a substantially lower computational cost than full-network approaches [2]. While this restriction leaves uncertainty in the preceding feature extractor unmodeled, it provides a practical way to quantify uncertainty in the encoder without requiring inference over the entire backbone.
[1] Warburg, F., Miani, M., Brack, S., and Hauberg, S. Bayesian Metric Learning for Uncertainty Quantification in Image Retrieval. ICLR, 2023. Paper.
[2] Daxberger, E., Kristiadi, A., Immer, A., Eschenhagen, R., Bauer, M., and Hennig, P. Laplace Redux – Effortless Bayesian Deep Learning. NeurIPS, 2021.