acceptodds
Under review as a conference paper at ICLR 2027

Contrastive Linearized Laplace: Loss Geometry, Valid Repairs, and Reliable Cores

Abstract

While contrastive learning is widely used to learn deterministic representations, it does not inherently provide estimates of uncertainty. Post-hoc uncertainty estimation methods, such as the Linearized Laplace approximation (LLA), offer a way to quantify uncertainty without retraining the encoder. However, we show that directly using the LLA induced distribution over the contrastive embedding does not yield reliable pointwise uncertainty estimates for downstream tasks: embedding variance alone does not reflect whether decisions such as nearest-neighbor retrieval or cluster assignment are stable. To address this limitation, we develop a post-hoc framework based on LLA that propagates encoder uncertainty through the downstream task. We characterize the curvature of contrastive objectives and introduce positive-semidefinite repairs where needed to obtain valid local Gaussian approximations without retraining the encoder. We test the proposed approach in metagenomic binning and image classification tasks. For metagenomic binning, posterior coassignment probabilities support consensus partitions and the selection of reliable cores, leading to improvements over existing baselines. Additional image classification experiments show calibration improvements on standard benchmarks and real histopathology datasets.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Contrastive learning, Linearized Laplace Approximation, uncertainty quantification, Bayesian deep learning, posterior uncertainty, reliable cores, metagenomic binning, calibration

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_RbivOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a post-hoc uncertainty quantification framework based on the Linearized Laplace Approximation (LLA) to address the lack of uncertainty estimation in contrastive learning, equipping contrastive representations with statistical confidence without requiring encoder retraining. First, by analyzing the output geometric curvature of loss functions, the authors reveal that distance- and margin-based contrastive objectives acting on relational outputs exhibit negative tangential curvature, which renders the LLA precision matrix indefinite and induces an invalid Gaussian posterior; to address this, they systematically characterize the curvature properties of various contrastive objectives and develop positive-semidefinite (PSD) spectral and radial repair schemes. Second, the authors theoretically demonstrate that the marginal variance of an individual embedding or pair score cannot directly reflect downstream decision uncertainty, and instead propose propagating encoder posterior perturbations directly through downstream decision rules (e.g., via predictive probability averaging for classification and posterior co-assignment matrices for clustering), further introducing a quality-coverage tradeoff mechanism based on "reliable cores". Extensive experiments on metagenomic binning as well as multiple image classification benchmarks (including histopathology datasets) demonstrate that the proposed framework substantially enhances calibration (effectively reducing ECE and NLL) and reliably filters ambiguous boundary instances in clustering tasks, outperforming deterministic baselines and domain-specific approaches.

Critical strengths

1.The paper rigorously reveals the curvature deficiencies within the loss geometry of contrastive objectives and provides mathematically sound positive-semidefinite (PSD) repair mechanisms. Furthermore, it theoretically clarifies and challenges the conventional assumption that representation variance directly equates to downstream decision uncertainty.

2.The ablation studies are thorough and comprehensive, accompanied by objective and multifaceted evaluation dimensions.

Critical weaknesses

1.Disconnected motivation for propagating encoder uncertainty to downstream tasks: When the downstream task is classification, what is the justification for artificially constructing the posterior using a contrastive loss (e.g., InfoNCE)? For standard classification tasks, the conventional cross-entropy loss is inherently convex and positive semidefinite—it suffers from neither negative eigenvalues nor the need for curvature repairs. Why, then, should one impose an auxiliary contrastive objective to fit the LLA in this setting?

2.Deep representation uncertainty is completely overlooked: When adapting foundation models to downstream applications, the primary source of epistemic uncertainty often stems from the domain shift experienced by the deep feature extractor between the upstream pretraining distribution and the downstream target distribution. Completely freezing the backbone and perturbing only the final linear projection layer leaves this uncertainty unmodeled, failing to address the fundamental discrepancy between upstream representation objectives and downstream task constraints.

3.Significantly weak baselines for downstream uncertainty quantification: In the image classification benchmarks (Table 2), the proposed approach is essentially compared only against the deterministic point estimate (MAP). More competitive and standard uncertainty quantification baselines (e.g., standard cross-entropy-based LLA, Deep Ensembles, MC-Dropout, and temperature scaling) are necessary to substantiate the claims of empirical superiority.

Questions for authors

See the analysis of weaknesses above.

6 comments

Thank you for your careful review and for raising these questions about the classification experiments.

Question 1.

First, we emphasize that we are not constructing a posterior over a classifier using a contrastive loss, but rather estimating uncertainty in the pretrained encoder itself. We are sorry that this principal motivation was not clear. We now further develop the motivation of our paper.

In modern machine learning pipelines, a very common practice is to use large pretrained encoders, trained on datasets substantially larger than, and different from those available for the downstream task, to extract representations that are subsequently used for classification, clustering, or other tasks. In this setting, the encoder and the downstream model play distinct roles: the encoder defines the representation space, while the downstream method operates on the extracted embeddings. Our work focuses on quantifying epistemic uncertainty in this representation space by applying LLA to the encoder parameters, using a contrastive objective consistent with the relational structure of the learned representations. We then study how this encoder uncertainty propagates to downstream decisions.

As the reviewer points out, for classification, LLA could instead be applied to the downstream classifier using a cross-entropy objective, which yields positive-semidefinite GGN curvature. However, this would quantify uncertainty in the classifier parameters conditional on the extracted representations, rather than uncertainty in the encoder itself. Similarly, in clustering, post-hoc methods could be used to assess the stability of cluster assignments without explicitly modeling uncertainty in the encoder parameters. These approaches are complementary to ours, as they characterize different sources of uncertainty. Investigating their relative contributions, or combining them within a unified framework, is an interesting direction for future work that we leave outside the scope of the present study.

This is clearly not explained well enough in the paper and we will make use of the extra space to make sure it is.

Question 2.

As the reviewer correctly points out, we model uncertainty in the encoder’s final projection layer and propagate it through downstream decisions, while leaving uncertainty over the preceding feature extractor unmodeled. As discussed in our previous response, our work considers the common setting in which large pretrained encoders are reused as feature extractors for downstream tasks. In this setting, our approach builds on Laplace-based uncertainty estimation for metric-learning encoders, as introduced by Warburg et al. [1], which uses a restricted parameter scope for computational efficiency. Last-layer Laplace is a well-established and computationally efficient approximation, which has been shown to provide competitive uncertainty estimates at a substantially lower computational cost than full-network approaches [2]. While this restriction leaves uncertainty in the preceding feature extractor unmodeled, it provides a practical way to quantify uncertainty in the encoder without requiring inference over the entire backbone.

[1] Warburg, F., Miani, M., Brack, S., and Hauberg, S. Bayesian Metric Learning for Uncertainty Quantification in Image Retrieval. ICLR, 2023. Paper.

[2] Daxberger, E., Kristiadi, A., Immer, A., Eschenhagen, R., Bauer, M., and Hennig, P. Laplace Redux – Effortless Bayesian Deep Learning. NeurIPS, 2021.

Question 3.

This is highly related to your first question. First, an important distinction is the source of uncertainty being modeled. Our work focuses on estimating epistemic uncertainty in the pretrained contrastive encoder and studying how this uncertainty propagates to downstream classification and clustering decisions. In contrast, conventional approaches (such as the baselines proposed by the reviewer), when applied to a classifier trained on fixed pretrained embeddings, primarily quantify uncertainty associated with the downstream classifier. Both sources of uncertainty are relevant and complementary, and could potentially be combined within a unified framework. Nevertheless, our main objective is to investigate the contribution of uncertainty originating from the representation itself. We use large amounts of unlabeled data to build the encoder and our paper focus on modeling the uncertainty of the encoder producing the representations, later used for the classifier. We could add uncertainty modeling on the classifier on top, but this outside the scope of our paper.

Regarding uncertainty in the encoder, an important baseline that we missed was LAM, as pointed by reviewer 2Ggh. We added a LAM-style binning baseline. We sample encoder weights from a Laplace posterior, summarize each contig’s normalized embeddings by its vMF mean direction and concentration, run k-means once on the mean directions, and reject low-concentration contigs first at reduced coverage. We compare this with our posterior coassignment pipeline on the four benchmark panels. High-quality bins use the aligned F1>0.9F_1>0.9 criterion from §6.1.

PanelLAM-style at 50%LLA consensus at 50%LAM-style at 100%LLA consensus at 100%
Marine 534.2 ± 3.3111.3 ± 1.519.8 ± 3.040.0 ± 1.0
Marine 638.8 ± 2.2132.3 ± 1.219.2 ± 1.148.7 ± 1.5
Plant 55.8 ± 0.833.3 ± 1.23.6 ± 0.913.0 ± 0.0
Plant 64.8 ± 0.836.7 ± 1.22.6 ± 0.914.0 ± 1.0

The gap at full coverage reflects differences in the partitions before any contigs are rejected. At 50% coverage the comparison also reflects the retention rules. We therefore interpret the table as evidence for the full coassignment pipeline in this binning setting.

Classifier-uncertainty baselines. To further address the reviewer's concern, we have also extended our CIFAR-100 experiments using frozen CLIP representations to include cross-entropy-based LLA, an eight-head ensemble, and MC-Dropout applied to the downstream classifier. Note that these approaches model uncertainty in a tangential component of the pipeline (the classifier, rather than in the encoder) and, thus, do not represent a completely fair comparison. Nevertheless, they provide relevant reference points for evaluating the quality of the resulting predictive distributions.

MethodAccuracy (%) ↑NLL ↓ECE ↓
Paper MAP74.00 ± 0.001.424 ± 0.0000.165 ± 0.000
Paper sampled LLA76.32 ± 0.050.993 ± 0.0040.084 ± 0.000
Paper Gaussian LLA77.34 ± 0.210.766 ± 0.0020.014 ± 0.003
Cross-entropy LLA, 16 draws75.79 ± 0.140.9907 ± 0.00120.0232 ± 0.0015
Eight-head ensemble79.71 ± 0.050.7630 ± 0.00330.0444 ± 0.0008
MC dropout, 16 draws75.22 ± 0.340.8344 ± 0.00550.0531 ± 0.0024

As it is shown in the table, an ensemble is a strong baseline to compete with. However, Gaussian LLA gives comparable results in NLL, better in ECE, and worse in ACC. Importantly, we do not interpret these comparisons as demonstrating that encoder uncertainty is inherently more informative than classifier uncertainty. Rather, they illustrate that uncertainty originating from different components of the pipeline can contribute to downstream predictive performance. Investigating their relative contributions and potential combination is a meaningful direction for future work, while the present paper focuses on modeling uncertainty in contrastive representations and propagating it through downstream decisions.

Reviewer_RbivOpenReview ↗

Clarifying the Practical Value and Frozen-Encoder Setting

I acknowledge the value of uncertainty quantification, but would appreciate clarification of the specific downstream reliability problem addressed and the additional value of the estimated uncertainty over existing confidence measures. Improved calibration and reliable-core selection provide initial evidence, but comparisons with simple calibration and selective prediction methods are missing. Moreover, the computational efficiency of last-layer LLA does not fully justify freezing the encoder. If avoiding retraining is a central motivation, please specify the relevant application constraints and compare the proposed approach with lightweight fine-tuning in terms of predictive performance, reliability, and computational cost, thereby establishing its practical value beyond improvements over a frozen MAP baseline.

Reviewer_6nsNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies contrastive learning with Linearized Laplace approximation (LLA), which offers a way to quantify uncertainty without retraining the encoder. The authors pointed out that LLA application to contrastive embeddings does not provide reliable pointwise uncertainty estimates for downstream tasks. For example, embedding variance does not reflect whether decisions such as nearest-neighbor retrieval or cluster assignment are stable. To address the problem, the paper proposes a method that uses curvature of contrastive objectives and introduces positive-semidefinite repairs to obtain valid local Gaussian. The authors test their method on metagenomic binning and image classification task.

Critical strengths

The paper provides several interesting things:

  1. First of all, the paper clearly distinguishes the representation uncertainty and downstream task uncertainty. This is something I have thought several years ago, and this paper clearly points out the difference. The theoretical results in proposition 1 and 2 also explicitly show what they want to convey.
  2. The paper distinguishes losses such as InfoNCE, whose output space curvature is positive semidefinite, from distance based and margin losses that can have negative curvature. Its spectral repair removes negative eigenvalues, together with a positive definite prior or damping, this produces a valid Gaussian precision.
  3. It also shows meaningful empirical improvements.
Critical weaknesses

Although I enjoyed reading the paper, there are some points I have doubt with:

  1. The derivation starts with the MAP θ∗\theta^*, which need to be justified more. How to obtain it?
  2. The experiments do not fully isolate what produces the gains. For clustering, the comparison changes from deterministic k-means to repeated posterior-sampled k-means followed by spectral consensus. For sampled classification, it changes from one logistic regressor to separately fitted regressors whose predictions are averaged. Consequently, gains could partly reflect ensembling, refitting, or the changed clustering procedure
Questions for authors

Please find the weaknesses section, and answer those parts.

1 comment

Thank you for your thoughtful review and for highlighting the distinction between representation uncertainty and downstream decision uncertainty. Your questions helped us clarify how we choose the MAP reference point and examine which parts of the experimental pipeline contribute to the reported gains.

Where does the MAP come from?

It is the published pretrained checkpoint, with a Gaussian fitted around its last linear projection and all earlier weights frozen. Equation 2 defines an ideal MAP for a given loss and prior. In the reported experiments, we used the released weights directly and evaluated curvature under the target objective and prior. We will clarify this point in the final version of the paper.

What produces the gains?

We agree that the main comparisons change several components together.

Clustering. We added MAP spectral, which runs spectral clustering directly on a 15-nearest-neighbor graph of the fixed MAP embeddings. Comparing it with MAP k-means shows what changes when we use spectral clustering on the same representations. Comparing it with LLA consensus then shows the additional difference when spectral clustering uses a graph built from posterior coassignments.

At full coverage, MAP spectral accounts for a substantial part of the HQ-F1 improvement on the Marine panels. Relative to MAP k-means, it adds 13.0 and 21.6 HQ-F1 bins on Marine 5 and 6, respectively. LLA consensus adds a further 3.7 and 7.4 relative to MAP spectral. ARI favors MAP k-means on both Marine panels.

Full coverage. Mean ± sample SD. HQ-F1 counts reference labels with aligned F1>0.9F_1>0.9.

PanelMethodHQ-F1Accuracy (%)ARI
Marine 5MAP k-means23.3 ± 0.641.61 ± 0.120.308 ± 0.000
MAP spectral36.3 ± 0.639.83 ± 0.220.112 ± 0.004
LLA consensus40.0 ± 1.043.07 ± 0.420.215 ± 0.021
Marine 6MAP k-means19.7 ± 2.540.43 ± 0.190.300 ± 0.004
MAP spectral41.3 ± 2.138.70 ± 0.100.098 ± 0.004
LLA consensus48.7 ± 1.541.84 ± 0.250.207 ± 0.007

HQ-F1 with rejection. Each cell gives the mean for MAP spectral / LLA consensus. In this case, we compare how does the number HQ bins evolve when we do the rejection based on the reliability score (this depends on LLA uncertainty in both cases). At every tested coverage from 10% to 60%, LLA consensus yields more HQ-F1 bins than MAP spectral on both panels.

Contigs retainedMarine 5Marine 6
10%139.3 / 168.7133.0 / 160.0
20%133.3 / 164.7140.3 / 158.7
30%128.7 / 156.7137.7 / 159.7
40%128.7 / 154.3141.3 / 158.3
50%119.0 / 136.0143.7 / 146.0
60%104.7 / 113.7126.7 / 130.7
70%86.3 / 93.0112.7 / 112.3
80%71.0 / 70.385.3 / 87.0
90%55.0 / 53.363.0 / 65.7

The two spectral methods build their graphs differently, so the remaining differences reflect the posterior coassignment pipeline as a whole. We are adding the MAP-spectral baseline to Section 6.1 and will include the Marine full-coverage and quality–coverage results in Appendix F.

Classification. Regarding the classification experiments, the difference between MAP and Gaussian LLA is uniquely that the final layer weights are Gaussian (from a KFAC LLA approach). Hence, there are not several steps that can be compared with here. Both approaches train a logistic regressor classifier on the same data, with the only difference that the encoder is random in one of them.

Reviewer_QB5oOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how uncertainty in contrastively learned representations should be translated into uncertainty relevant to downstream decisions. The motivation is sensible: large embedding variance need not imply unstable predictions, while a point close to a decision boundary can change its assignment despite having small marginal variance. Correlations induced by a shared encoder also matter and cannot be summarized by pointwise variances.

The method constructs a last-layer Linearized Laplace Approximation around a pretrained encoder checkpoint, using GGN/KFAC curvature. For distance and margin losses with indefinite output Hessians, the paper considers PSD repairs, including spectral and radial constructions. Shared encoder samples are then propagated through the downstream task. For clustering, sampled partitions produce a posterior coassignment matrix, from which a consensus partition and reliable cores are obtained. For classification, predictive probabilities are averaged using sampled or Gaussian-logit readouts.

The evaluation includes four metagenomic binning panels and classification experiments on CIFAR-100, EMNIST, and SICAPv2, with ablations on curvature construction, cluster count, and posterior sample count.

Critical strengths

1. The motivation addresses a genuine problem, and the chosen methodology is relevant accordingly.

I agree with the paper’s central observation: downstream tasks care about changes in decisions, rather than the magnitude of representation changes alone. In clustering, pointwise variance misses how examples move together under perturbations of a shared encoder. Generating all representations under the same parameter draw and then examining relational stability is a natural response to this problem.

This gives the method a clear motivation. Propagating uncertainty through the decision rule, instead of treating embedding variance as a universal confidence score, is the most valuable aspect of the paper.

2. The experiment shows substantial improvement which is not only dependent on the exclusion of samples.

There are meaningful empirical improvements. At full coverage in Table 5, the number of HQ clusters increases from 20.4±1.120.4\pm1.1 to 42.6±2.142.6\pm2.1 on Marine 5, and from 18.6±0.918.6\pm0.9 to 47.4±1.747.4\pm1.7 on Marine 6. The gains therefore cannot simply be explained by removing difficult examples.

Classification also provides encouraging evidence. On CLIP/CIFAR-100, Gaussian LLA reduces NLL from 1.424 to 0.766 and ECE from 0.165 to 0.014. Multiple seeds and several modeling ablations are reported. Although the source of these gains still needs to be isolated, the results make the direction worth investigating.

Critical weaknesses

1. The specific algorithm is obscured by a descriptive, analytical narrative; one must piece the method together.

Sections 3–5 have a reasonable logical connection, but the paper does not distinguish early enough between what is being analyzed and what is actually executed. I can follow the individual arguments, yet still need the experiments and appendix to determine which layer receives the posterior, which objective supplies the curvature, what is rerun after sampling, and how the reliable core is constructed.

This is particularly noticeable in classification. Section 5 describes an expectation over pθ(y∣x)p_\theta(y\mid x), whereas Section 6.2 reveals that Sampled LLA retrains a logistic classifier in every sampled embedding space, while Gaussian LLA uses a different training procedure. This distinction affects both the meaning of the method and the fairness of the comparison.

An explicit end-to-end algorithm, with the experimental variants clearly identified, would substantially improve the paper. The current exposition makes the procedure more abstract than necessary and obscures the additional contribution.

2. The gains from the Laplace posterior have not been separated from those of standard consensus clustering.

The clustering experiments compare k-means on a single MAP representation against a pipeline containing posterior sampling, repeated k-means, a coassignment matrix, and a final spectral-consensus step. The perturbation source, aggregation procedure, and final clustering operation therefore change together. The current comparison does not establish how much of the improvement comes from the inferred posterior geometry rather than repeated partitions and consensus.

“Consensus + random” tests the selection score against random rejection. “MAP + same reliability” partly separates partition quality from ranking. Neither replaces the key control: retain the same number of draws and the same clustering/consensus pipeline, but use repeated MAP k-means initializations, bootstrap, or validation-tuned simple perturbations instead of Laplace samples.

This also affects the novelty assessment. Warburg et al. (2023) already studied Laplace encoders and positive-definite covariance constructions for metric learning, and the present paper appropriately discusses that work. The candidate-set treatment and downstream relational propagation are meaningful extensions, but their additional value needs a more direct empirical demonstration.

Questions for authors
  1. Could you specify the complete procedure used in each experiment, including the curvature objective, posterior target layer, need for repair, and downstream fitting after sampling? In particular, how do the readout-training procedures differ between Sampled LLA and Gaussian LLA?
  2. What happens when the same partition count, k-means settings, consensus algorithm, and reliability computation are used with repeated MAP initializations, bootstrap, or simple perturbations? What additional improvement does LLA provide over these controls?
  3. Is the checkpoint close to stationary for the downstream objective used to construct the posterior? Are gradient diagnostics or results after reoptimizing only the last layer available? How much does this choice affect predictions and reliability rankings?
  4. Under the selected prior and damping, does the unrepaired assembled precision still have materially negative eigenvalues? Under matched settings, how do predictions and core selection change with repair and with empirical Fisher?
  5. How do completeness relative to the original genomes and retained base-pair counts change after core selection? Can the external methods in Figure 2 be compared at matched coverage or matched completeness requirements?
  6. How much of the classification improvement remains against temperature scaling and simple ensembles with matched training procedures? Could you report end-to-end runtime and memory costs, and clarify whether the implementation and experimental configurations will be released? I could not locate an implementation link or an explicit release statement in the submission.
4 comments

Thank you for the detailed feedback on the method and its evaluation. Your comments helped us clarify the steps of the algorithm, and your suggestion to use matched clustering controls prompted us to examine what sampling from the fitted encoder posterior adds to consensus clustering.

Question 1.

We will add the following summary to the final version of the Appendix. In each benchmark, we load the pretrained checkpoint, freeze the feature extractor, and fit a checkpoint-centered KFAC Gaussian over the weights of one encoder projection. The encoder posterior stays fixed during downstream fitting. Appendix E.2, Table 4 lists the exact layer for each encoder.

Main-text experimentPosterior target and curvature objectiveDownstream procedure
CLIP repair ablation (Table 1)CLIP ViT-B/32 visual projection. Fit GGN curvature from supervised pairs using distance-Bernoulli or squared-hinge margin loss. These losses can give negative tangential curvature on negative pairs, so we compare spectral and radial-only repair.The CIFAR-100 protocol uses the Sampled LLA readout below with 64 draws (Appendix E.3). Table 1 also reports EMNIST, whose readout settings we will document explicitly alongside CIFAR-100.
Metagenomic binning (Fig. 2, §6.1)Last linear layer of the pretrained SemiBin2 encoder. Fit GGN curvature on the corresponding Marine or Plant calibration panel using must-link split pairs, sampled cannot-links, and the pairwise margin loss (m=1m=1). Apply spectral repair.Transfer the fitted posterior to the held-out panel. For each of 32 shared draws, re-embed all contigs and run K=500K=500 k-means with 20 initializations. Average coassignment indicators, run spectral consensus on the resulting 15-neighbor graph, and retain the highest-reliability contigs within each consensus bin (Eq. 9). The MAP baseline runs k-means on the fixed checkpoint embeddings.
CIFAR-100 and EMNIST classification (Table 2, §6.2)CLIP visual projection or final SimCLR ResNet fully connected layer. Fit GGN curvature from training-only supervised InfoNCE with 10 candidates per anchor. The candidate-logit Hessian is PSD, so no repair is needed.Use the readouts below, with 16 posterior draws for final LLA evaluation.
SICAPv2 classification (Table 2, §6.2)CLIP visual projection or CONCH-v1.5 attention-pool output map. Fit GGN curvature from training-only supervised InfoNCE with 8 candidates per anchor. No repair is needed.Use the same readouts and final draw count on the patient-separated split.

How the readouts are trained.

  • MAP fits one multinomial logistic readout on the deterministic checkpoint embeddings.
  • Sampled LLA draws an encoder projection, embeds the training data with that draw, and fits a separate logistic readout on those embeddings. Each readout predicts on test embeddings from the same draw. We then average the class probabilities.
  • Gaussian LLA fits one logistic readout with Adam. The fixed encoder posterior induces Gaussian logits for a given readout. During training, we average the softmax probabilities over logit draws and take the negative log of the resulting probability of the observed class. At test time we keep that single readout fixed and again average probabilities over logit draws.

Where the existing ablations are reported.

  • Curvature and repair. Appendix B.3 and Table 3 define the repair options. Appendix C describes the empirical Fisher (EF) alternative, and Fig. 3 shows the binning curves when EF replaces repaired GGN. Fig. 8 compares EF and radial-only GGN against GGN with spectral repair at K=500K=500.
  • Binning and reliable cores. Tables 5 and 6 give the full-coverage bin counts and the matched-coverage comparisons between selection rules, respectively. Figs. 6 and 7 show the coverage curves at K=100K=100 and K=200K=200. Fig. 11 compares all three values of KK at full coverage.
  • Number of posterior draws. Appendix G, Fig. 9 varies the draw count from 1 to 64 for CLIP and SimCLR on CIFAR-100 and EMNIST, using both GGN and EF without repair. Fig. 10 varies the same draw count for SemiBin2 under spectral GGN, radial-only GGN, and EF.
  • Synthetic illustration. Fig. 1 is a separate single-seed illustration of decision instability. Appendix D, Figs. 4 and 5 show how the decision margin, covariance orientation, and dependence between scores affect assignment and rank reversals.

Question 2.

We reran the comparison at k=500k=500 on Marine 6 contigs across three seeds. For each of 32 bootstrap partitions, we sampled contigs with replacement, fit k-means on their frozen MAP embeddings, and assigned every evaluation contig to its nearest fitted centroid. The same 15-neighbor MAP graph, and identical spectral consensus and reliability scoring.

Full coverage

DatasetMethodHQ-F1 count ↑Accuracy (%) ↑
Marine-6Single MAP19.00 ± 2.8340.25 ± 0.37
Marine-6Bootstrap29.60 ± 2.0741.90 ± 0.18
Marine-6LLA + consensus49.00 ± 3.5442.04 ± 0.21

This experiment showcases that consensus itself accounts for much of the improvement over single MAP clustering. However, we agree that the baseline is informative and bootstrap actually increases metrics, so we will include it in the final paper.

Question 3.

In this work we are using already trained checkpoints from the literature, whose training data is not publicly available. For this reason, testing stationarity in an absolute way is not possible.

However, we have examined the gradient norms of the objectives used to construct our posterior approximations at the pretrained checkpoints using the datasets we have. For SemiBin2, we observe very low gradient norms, indicating that the checkpoint is close to stationary for the pairwise margin objective evaluated on the calibration data. In contrast, the checkpoints used in our classification experiments are not stationary for the corresponding supervised InfoNCE objectives.

Model / datasetPP∥∇θL∥2/P\|\nabla_\theta\mathcal L\|_2/\sqrt{P}
CLIP / CIFAR-100393,2160.832740.83274
CLIP / EMNIST393,2160.722440.72244
SemiBin2 / Plant51,3001.3782×10−41.3782\times10^{-4}
SemiBin2 / Marine51,3001.0357×10−41.0357\times10^{-4}

This behavior is not unexpected, as we use pretrained checkpoints obtained from different training datasets and, particularly in the classification experiments, different training objectives. Importantly, as stated in Section 6, our goal is to construct a checkpoint-centered local Gaussian approximation using the curvature of the contrastive objective, without reoptimizing the pretrained encoder. This construction is mathematically well-defined under a positive-definite precision matrix and follows the general practice of post-hoc curvature-based uncertainty estimation. Although the lack of stationarity means that the resulting approximation should not be interpreted as a conventional Laplace posterior around a MAP optimum of the downstream objective, it remains a valid local uncertainty model around the pretrained representation.

Furthermore, our experimental results demonstrate that propagating the uncertainty captured by this approximation leads to improvements in downstream classification and clustering performance.

Regarding the question on "...after reoptimizing only the last layer available?", we are not reoptimizing any last-layer for any of the models. The checkpoints are used as given.

Question 4.

Using the selected prior and damping, the unrepaired precision can have negative eigenvalues. This is shown in Table 1, which measures output Hessians before aggregation. For example, on CLIP/CIFAR-100, the minimum output eigenvalue is approximately −0.992-0.992 for the margin loss and −3.32×10−11-3.32\times10^{-11} for distance-Bernoulli. Even if a negative direction survives projection through the Jacobian, positive contributions, the prior, and damping can offset it in the assembled precision. The result we establish is a sufficient PSD guarantee at the block level. Empirically, spectral and radial repairs give almost identical NLL gains in Table 1.

In binning, spectral GGN, radial GGN, and empirical Fisher use the same calibration procedure and consensus pipeline, with priors selected separately. We compared whether the selected reliable cores match across empirical Fisher, radial-repaired GGN, and spectral-repaired GGN, see the following table. Using spectral GGN as the reference, we measured the Spearman correlation between per-contig reliability rankings and the Jaccard overlap between the most reliable 50% of contigs, across five seeds. Rank correlations ranged from 0.938 to 0.969, and core Jaccard overlaps ranged from 0.837 to 0.898. Thus, the methods largely identify the same contigs as reliable.

DatasetCompared methodReliability rank correlation ↑Core Jaccard at 50% ↑
Marine-5EF0.969 ± 0.0040.898 ± 0.007
Marine-5GGN + Radial0.968 ± 0.0020.896 ± 0.004
Marine-6EF0.968 ± 0.0030.892 ± 0.005
Marine-6GGN + Radial0.968 ± 0.0010.892 ± 0.002
Plant-5EF0.954 ± 0.0040.868 ± 0.005
Plant-5GGN + Radial0.953 ± 0.0100.868 ± 0.016
Plant-6EF0.944 ± 0.0060.848 ± 0.010
Plant-6GGN + Radial0.938 ± 0.0030.837 ± 0.005

Question 5.

We quantified how the amount of retained sequence and the clustering quality in the following table. The following table shows that core selection improves the assignment quality of retained DNA while reducing the total amount of correctly recovered sequence, illustrating a precision–recovery trade-off. For example, retaining the most reliable 50% of Marine-5 contigs preserves 57.29% of the evaluated DNA and increases BP precision from 48.07% to 65.96%, while BP recovery decreases from 48.07% to 37.79%. Within the retained subset, macro precision and recall increase from 52.77% to 59.20% and from 48.59% to 56.51%, respectively.

DatasetContigs retained (%)Retained DNA (Mb)Total DNA retained (%)BP precision (%)BP recovery (%)Retained macro precision (%)Retained macro recall (%)
Marine-5100706.38 ± 0.00100.00 ± 0.0048.07 ± 0.3348.07 ± 0.3352.77 ± 0.1648.59 ± 0.15
Marine-550404.70 ± 0.3157.29 ± 0.0465.96 ± 0.4837.79 ± 0.2659.20 ± 0.1556.51 ± 0.09
Marine-6100755.42 ± 0.00100.00 ± 0.0047.68 ± 0.3347.68 ± 0.3352.02 ± 0.4049.04 ± 0.35
Marine-650431.19 ± 0.5357.08 ± 0.0766.46 ± 0.4837.94 ± 0.2657.81 ± 0.5155.76 ± 0.32
Plant-5100372.76 ± 0.00100.00 ± 0.0038.22 ± 0.2738.22 ± 0.2738.65 ± 0.4638.31 ± 0.29
Plant-550224.68 ± 0.4660.27 ± 0.1251.93 ± 0.4431.30 ± 0.2742.79 ± 0.2642.35 ± 0.25
Plant-6100340.37 ± 0.00100.00 ± 0.0033.89 ± 0.8033.89 ± 0.8035.08 ± 0.8235.84 ± 0.80
Plant-650200.16 ± 0.1458.81 ± 0.0445.67 ± 0.8726.86 ± 0.5139.56 ± 1.1439.67 ± 0.90

The external-method results in Figure 2 were taken from their original paper. Since the authors do not provide the models or report the selected coverage for each result, we cannot reconstruct comparisons at matched coverage or completeness. We therefore present these values as published reference results, rather than as comparisons at matched operating points.

Question 6.

Our uncertainty model is placed on the pretrained encoder, and its uncertainty is propagated through downstream classification or clustering. We do not fit an additional posterior over downstream classifier parameters: the readouts are deterministic conditional on the sampled representations, although the sampled-readout variant refits a classifier for each encoder draw. Consequently, applying ensembles or temperature scaling to the classifier would assess a different source of uncertainty. Jointly modeling encoder and downstream classifier uncertainty is a natural direction for future work.

For the sake of completeness, we are running a comparison of the proposed method with other baselines on the classifier on Cifar 100, so here the source of uncertainty is the not in the embedding network but rather in the classifier with frozen CLIP as embedding. Again, the comparison is not completely fair as the uncertainty estimates differ in where they are performed (encoder in Contrastive LLA versus classifier in the rest of the baselines), but we think its worth including it.

MethodAccuracy (%) ↑NLL ↓ECE ↓
Paper MAP74.00 ± 0.001.424 ± 0.0000.165 ± 0.000
Paper sampled LLA76.32 ± 0.050.993 ± 0.0040.084 ± 0.000
Paper Gaussian LLA77.34 ± 0.210.766 ± 0.0020.014 ± 0.003
Cross-entropy LLA, 16 draws75.79 ± 0.140.9907 ± 0.00120.0232 ± 0.0015
Eight-head ensemble79.71 ± 0.050.7630 ± 0.00330.0444 ± 0.0008
MC dropout, 16 draws75.22 ± 0.340.8344 ± 0.00550.0531 ± 0.0024

As you can see in the table, an ensemble is a strong baseline to compete with. However, Gaussian LLA gives comparable results in NLL, better in ECE, and worse in ACC. It is worth mentioning, that you could potentially "combine" these sources of uncertainty.

We have also measured the runtime and process RAM of our 32-draw LLA pipeline on each genome panel. Each cell reports wall time / peak observed RAM.

StepMarine 5Marine 6Plant 5Plant 6
Training setup and MAP baseline58.7 s / 1.07 GiB52.9 s / 1.07 GiB16.6 s / 1.00 GiB16.5 s / 1.00 GiB
KFAC curvature fit0.9 s / 1.54 GiB1.0 s / 1.54 GiB†0.6 s / 1.48 GiB0.6 s / 1.48 GiB†
Four-prior validation90.8 min / 2.36 GiB101.2 min / 2.36 GiB13.1 min / 1.63 GiB11.5 min / 1.56 GiB
Transfer setup and MAP encoding15.7 s / 1.45 GiB16.3 s / 1.48 GiB12.6 s / 1.15 GiB11.7 s / 1.14 GiB
MAP k-means and embedding trace4.1 min / 1.34 GiB4.7 min / 1.36 GiB1.2 min / 1.16 GiB1.1 min / 1.15 GiB
Posterior draws and per-draw k-means136.2 min / 1.63 GiB149.5 min / 1.67 GiB36.7 min / 1.40 GiB36.7 min / 1.38 GiB
Partition stability6.1 s / 1.34 GiB6.1 s / 1.37 GiB3.2 s / 1.19 GiB3.1 s / 1.18 GiB
Spectral consensus and reliability105.7 min / 16.87 GiB118.3 min / 16.11 GiB68.5 min / 13.33 GiB77.3 min / 14.94 GiB
Reliable-core evaluation7.3 s / 1.35 GiB7.6 s / 1.38 GiB2.9 s / 1.20 GiB3.1 s / 1.18 GiB
End-to-end time5 h 44 min6 h 22 min2 h 02 min2 h 09 min

Per-draw clustering and spectral consensus accounted for 96–97% of transfer time, and RAM peaked during spectral consensus on every panel. RAM was sampled every 15 seconds, so brief peaks may be missed. All this is measured in a RTX 5070, and will be included in the manuscript.

We would also like to point out that the source code is submitted in the supplementary material and will also be published on GitHub upon acceptance of the paper.

Reviewer_2GghOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper considers linearised Laplace approximations for contrastive learning problems. The paper argues that some contrastive losses do not lead to semidefinite Laplace approximations, and that correlations between Bayesian encodings should be taken into account.

Critical strengths
  • Contrastive losses are often used, and Bayesian approximations for such settings are useful.
  • The paper raises an important point that correlations between encodings are important for downstream tasks.
Critical weaknesses

First, as far as I can tell, the paper has quite limited novelty. The majority of the reported findings appear to have already been published by Warburg et al. The paper merely states that:

This provides a unified treatment of pairwise and candidate-set contrastive objectives and extends previous observations for Bayesian metric learning (Warburg et al., 2023).

No further details are provided. As far as I can tell, Warburg et al. develop similar corrections to ensure that the approximate posterior has positive semidefinite covariances. One key difference is that Warburg et al. seem to capture correlations between embeddings by Monte Carlo Sampling encoder networks, while the present paper relies on network linearization. As far as I can tell, this is only a minor implementation detail, but I cannot assess this, as the paper does not provide baseline comparisons to Warburg et al.

Secondly, a large part of the paper is dedicated to a discussion about how to propagate uncertainty to the final task. This is an interesting discussion, but the finding seems to be that we should just marginalise the embeddings, which is also what prior work does. So, I am left to wonder what the novel insight actually is.

Questions for authors
  1. Please respond to Weaknesses above.
  2. Why do you not provide results for baseline methods?
  3. It seems that you use k-NN to select which embeddings points to correlate. How do you compute those nearest neighbors? Is this based on mean embeddings?
1 comment

Thank you for your careful reading and for asking us to clarify how our work builds on Warburg et al. Your feedback helped us explain more clearly why we propagate encoder uncertainty through the downstream task before summarizing it. We also added a LAM-style baseline to illustrate this distinction in metagenomic binning.

Q1. Relation to Warburg et al.

We cited Warburg et al. too briefly and did not explain how our work builds on LAM. LAM already applies Laplace inference to contrastive encoders, identifies indefinite curvature in a pairwise contrastive loss, and proposes PSD approximations. We extend this analysis to pairwise and candidate-set objectives, characterize when the output curvature is already PSD, and give repairs at the level of the loss outputs. We will make this relationship explicit in the revision.

The main distinction is when we summarize uncertainty. LAM uses encoder draws to estimate a von Mises-Fisher distribution for the embedding. Our approach does not rely on a marginal embedding distribution as the final uncertainty measure. Instead, we propagate the estimated uncertainty propagate encoder uncertainty through the downstream task and summarize its outputs. In classification, we average class probabilities after applying the classifier (Eq. 7). In binning, we embed all contigs with the same encoder draw, run kk-means, and repeat the procedure for each draw. We then use the fraction of draws in which two contigs cluster together to build a consensus partition and score assignment reliability.

This is why the quantity being marginalized matters. Two contigs can each have high embedding variance and still reliably belong to the same bin if they move together. Propositions 1–2 formalize the same point for assignment and ranking: decision stability depends on the margin and on how the relevant quantities vary together.

Q2. Why do you not provide results for baseline methods?

The original submission included several binning references but omitted a LAM comparison. We will add a LAM-style vMF baseline that clusters the mean directions of the sampled contig embeddings and retains the highest-concentration contigs at reduced coverage.

The following table reports the number of aligned bins with retained-set F1>0.9F_1>0.9, following §6.1. At full coverage, the differences reflect the partitions themselves. At 50% coverage, they also reflect the retention rules.

PanelvMF, 50%LLA consensus, 50%vMF, 100%LLA consensus, 100%
Marine 534.2 ± 3.3111.3 ± 1.519.8 ± 3.040.0 ± 1.0
Marine 638.8 ± 2.2132.3 ± 1.219.2 ± 1.148.7 ± 1.5
Plant 55.8 ± 0.833.3 ± 1.23.6 ± 0.913.0 ± 0.0
Plant 64.8 ± 0.836.7 ± 1.22.6 ± 0.914.0 ± 1.0

For CIFAR-100, we also compared against three baselines that place uncertainty in the classifier on fixed CLIP embeddings. We reiterate that this is not exactly a fair comparison, as the main idea of the paper is to propagate the uncertainty from the encoder to the downstream task; which is fairly different from putting uncertainty in the classifier. However, we are including it here. Sampled and Gaussian LLA instead propagate uncertainty from CLIP’s final embedding layer. The eight-head ensemble achieves higher accuracy, while Gaussian LLA achieves lower ECE and similar NLL. Modeling both sources jointly is a direction for future work.

MethodAccuracy (%) ↑NLL ↓ECE ↓
MAP74.00 ± 0.001.424 ± 0.0000.165 ± 0.000
Sampled LLA76.32 ± 0.050.993 ± 0.0040.084 ± 0.000
Gaussian LLA77.34 ± 0.210.766 ± 0.0020.014 ± 0.003
Cross-entropy LLA, 16 draws75.79 ± 0.140.9907 ± 0.00120.0232 ± 0.0015
Eight-head ensemble79.71 ± 0.050.7630 ± 0.00330.0444 ± 0.0008
MC dropout, 16 draws75.22 ± 0.340.8344 ± 0.00550.0531 ± 0.0024

Q3. How do you compute those nearest neighbors? Is this based on mean embeddings?

Yes. We build the 15-nearest-neighbor graph once using Euclidean distances between embeddings evaluated at the posterior mean encoder weights. For each edge, we estimate its coassignment probability as the fraction of posterior draws in which its two contigs are assigned to the same cluster. The graph therefore determines which coassignment probabilities we compute.

Comments on the paper

1 comment

Dear AC and reviewers,

We would like to kindly follow up on our rebuttal responses. As the discussion period is approaching its end, we would greatly appreciate it if the reviewers could take a moment to read our responses and any additional clarifications provided.

We hope that our answers have adequately addressed the concerns raised during the review process. If the reviewers find our responses satisfactory, we would be grateful if they could consider updating their assessments and, where appropriate, increasing their scores.

We remain available to address any further questions or concerns before the discussion period closes.

Thank you very much for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.