acceptodds
Under review as a conference paper at ICLR 2027

Embrace Ambiguity: Rethinking Contrastive Learning through Similarity Uncertainty

Abstract

Contrastive learning has become a dominant paradigm for self-supervised representation learning, but two key limitations persist: (1) embedding similarities are treated as deterministic, causing ambiguous pairs to be optimized with unwarranted confidence and potentially distorting fine-grained semantic structure; and (2) prior probabilistic approaches model uncertainty in individual representations or learn temperatures heuristically, leaving the uncertainty of the pairwise similarity itself, which directly governs contrastive objectives, unmodeled. To address these limitations, we propose Distributional Contrastive Learning (DistCLR), a self-supervised framework that models both representations and pairwise similarities distributionally, inducing similarity uncertainty in closed form from the underlying representation posteriors. Motivated by an evidence lower bound (ELBO), DistCLR formulates an uncertainty-aware InfoNCE objective in which the global temperature is replaced by this per-pair similarity uncertainty, softening ambiguous pairs while preserving sharp alignment for confident ones. DistCLR changes only the pairwise contrastive score: it requires no Monte Carlo sampling, adds less than 1% wall-clock overhead, recovers InfoNCE under homoscedastic uncertainty, and plugs into existing InfoNCE-based frameworks without altering their training pipelines. Experiments on CIFAR-10, CIFAR-100, ImageNet-100, and ImageNet-1k show that DistCLR (1) matches or improves classification performance across three contrastive frameworks and two architectures, (2) induces clearer semantic organization in the representation space, as evidenced by neighborhood consistency, effective rank, clustering, and cross-domain transfer assessments, (3) improves strict open-set recognition, robustness under corrupted data, and misclassification detection, and (4) outperforms representative uncertainty-aware contrastive methods in both linear-probe accuracy and error detection. Code is provided in the supplementary material.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Deep Learning, Self-supervised Learning, Contrastive Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5v1wOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper proposes Distributional Contrastive Learning (DistCLR), which models pairwise similarity uncertainty through an adaptive temperature derived from probabilistic representations. The method combines uncertainty-aware InfoNCE with a directional prior regularizer and can be integrated into existing contrastive learning frameworks. Experiments on CIFAR-10, CIFAR-100, ImageNet-100, and ImageNet-1k evaluate classification, transfer learning, representation quality, open-set recognition, and robustness under distribution shifts.

Critical strengths
  1. The paper introduces pairwise similarity uncertainty into contrastive learning through an adaptive temperature formulation. The method can be integrated into existing contrastive frameworks with limited changes and low computational overhead, making it practically useful.
  2. The experiments cover several contrastive frameworks and datasets, including ImageNet-1k. The evaluation also considers transfer learning, open-set recognition, and corruption robustness, providing evidence of the method's potential benefits beyond classification accuracy.
Critical weaknesses
  1. The theoretical foundation of the proposed uncertainty formulation is not sufficiently convincing. The similarity variance approximation in Proposition 1 relies on concentrated Gaussian posteriors, but it is unclear whether this assumption holds in practice. For example, the learned scale reported in Figure 3 appears inconsistent with the concentrated-posterior regime at the given representation dimension. This raises concerns about whether the derived similarity uncertainty accurately reflects the underlying representation uncertainty.

  2. The experiments do not clearly establish the benefit of learning similarity uncertainty over simpler temperature adaptations. In Table 5(b), the constant-scale variant with the same geometric factor achieves nearly the same ARI as the full DistCLR (18.16 vs. 18.21). While the learned scales improve error detection, their contribution to representation quality remains less clear. This weakens the evidence that the proposed uncertainty modeling, rather than the geometric temperature adjustment, drives the reported representation improvements.

Questions for authors
  1. Can the authors provide empirical evidence that the first-order similarity variance approximation in Proposition 1 remains accurate under the learned scales? In particular, how well does Eq. (11) match the actual similarity variance estimated through Gaussian sampling at the operating point reported in Figure 3?
  2. Could the authors compare the full DistCLR with the constant-scale-plus-geometry variant on linear-probe accuracy and corruption robustness under the same training settings? Given their nearly identical ARI results in Table 5(b), this would help clarify whether the learned uncertainty provides additional benefits for representation learning beyond the geometric temperature adjustment.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 5v1w [Part 1/3]

Thank you for the two precise questions. We ran both checks. They led to an exact result that now anchors §3.2 (Proposition 2) and to the comparison you proposed in six settings (all changes in blue).

Q1. Is Eq. (11) accurate under the learned scales? We sampled it at the operating point of Fig. 3 (App. B.8, three checkpoints, 256 test pairs × 16,384 Gaussian samples each). There (L − 1)u ≈ 7.9, outside the condition (L − 1)u ≪ 1 of Proposition 1, and Eq. (11) overstates the sampled variance about 16×. Two properties do hold at this operating point. The dependence of the variance on the pair is exactly (1 − s²) at any concentration (Proposition 2 below), and within each anchor's softmax row Eq. (11) orders the candidates as sampling does (median Spearman .96 in the subspace the embeddings occupy, App. B.8). The form that DistCLR's temperature uses is therefore exact, and its magnitude is learned. Where Proposition 1 does hold (a 16-dimensional projector output, (L − 1)u ≈ 0.43), DistCLR improves on InfoNCE at the same temperature on every metric and every seed (+0.76 linear, +1.17 probe, +1.31 CIFAR-100-C, p ≤ .014), also against InfoNCE at DistCLR's mean temperature τ = 0.20 (App. B.8).

Proposition 2 (exact similarity variance). For z_i = normalize(ẑ_i + ε_i) with independent isotropic Gaussian noise of any variance, write a_i = ⟨z_i, ẑ_i⟩, μ_i = E[a_i], m_i = E[a_i²], c_i = (1 − m_i)/(L − 1). Rotational symmetry (Mardia & Jupp, 2000) gives

Var⟨z_1, z_2⟩ = (B + C) + (A − B)(1 − s²), with A = m_1c_2 + c_1m_2 − c_1c_2, B = m_1m_2 − μ_1²μ_2², C = (L − 1)c_1c_2.

Eq. (11) is its first-order limit (A − B → u_1 + u_2, B + C → 0), and direct sampling matches the identity within 0.5%. DistCLR's temperature τ_A + (u_r + u_t)(1 − s²) has this form at every concentration: τ_A carries the pair-independent part and u the slope, learned per view. At any operating point, (1 − s²) is also the squared spherical gradient norm of s, so each view's scale is weighted by how strongly the pair's similarity responds to that view (App. B.15).

What u is in the trained model. As the submission states (§4.5), the learned scale ranks uncertainty rather than estimating a calibrated variance: harder views receive smaller u, and U = 1 − u ranks each image's spread over 16 training augmentations (Spearman .69), errors (AUROC .73–.78 in every framework and backbone), shifted inputs and unseen classes (comment 2/3).

Framing and title. Following your review, the abstract, Fig. 1, the introduction, §3 and the Conclusion present DistCLR as (i) a geometric per-pair temperature with the functional form of the similarity variance (the form is exact at any concentration, and its magnitude is learned) and (ii) a learned per-view scale that scores view hardness. The word "exactly" now refers only to this form. For the camera-ready we can retitle the paper accordingly: "Embrace Ambiguity: Rethinking Contrastive Learning through the Geometry of Similarity Uncertainty".

AuthorsOpenReview ↗

Response to Reviewer 5v1w [Part 2/3]

Q2 in one line. The geometric factor (1 − s²) is DistCLR's similarity-uncertainty temperature (its exact variance form, Proposition 2), so the shared-scale arm is DistCLR with one tied scale. With either scale, DistCLR is more robust than InfoNCE and the cosine temperature schedules and keeps more structure than a learned per-view temperature without the variance form. Learning the scale per view adds an uncertainty signal that neither a shared scale nor a head trained afterwards reaches.

Q2. Learned scale vs shared scale + geometry under identical code and settings (Table 5c: CIFAR-100 SimCLR, ResNet-18, all at τ_A = 0.185, 3 seeds per arm, all arms in one code path). This experiment is separate from Table 1 and not comparable with it in absolute values. The shared arm uses one scale for all views and pairs (App. B.11). Online: linear classifier trained during pretraining. Linear: 100-epoch linear evaluation. CIFAR-100-C and AUROC: frozen probe.

OnlineLinearCIFAR-100-CError AUROC of U, clean / sev. 5
InfoNCE (τ = τ_A)66.26 ± 0.2166.78 ± 0.3248.52 ± 0.15–
DistCLR, shared scale66.17 ± 0.1866.76 ± 0.2048.91 ± 0.10.481 / .479
DistCLR, learned scales66.49 ± 0.2167.04 ± 0.1649.52 ± 0.11.756 / .743
  • Accuracy (linear evaluation, the primary metric): 67.04 with learned scales, 66.76 with the shared scale, 66.78 for InfoNCE. On the frozen probe the shared scale is higher (59.18 vs 58.78, InfoNCE 58.16; Table 13).

  • Corruption robustness: both DistCLR arms are more accurate than InfoNCE on CIFAR-100-C (48.91 and 49.52 vs 48.52, Welch p = .03 and .001), the learned scales most.

  • Uncertainty: the shared arm's scale head is trained by the prior alone, so this control isolates what learning the per-view scale adds: an error-ranking signal (AUROC .756, still .743 at severity 5, vs ~.5). U also outperforms every other label-free score we tested (embedding and feature norms, k-NN distance, augmentation disagreement, TaU and MCInfoNCE, App. B.2 and B.14):

    • covariate-shift detection on CIFAR-100-C: .565 → .690 from severity 1 to 5, vs at most .545 → .629 for the norms and the k-NN distance (ahead at every severity) and at most .55 at any severity for TaU and MCInfoNCE.
    • accuracy under shift, with no labels of the target set and no classifier: over the 96 clean and CIFAR-100-C test sets, the mean U of a set predicts the probe's accuracy on it (Spearman −.96, held-out-corruption error 3.2 points vs 3.7 to 7.3 for the feature norms and k-NN distances of DistCLR, shared-scale and InfoNCE encoders, and 2.1 to 2.3 for the probe's own average confidence, which needs the classifier; App. B.14).
    • unseen classes (CIFAR-10 images against CIFAR-100): .690 vs .649 for the k-NN distance, on every seed (App. B.14).
    • a head trained afterwards on the frozen InfoNCE encoder to predict each view's agreement with its positive ranks errors at .707, below U (.756, every seed): the signal comes from training with DistCLR's objective (App. B.14).
    • error ranking: .756 vs at most .733 for the norms and the k-NN distance, at most .61 for TaU and MCInfoNCE, and .749 vs .681 for augmentation disagreement under the protocol of App. B.2.

    Fused with the probe's MSP, U also gives the best open-set recognition (Table 4).

Table 5c and Table 1. Table 5c is a separate experiment (three pretraining seeds, the revision's code path, every arm at τ_A), which the Table 5c caption now states, so each table is compared within itself. At the same base temperature, DistCLR is ahead of InfoNCE in online, linear and CIFAR-100-C accuracy (+0.23, +0.26, +1.00).

Comment 3/3 repeats this comparison in NNCLR, All4One, ViT-Small and on CIFAR-10.

W2. Simpler temperature adaptations (Table 5b, matched optimization, 3 runs each):

VariantARIEffRankError AUROC
Fixed InfoNCE16.87 ± 0.81191.67 ± 0.69–
Learned per-view temperature, no geometry ("pointwise")17.34 ± 0.06185.29 ± 0.05.540 ± .001
Shared scale + geometry18.16 ± 1.03194.86 ± 0.17.502 ± .013
Learned scale, shuffled across samples16.80 ± 0.27193.56 ± 1.24.553 ± .042
DistCLR18.21 ± 0.61195.49 ± 0.68.749 ± .002

A learned per-view temperature without the geometry loses structure (EffRank 185.3) and the error signal (.540). Only the combination of the geometry with correctly assigned learned scales gives both.

What this establishes. Both parts of DistCLR's temperature contribute, as Contribution 4 now states:

"Matched ablations separate the two parts of the temperature. Both contribute: the similarity-uncertainty geometry (1−s²) improves semantic organization (clustering and effective rank) and robustness over InfoNCE, and learned per-view scales add a signal that ranks errors and unknown inputs."

AuthorsOpenReview ↗

Response to Reviewer 5v1w [Part 3/3]

Adapting the global temperature instead (App. B.12, Table 5c recipe and evaluation, 3 seeds each):

TemperatureOnlineLinearCIFAR-100-CError AUROC of U
Cosine schedule centred on the tuned τ (τ 0.1–0.3)65.86 ± 0.1366.26 ± 0.1047.70 ± 0.36–
DistCLR66.49 ± 0.2167.04 ± 0.1649.52 ± 0.11.756 ± .003

The strongest adaptive baseline, the cosine schedule centred on the tuned τ, stays below DistCLR by 0.63 online, 0.78 linear and 1.82 points on CIFAR-100-C, and no global temperature gives a per-sample signal.

A hard-pair baseline at the matched temperature (App. B.12). HCL (Robinson et al., 2021; published β = 1, τ⁺ = 0.1), run in the same pipeline at τ = 0.2, reaches 58.0 online with β = 1 and 56.86 ± 0.44 with the milder β = 0.5 (3 seeds). It needs its published τ = 0.5 (66.01 online, 66.07 linear), and there it remains below DistCLR (66.49, 67.04).

The same comparison in other frameworks, on a transformer backbone and on a second dataset (App. B.11, Table 13, CIFAR-100 unless stated, 3 seeds each, deterministic arms of NNCLR and All4One at their released τ_d = 0.2). Linear: standard linear evaluation. Probe, corrupted-data accuracy and AUROC: frozen probe. Shared scale → learned scale:

SettingLinear acc.Probe acc.CIFAR-100-C / CIFAR-10-CError AUROC of U
NNCLR, ResNet-1867.35 → 67.8061.81 → 62.6642.10 → 42.36.505 → .748
All4One, ResNet-1870.54 → 70.7265.22 → 65.3142.46 → 42.65.496 → .784
SimCLR, ViT-Small (800 ep)67.19 → 67.5560.45 → 61.0352.50 → 52.93.488 → .726
SimCLR, ResNet-18, CIFAR-10 (CIFAR-10-C)90.58 → 90.8189.25 → 89.1282.22 → 82.12.508 → .780

In NNCLR, All4One and ViT-Small the learned scale is also slightly more accurate on every metric, and only it ranks errors. On CIFAR-10 both DistCLR variants improve on InfoNCE (learned scale: probe 89.12 vs 88.62, Welch p = .02, and CIFAR-10-C 82.12 vs 81.55, p = .04). Only the learned scale ranks errors, in every setting. DistCLR also matches or exceeds each deterministic objective (NNCLR linear 67.80 vs 67.54 and CIFAR-100-C 42.36 vs 42.02, All4One probe 65.31 vs 64.99, CIFAR-100-C 42.65 vs 42.13 and linear 70.72 vs 70.42).

The shared scale sits where the prior puts it (u = 2σ_p² = 0.04, vs a learned mean ≈ 0.031). Doubling it (σ_p² = 0.04, App. B.10) keeps its error AUROC near chance (.511), so the error-ranking signal does not come from the scale's magnitude.

Over the tested hyperparameter ranges (τ_A 3×, λ_prior 10×, σ_p² 4×), its error AUROC stays within .733–.762 (App. B.10, 3 seeds per setting).

All additions are in the appendix. In the main text, §3.2 states Proposition 2 and Table 5c reports the comparison you proposed.

Thanks to your questions, the temperature now rests on an exact result at the learned operating point, and the comparison you proposed holds in SimCLR, NNCLR and All4One, on a ViT-Small backbone and on CIFAR-10: learned scales alone rank errors and detect shift. If these answers resolve your soundness concern, we would be grateful if you would consider updating your score. If anything still stands in the way, we would be glad to address it within the discussion period.

Reviewer_1GhcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies contrastive learning under uncertainty view points, and proposes DistCLR, which adjusts the strength of contrastive learning based on the uncertainty in the relationship between two images. It uses a learned scale for each image and the similarity between images to compute a pairwise temperature. The method modifies InfoNCE with a correction term and regularization, and the resulting objective can be computed without sampling.

Critical strengths

The paper is well written and gives motivation to the problem. Here are some strengths that I found:

  1. The additional computational cost is small. The method can be applied without changing the existing batch construction or sample selection strategy. Under the reported setup, the additional runtime is at most 0.8% that is negligible.
  2. The results show benefits beyond standard classification accuracy. Fig. 2 shows that the accuracy improvement over SimCLR becomes larger as the corruption severity increases. The method also improves misclassification detection when applied to NNCLR and All4One.
Critical weaknesses

While I enjoyed reading the paper, here I list some weaknesses of the paper:

  1. The gains in standard classification accuracy are modest. In the table on page 7, SimCLR improves from 65.80% to 66.79% on CIFAR-100, while NNCLR improves only from 75.4% to 75.5% on ImageNet-1k. In my opinion, the improvement is quite small in some settings, questioning the method's advantages and generalizability.
  2. The probabilistic interpretation needs clarification. The final objective is an ELBO-inspired surrogate rather than a strict ELBO. The paper also explains that the learned scale is better understood as a signal for ranking error likelihood than as an accurate estimate of variance. This distinction should be clear when interpreting what the method learns.
Questions for authors

With the weaknesses part above, please find the following questions:

  1. What is the main source of the improvement? uncertainty learning or geometric weighting? In Table 5, using a constant scale with only the geometric term gives nearly the same improvement in clustering quality. However, learned scales perform much better for error detection. Can the authors clarify which benefits come from learning uncertainty and which come from geometric weighting?
  2. Why do harder samples receive smaller scales? The method uses 1−u1-u for error detection. How does this relate to the earlier interpretation of the scale as uncertainty or variance? This connection needs further explanation.
  3. How sensitive is the method to the approximation assumptions? I would like to see a closer examination of the correlations induced by a shared anchor. Results with larger ViTs and other modalities would also help assess how broadly the method applies.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 1Ghc [Part 1/3]

Thank you for the careful review. Your questions led to several new analyses (in blue in the revised PDF):

  • Source of the improvement (Q1) → separated by new matched runs (Table 5c, App. B.11): the geometry gives the structural gains and part of the robustness gain, and the learned scale gives the uncertainty signal (and, in Table 5c, further robustness) (comment 2/3).
  • Probabilistic interpretation (W2) → §3.3 (surrogate), and Proposition 2 in §3.2, which gives the form of the temperature exactly at any concentration (comment 2/3).
  • Why hard views get smaller scales (Q2) → the gradient of Eq. B.1, written out and measured on the trained encoders (App. B.6 paragraph, comment 2/3).
  • Modest gains (W1) → the gains grow as labels become scarce (App. B.13) and hold at equal base temperature.
  • Shared-anchor correlations (Q3a) → App. B.9, computed exactly. Approximation regime (Q3b) → App. B.8 with an exact variance identity. ViT (Q3c) → new ViT-Small runs, 3 seeds each (comment 3/3).
  • The supplementary code is updated with the scripts for every new experiment.

W1. Gains. The clean-accuracy gains are consistent across all settings: every top-1 mean in Table 1 improves on its deterministic counterpart (9 of 9). The variance that includes pretraining is in the matched three-seed experiments (Table 5c, App. B.11). At the same base temperature, DistCLR is also more accurate than InfoNCE on corrupted data, at every severity, and with scarce labels (App. B.13). On ImageNet-1k (75.5 vs 75.4) DistCLR matches NNCLR, showing that the per-pair temperature scales without cost. The larger benefits appear beyond clean accuracy:

  • Distribution shift: at the same base temperature (Table 5c, τ_A = 0.185, frozen probe), DistCLR is more accurate on CIFAR-100-C than InfoNCE (49.52 vs 48.52, Welch p = .001, 3 seeds), and with ViT-Small it is ahead at every corruption severity (comment 3/3).
  • Label efficiency (App. B.13, frozen probe on 1%, 10% or all CIFAR-100 labels, 3 seeds): the gain over InfoNCE at the same base temperature grows from +0.62 with all labels to +1.34 with 10% (p = .01) and +1.71 with 1% (p = .04). With ViT-Small it reaches +3.4 points at 1% labels, higher on each seed. DistCLR is at least as accurate as its deterministic counterpart at every label budget in all five settings (SimCLR, NNCLR, All4One, ViT-Small, and SimCLR on CIFAR-10).
  • Open-set recognition (Table 4, strict CIFAR-10 6 known / 4 unknown, four standard splits, unknown classes excluded from pretraining):
Model, scoreAUROC ↑FPR95 ↓OSCR ↑
InfoNCE, MSP.876 ± .021.389 ± .064.846 ± .027
InfoNCE, energy.886 ± .025.386 ± .080.848 ± .027
InfoNCE, MSP+kNN.889 ± .029.371 ± .085.851 ± .028
DistCLR, MSP+U.905 ± .011.325 ± .012.871 ± .025

The fused score uses no fitted weights (mean rank of MSP and U) and varies less across splits.

  • An uncertainty signal from the encoder itself, which an InfoNCE encoder cannot provide (it has no per-sample scale), which needs no classifier at test time and which complements the classifier's confidence, as the MSP+U fusion above shows. It appears with every framework and backbone we tested: error AUROC .756 (SimCLR), .748 (NNCLR), .784 (All4One), .726 (ViT-Small) and .780 (SimCLR on CIFAR-10). The shared-scale control, whose head is trained by the prior alone, stays at ~.5, so the signal comes from learning u with the contrastive loss. U also outperforms the other label-free per-sample scores, the k-NN distance and the embedding and feature norms of InfoNCE and DistCLR encoders, for shift detection at every severity of CIFAR-100-C and on every seed (AUROC .690 vs at most .629 at severity 5, also on CIFAR-10-C and with ViT-Small) and for error ranking (.756 vs at most .733), and it ranks errors better than augmentation disagreement over 16 passes (.749 vs .681, App. B.2 and B.14). The per-sample uncertainties of TaU (its learned temperature) and MCInfoNCE reach at most .61 for error ranking and .55 for shift detection.

W1. Generalizability. Beyond the three frameworks and two architectures of the submission, the new ViT-Small runs show the same picture on a transformer backbone: DistCLR is more accurate on CIFAR-100-C at every severity, and adds the uncertainty signal (comment 3/3). The new matched runs on CIFAR-10 repeat the CIFAR-100 result on a second dataset: DistCLR is more accurate on the probe (89.12 vs 88.62, p = .02) and on CIFAR-10-C (82.12 vs 81.55, p = .04), with error AUROC .780 (App. B.11).

AuthorsOpenReview ↗

Response to Reviewer 1Ghc [Part 2/3]

W2. Thank you. The paper now states both points explicitly.

  • §3.3 states that omitting the decoder term defines "a decoder-free contrastive surrogate" and that this "precludes an exact ELBO interpretation": the method is ELBO-motivated.
  • On the scale, we checked Proposition 1 at the learned operating point (App. B.8) and derived the similarity variance exactly (comment 3/3). Proposition 2 (§3.2) gives the form of the temperature exactly at any concentration, with Proposition 1 as its small-noise limit, and u is learned per view. As you suggest, the paper reads U = 1 − u as a score of how hard a view is.

Q1. Which benefits come from which part (Table 5c and App. B.11, 3 seeds per arm, every arm at τ_A = 0.185, tables in our reply to Reviewer 5v1w, comments 2/3 and 3/3):

  • Geometric weighting (DistCLR with one shared scale) gives the structural gains (ARI and EffRank, Table 5b) and higher accuracy than InfoNCE on corrupted data (CIFAR-100-C 48.91 vs 48.52, p = .03).
  • Learned uncertainty raises that robustness further (49.52, p = .001 vs InfoNCE) and adds the error-ranking signal (AUROC .756, .743 at severity 5, vs ≈ .5 with a shared scale).
  • The error-ranking split holds in NNCLR, All4One, ViT-Small and on CIFAR-10 (App. B.11): only the learned scale ranks errors (.748 / .784 / .726 / .780 vs ≈ .5).

The abstract and Contributions 1, 2 and 4 are rewritten to state this split.

Q2. Why harder views receive smaller scales. The scale is the amount of softening a view can afford. From App. B.6 (Eq. B.1), with logits ℓ_j = s_j/υ_j − 1/(2Lυ_j²) and υ_j = τ_A + (u_r + u_j)(1 − s_j²):

∂L/∂u_r = (1 − p⁺)·(1 − s⁺²)(s⁺ − 1/(Lυ⁺))/υ⁺² − Σ_j p_j·(1 − s_j²)(s_j − 1/(Lυ_j))/υ_j².

  • The positive term is positive whenever s⁺ > 1/(Lυ⁺), so gradient descent lowers u_r: a partially aligned positive needs a sharper temperature to stand out among the negatives.
  • This term vanishes at perfect alignment and is largest at intermediate alignment, so partially aligned (hard) views are pushed hardest toward small u. Easily identified views keep their positive even at a softer temperature, so the prior lets their u stay higher.
  • Negatives raise u_r only through candidates with s_j > 1/(Lυ_j).
  • In NNCLR and All4One the anchor is a support-set neighbour whose scale carries no gradient, so a view's scale is trained only as the positive. Its gradient is then the first term alone, and the same conclusion holds (App. B.6).

The trained encoders show exactly this (paragraph in App. B.6, 3 seeds, 2,000 test images, one random crop per image and crop-area bin, no other augmentation):

Crop area80–100%60–80%40–60%20–40%8–20%
Cosine of the view to the full image.983.965.930.852.647
Mean u.0312.0310.0302.0280.0247

The harder the view (the less it agrees with its own full image), the smaller its scale, and across individual views Spearman(u, cosine to the full image) = .81 on every seed. Per image, U = 1 − u ranks the dispersion of the representation over 16 training augmentations with Spearman .69 on each seed (App. B.6).

A small u therefore marks a hard view and a large u a confident one, so U = 1 − u is the uncertainty score. As §4.5 states, the learned scale ranks uncertainty rather than estimating a calibrated variance. The two parts thus play complementary roles, which §4.5 now states: the learned scale gives hard views a sharper temperature and ranks their hardness, while ambiguous pairs are softened through (1 − s²).

AuthorsOpenReview ↗

Response to Reviewer 1Ghc [Part 3/3]

(a) Shared anchor (App. B.9). All logits of an anchor share its noise, which adds a covariance to each margin s_t − s_j. We computed it exactly for isotropic Gaussian noise, with no expansion, using the moments of Eq. B.2: Cov(⟨z_r, z_t⟩, ⟨z_r, z_j⟩) = μ_tμ_j[(m_r − μ_r²)s_rt s_rj + c_r(s_tj − s_rt s_rj)]. On 522,240 anchor–positive–negative triples (SimCLR augmentations of 2,048 test images, batches of 256) at the learned scales, twice this covariance is a median 0.8% of the sum of the two per-logit variances (95th percentile 3.3%), identically on three seeds. Direct sampling of the full vectors confirms the exact covariance (slope 0.99–1.00, correlation .999). Moderating each logit separately is therefore an accurate treatment.

(b) Regime of the approximation (App. B.8). On three checkpoints (256 test pairs × 16,384 Gaussian samples each), Eq. (11) overstates the sampled variance at the learned scale ((L − 1)u ≈ 7.9), while it orders each anchor's candidates as sampling does (median Spearman .96). With a 16-dimensional projector output the learned scale lies in the concentrated regime ((L − 1)u ≈ 0.43, Eq. (11) within a factor of 1.7 of the exact variance), and there DistCLR improves on InfoNCE on every metric and every seed (+0.76 to +1.31 points, p ≤ .014, App. B.8). We also derived the similarity variance exactly (Eq. B.2): Var⟨z_1, z_2⟩ = (B + C) + (A − B)(1 − s²) at any concentration, with Eq. (11) as its small-noise limit. §3.2 now presents the temperature τ_A + (u_r + u_t)(1 − s²) as a critic design whose shape Proposition 1 motivates and whose scale u learns, and the matched ablations (comment 2/3) establish its benefit. In line with your Q2, §3.2 also states that a harder view receives a smaller u, hence a sharper temperature (comment 2/3). The App. B.15 adds exact properties of the critic that hold at any operating point, including that the variance floor of Eq. B.2 is absorbed by τ_A.

(c) ViT. We trained ViT-Small (patch 4, about 21M parameters, twice ResNet-18's 11M) on CIFAR-100 with DistCLR and with InfoNCE, 3 seeds each, with the same recipe and 800 pretraining epochs (ViT-Small is still improving steeply at 400 epochs on CIFAR-100). DistCLR is more accurate on the frozen probe (61.03 ± 0.30 vs 60.55 ± 0.53). On CIFAR-100-C (frozen probe, mean over the 19 corruptions):

SeverityClean12345
InfoNCE60.5558.0355.5952.9049.8445.73
DistCLR61.0358.5456.1553.3950.3046.24
Error AUROC of U (DistCLR).726.730.731.731.729.729

DistCLR is ahead at every severity, and its uncertainty signal does not degrade under corruption (InfoNCE has no per-sample scale, AUROC .5). The method therefore carries over to a transformer backbone without changes. Scaling to larger ViTs at ImageNet scale is a natural next step that we list in App. A.1. At ImageNet scale, U also detects covariate shift on ImageNet-100-C without labels, ahead of the k-NN distance at severities 1–4 (AUROC .627 vs .597 at severity 1) and equal at 5 (2 seeds, App. B.14).

Other modalities. DistCLR changes only the pairwise score, so it plugs into other InfoNCE objectives (e.g. sentence or image–text contrastive learning) without changes, and we list their evaluation as future work (App. A.1).

Thank you again for the careful review. We hope the separation of geometry and learned scale, the explanation and measurement of the scale's behaviour, and the new exact approximation analyses address your questions. If they do, we would be grateful if you would consider updating your score. If anything still stands in the way, we would be glad to address it within the discussion period.

Reviewer_yqLPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes an approach to model the uncertainties in representations and embedding similarities. The paper introduces DistCLR, a new contrastive framework derived from the evidence lower bound (ELBO) that models a per-view scale that yields a pair-specific temperature. It can be plugged into existing contrastive frameworks without inducing noticeable wall-clock overhead. The proposed method matches or improves classification, strict open-set recognition, robustness under corrupted data, and other tasks.

Critical strengths
  1. Clear motivation for embracing the uncertainty in contrastive learning's pairwise similarities through the ELBO. Supported by thorough mathematical derivations. The proposed loss can be easily adopted in existing contrastive frameworks.
  2. Clean setup: uses a commonly known SSL framework, training parameters follow existing studies, evaluations are multi-seed, and the code is published for reproducibility.
  3. Extensive analysis on common datasets and metrics in contrastive learning. The proposed method consistently matches or improves the previous methods' performance. In addition, the learned scale provides a usable uncertainty signal that deterministic InfoNCE lacks: it ranks linear-probe errors well above a constant scale and improves open-set recognition over the best InfoNCE-based score with lower variance across splits.
  4. Transparency in the ablation about geometry driving the effect and the scale ranking rather than estimating uncertainty.
Critical weaknesses
  1. Global temperature replacement: One of the contributions stated in the paper is replacing the global temperature τ\tau and adapting contrastive scores to per-view uncertainty and pairwise geometry. However, the proposed method introduces three hyper-parameters (σp\sigma_p, τA\tau_A, and λprior\lambda_{prior}), which are adjusted based on the global temperature (Hyper-parameters sub-section). In particular, τA\tau_A is tuned so that E[vr,t]≈τd\mathbb{E}[v_{r,t}] \approx \tau_d, so the method still requires the tuned global temperature τd\tau_d as input and modulates around it rather than replacing it. Moreover, a sensitivity analysis over λprior\lambda_{prior} and σp\sigma_p is needed to justify the chosen values.
  2. Marginal accuracy gains: While the proposed method brings improvements across datasets and frameworks, the gains are marginal: +0.1–1.0 on top-1, and the NNCLR gains are within noise (e.g. 91.54 vs 91.51 on CIFAR-100 top-5). Moreover, no std is reported in Tables 2, 3, and 5a.
  3. Attribution of the representation gains: The ablation (Table 5b) and the paper's own text indicate that geometry drives most of the structural effect: a constant per-view scale with the (1−s2)(1-s^2) factor already reproduces the full method's ARI and effective rank within one std, while the learned scale mainly contributes the error-ranking signal (error AUROC 0.7490.749 vs 0.5020.502). However, Contributions 2 and 4 place the emphasis on per-view uncertainty. It would strengthen the paper to align the contributions with this finding, to include the constant-scale variant as a baseline in the main tables, and to briefly relate the (1−s2)(1-s^2) reweighting to existing adaptive-temperature or hard-pair contrastive objectives.
Questions for authors
  1. Which base methods are used in Tables 3 and 5b, and why does Table 5b's InfoNCE EffRank differ from Table 4?
  2. Were the 12 baselines in Table 1 rerun or taken from solo-learn? Why does the All4One number appear to be lower than in the original paper?
6 comments
AuthorsOpenReview ↗

Response to Reviewer yqLP [Part 1/4]

Thank you for the constructive review and for the concrete suggestions, which we have adopted throughout (in blue in the revised PDF):

RequestDone
Sensitivity to λ_prior and σ_p² (W1)App. B.10 (3 seeds per cell): online and linear accuracy within half a point over 10× λ_prior and 4× σ_p². τ_A (3×) behaves like InfoNCE's temperature, best at the default
Stds (W2)Table 5 now mean ± std over 3 seeds, and Table 5a rerun with stds (comment 2/4)
Size of the gains (W2)Gains per Table 1 cell, matched three-seed controls, and gains that grow as labels become scarce (App. B.13, comment 2/4)
Align contributions with the geometry/scale finding (W3)Contributions 2 and 4 rewritten (comment 3/4)
Shared-scale variant in the main tables (W3)Table 5c (SimCLR) and App. B.11 (NNCLR, All4One, ViT-Small, CIFAR-10), 3 seeds each
Relation to adaptive-temperature / hard-pair objectives (W3)§2 paragraph, plus cosine schedules and HCL run matched (App. B.12, comment 3/4)
Q1, Q2Comment 4/4
ReproducibilitySupplementary code updated with the scripts for every new experiment

W1. Global temperature. Thank you for this precise reading. DistCLR keeps the global temperature τ_A as its base level, and the similarity-uncertainty term makes it pair-specific. The abstract now says the per-pair similarity uncertainty "augments the global temperature", and Contribution 2 is worded accordingly. The hyperparameter paragraph (§4) now gives the exact rule: τ_A is chosen in (τ_d − σ_p², τ_d), σ_p² is set from τ_d (τ_d/10 as a rule), and λ_prior = 1 except 0.05 for NNCLR on CIFAR, the value in the released configuration. App. B.7 (Table 10) lists every setting, including the learning-rate multiplier of the scale head.

Sensitivity (App. B.10, Table 12, CIFAR-100 SimCLR, 1,000 epochs, mean ± std over 3 seeds per cell, bold = default). Online: online linear accuracy, last 10 epochs. Linear: standard 100-epoch linear evaluation. AUROC: error AUROC of U on the frozen probe.

τ_A0.100.150.1850.250.30
Online acc.65.72 ± 0.4366.39 ± 0.3766.49 ± 0.2166.31 ± 0.4165.73 ± 0.23
Linear acc.66.16 ± 0.4966.82 ± 0.6867.04 ± 0.1666.85 ± 0.2665.94 ± 0.19
Error AUROC of U.733 ± .004.750 ± .003.756 ± .003.755 ± .003.748 ± .001
λ_prior 0.313σ_p² 0.010.020.04
Online acc.66.70 ± 0.2266.49 ± 0.2166.40 ± 0.4266.29 ± 0.0866.49 ± 0.2166.54 ± 0.28
Linear acc.66.97 ± 0.1967.04 ± 0.1667.13 ± 0.6966.70 ± 0.3767.04 ± 0.1667.04 ± 0.15
Error AUROC of U.760 ± .001.756 ± .003.755 ± .004.756 ± .006.756 ± .003.762 ± .002
  • The two parameters you asked about matter little: over 10× λ_prior and 4× σ_p², online and linear accuracy each vary by less than half a point (66.29–66.70 and 66.70–67.13). τ_A behaves like the temperature of InfoNCE, with its best value at the default on all three metrics and a smooth decline on both sides. The error AUROC of U stays within .733–.762 over all nine settings, so the uncertainty signal does not depend on tuning.
  • The defaults are not the best cells (λ_prior = 0.3 and σ_p² = 0.04 are higher in online accuracy and error AUROC, λ_prior = 3 in linear accuracy), so they were not tuned to these results. σ_p² = 0.04 gives the same linear accuracy as the default (67.04) with higher probe accuracy (59.41 vs 58.78), CIFAR-100-C accuracy (49.64 vs 49.52) and error AUROC (.762 vs .756).
AuthorsOpenReview ↗

Response to Reviewer yqLP [Part 2/4]

Std. All of Table 5 now reports mean ± std over 3 seeds. Table 5a now reports the mean over 3 pretraining seeds, evaluated with one script:

L_U-InfoNCEL_priorProbe (L-BFGS)ARIEffRank
65.54 ± 0.2616.08 ± 0.88192.3 ± 0.4
✓65.79 ± 0.4716.78 ± 0.89191.6 ± 1.4
✓65.40 ± 0.3716.20 ± 0.23191.3 ± 0.6
✓✓65.84 ± 0.5517.48 ± 0.43193.5 ± 1.2

The full objective has the highest value on all three metrics (Table 5b keeps the evaluation of the submission, so its absolute ARI values differ from Table 5a's). Table 2 baselines are published single runs. On ImageNet-1k DistCLR reaches 75.5 (NNCLR 75.4). Table 3 reports means over three seeds.

Consistency. Every top-1 mean in Table 1 improves on its deterministic counterpart (9 of 9 settings). The variance that includes pretraining is in the matched three-seed experiments (Table 5c, App. B.11), where DistCLR is ahead of the deterministic objective in all five settings on corrupted data. At the same base temperature, DistCLR is also more accurate than InfoNCE on corrupted data, at every severity, and with scarce labels (App. B.13).

Top-1 gainCIFAR-10CIFAR-100ImageNet-100
SimCLR+0.31+0.99+0.48
NNCLR+0.11+0.13+0.19
All4One+0.31+0.26+0.31

Following your comment, §4.1 now says that DistCLR "matches or improves on" its deterministic counterpart, abstract claim (2) speaks of "small but consistent gains in neighborhood consistency, clustering, and cross-domain transfer", and the revision puts the emphasis on the benefits below.

Where the gains are. Beyond clean accuracy (summarised in §4.4, details in App. B.13 and B.14):

  • Under distribution shift: at the same base temperature (Table 5c, frozen probe), both DistCLR variants are more accurate on CIFAR-100-C than InfoNCE (48.91 with a shared scale and 49.52 with learned scales vs 48.52, Welch p = .03 and .001, 3 seeds each), and with ViT-Small DistCLR is ahead at every corruption severity (App. B.11).
  • With scarce labels (App. B.13, frozen probe on 1%, 10% or all CIFAR-100 labels, 3 seeds per arm): in the matched setting of Table 5c, the gain over InfoNCE grows from +0.62 with all labels to +1.34 with 10% (p = .01) and +1.71 with 1% (p = .04). With ViT-Small it is +3.4 points at 1% labels, higher on each seed. On CIFAR-10 the 1%-label gain is +0.97 (p = .05). DistCLR is at least as accurate as its deterministic counterpart at every label budget in SimCLR, NNCLR, All4One, ViT-Small and on CIFAR-10.
  • An uncertainty signal from the encoder itself, which an InfoNCE encoder cannot provide, which needs no classifier at test time and which complements the classifier's confidence: it ranks errors (AUROC .756, still .743 at severity 5) at every hyperparameter setting we tested (.733–.762, 3 seeds each), and it detects covariate shift better than every other label-free score we tested, the k-NN distance and the embedding and feature norms of InfoNCE and DistCLR encoders, at every severity (AUROC .690 vs at most .629 at CIFAR-100-C severity 5, also on CIFAR-10-C and with ViT-Small, App. B.14).
  • Open-set recognition (Table 4): MSP+U reaches AUROC .905 and FPR95 .325, against .889 and .371 for the best InfoNCE score, with lower variance across splits.
AuthorsOpenReview ↗

Response to Reviewer yqLP [Part 3/4]

Thank you for this suggestion. Adopting it made the paper's message clearer.

Contributions. Contribution 4, quoted from the revised paper:

"Matched ablations separate the two parts of the temperature. Both contribute: the similarity-uncertainty geometry (1−s²) improves semantic organization (clustering and effective rank) and robustness over InfoNCE, and learned per-view scales add a signal that ranks errors and unknown inputs."

Contribution 2 now states that, even with a single shared per-view scale, the temperature remains pair-dependent through the geometric factor (1−s²).

Shared-scale variant in the main tables. Table 5c now reports InfoNCE, DistCLR with one shared scale and DistCLR with learned scales (CIFAR-100 SimCLR, every arm at τ_A = 0.185, 3 seeds per arm in one code path), and App. B.11 repeats the comparison in NNCLR, All4One, ViT-Small and on CIFAR-10. Both tables are in our reply to Reviewer 5v1w (comments 2/3 and 3/3). Both DistCLR arms are more accurate than InfoNCE on CIFAR-100-C (+0.39 and +1.00, p = .03 and .001), and only the learned scale ranks errors (AUROC .73 to .78 vs ≈ .5).

Relation to adaptive-temperature and hard-pair objectives. The new §2 paragraph relates (1−s²) to hardness-aware weighting (Wang & Liu 2021; HCL) and adaptive temperatures (Kukleva et al. 2023; Huang et al. 2023), and names the differences: the form is derived from the similarity variance, one expression acts on positives and negatives, and it carries the learned scale that gives the uncertainty signal. Table 5b also contains a learned per-view temperature without the geometry ("pointwise"): it has lower effective rank (185.3 vs 195.5) and almost no error signal (.540 vs .749).

Adaptive global temperatures, matched (App. B.12, Table 5c recipe, 3 seeds each). The strongest adaptive global temperature, the cosine schedule of Kukleva et al. (2023) centred on the tuned temperature (τ 0.1–0.3), reaches 66.26 ± 0.10 linear accuracy, against 67.04 ± 0.16 for DistCLR. Only DistCLR gives a per-sample signal. App. B.12 also reports the schedule's published settings and HCL, which at the matched τ = 0.2 reaches 56.9 online (β = 0.5) and at its published τ = 0.5 stays below DistCLR.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment to Area Chair and Reviewers

We thank all reviewers. Their questions led to the main additions of the revision (in blue, with a script in the supplementary code for every new experiment). The method, its implementation and all Table 1 results are unchanged, and the revision adds the analyses and controls the reviewers asked for. In brief:

  • Theory (5v1w, 1Ghc): the variance of the cosine similarity is now derived exactly at any noise level (Proposition 2, verified by sampling within 0.5%). It is affine in (1 − s²), the form of DistCLR's temperature, and Proposition 1 is its small-noise limit.
  • Matched controls (all reviewers, same code, same base temperature, 3 seeds): DistCLR's temperature is more accurate than InfoNCE on CIFAR-100-C with one shared scale (+0.39, p = .03) and with learned scales (+1.00, p = .001, ahead at every severity), and more label-efficient (+1.34 with 10% of the labels, p = .01, and +1.71 with 1%, p = .04). It stays ahead of adaptive global temperatures (cosine schedules) in online, linear and corrupted-data accuracy.
  • Learned scales: a label-free uncertainty signal that a head trained afterwards on a frozen InfoNCE encoder does not reach (error AUROC .756 vs .707), present in every framework and backbone tested (error AUROC .73–.78 vs ≈ .5 with a shared scale), covariate-shift and unseen-class detection ahead of the k-NN distance (CIFAR-100-C, CIFAR-10-C, ViT-Small, ImageNet-100-C at severities 1–4, and CIFAR-10 vs CIFAR-100) and of feature norms (CIFAR-100-C), prediction of accuracy on shifted data without its labels (Spearman −.96 over 96 clean and CIFAR-100-C sets), and the best open-set recognition when fused with the probe's confidence.
  • Generality: the same three-arm comparison holds in NNCLR and All4One (deterministic arms at their released τ_d = 0.2), ViT-Small and on CIFAR-10. Every Table 1 setting improves on its deterministic counterpart, and DistCLR matches NNCLR on ImageNet-1k.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.