Embrace Ambiguity: Rethinking Contrastive Learning through Similarity Uncertainty

Contrastive learning has become a dominant paradigm for self-supervised representation learning, but two key limitations persist: (1) embedding similarities are treated as deterministic, causing ambiguous pairs to be optimized with unwarranted confidence and potentially distorting fine-grained semantic structure; and (2) prior probabilistic approaches model uncertainty in individual representations or learn temperatures heuristically, leaving the uncertainty of the pairwise similarity itself, which directly governs contrastive objectives, unmodeled. To address these limitations, we propose Distributional Contrastive Learning (DistCLR), a self-supervised framework that models both representations and pairwise similarities distributionally, inducing similarity uncertainty in closed form from the underlying representation posteriors. Motivated by an evidence lower bound (ELBO), DistCLR formulates an uncertainty-aware InfoNCE objective in which the global temperature is replaced by this per-pair similarity uncertainty, softening ambiguous pairs while preserving sharp alignment for confident ones. DistCLR changes only the pairwise contrastive score: it requires no Monte Carlo sampling, adds less than 1% wall-clock overhead, recovers InfoNCE under homoscedastic uncertainty, and plugs into existing InfoNCE-based frameworks without altering their training pipelines. Experiments on CIFAR-10, CIFAR-100, ImageNet-100, and ImageNet-1k show that DistCLR (1) matches or improves classification performance across three contrastive frameworks and two architectures, (2) induces clearer semantic organization in the representation space, as evidenced by neighborhood consistency, effective rank, clustering, and cross-domain transfer assessments, (3) improves strict open-set recognition, robustness under corrupted data, and misclassification detection, and (4) outperforms representative uncertainty-aware contrastive methods in both linear-probe accuracy and error detection. Code is provided in the supplementary material.