CorrCLR: Correlation-based Contrastive Learning of Representations

Contrastive learning (CL) is a central self-supervised paradigm that has driven major advances in multimodal representation learning. However, its performance is highly sensitive to the temperature hyperparameter in loss functions such as InfoNCE, and the literature lacks a systematic approach that generalizes across diverse settings. In this work, we establish a fundamental relationship between the temperature hyperparameter and the correlation coefficient of multimodal representations. Leveraging this insight, we reformulate contrastive loss functions and introduce their correlation-based counterparts. While temperature hyperparameters are difficult to tune, correlation coefficients provide an interpretable and measurable alternative. Consequently, we propose CorrCLR, a novel algorithm that learns multimodal representations by dynamically updating correlation coefficients as the representations evolve. Experiments across diverse multimodal settings, including 3D point-image-text, video-audio-text, and mmWave-WiFi-RFID, demonstrate that CorrCLR delivers robust performance improvements over state-of-the-art baselines across various downstream tasks, such as zero-shot classification and retrieval.