- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper addresses the difficulty of tuning temperature hyperparameters in multimodal contrastive learning (CL). By modeling the joint representation distribution as a multivariate Gaussian, the authors establish a mathematical link connecting contrastive temperature to cross-modal correlation coefficients: for two modalities, and elements of the inverse correlation matrix for modalities. Based on this, they introduce -InfoNCE and -Muscle, along with CorrCLR, an alternating optimization algorithm that iteratively estimates empirical correlation matrices and updates modality encoders. Validations on 3D point-image-text, video-audio-text, and RF sensing (mmWave-WiFi-RFID) demonstrate improvements over fixed/adaptive temperature baselines and pairwise alignment methods.
Critical strengths
-
Bridging contrastive density ratios under a Gaussian prior to representation correlation coefficients turns an uninterpretable heuristic hyperparameter () into a physically meaningful, measurable quantity ().
-
Effective scaling to multi-modality (): Joint multimodal losses (like Muscle loss) suffer from a quadratic explosion of temperature parameters. CorrCLR bypasses grid searches over temperature combinations via empirical matrix inversion while theoretically guaranteeing positive definiteness and numerical invertibility.
Critical weaknesses
-
Strong assumptions on representation distribution: The derivations strictly rely on representations following a zero-mean multivariate Gaussian with an isotropic block covariance . However, CL embeddings normalized to the unit hypersphere typically follow directional distributions (e.g., von Mises-Fisher), and assuming uniform correlation across all dimensions is a heavy simplification.
-
Computational overhead in frequent representation caching: Updating the empirical correlation matrix requires periodic forward passes across training subsets ( samples) and feature standardization. Although the paper argues that low works, the forward caching still breaks the standard end-to-end streaming training pipeline of modern large-scale contrastive learning (e.g., CLIP-scale pre-training).
Questions for authors
-
Dimensional isotropy assumption: How does the performance degrade if cross-modal correlations are highly anisotropic across embedding dimensions, and has the empirical variance across channels been inspected in larger vision-language models?
-
Computational scalability on web-scale data: How does CorrCLR scale to web-scale datasets (e.g., DataComp/LAION with millions of pairs)? Does running external forward passes to compute every few epochs introduce memory/caching bottlenecks compared to standard in-batch contrastive estimators?
-
Comparison with learnable temperature: Why was a learnable temperature baseline (as used in CLIP/OpenShape) not included across all experimental settings?
-
Hypersphere vs. Gaussian geometry: Given that embeddings are -normalized onto a unit hypersphere, what theoretical justification guarantees that density ratios derived from unconstrained Euclidean Gaussians remain strictly valid after normalization? How about formulating an equivalent temperature relationship directly on the hypersphere using von Mises-Fisher (vMF) distributions or matrix Bingham distributions?
2 comments
Response to Reviewer gKZU - Part 2
Q4. Hypersphere vs. Gaussian geometry: Given that embeddings are -normalized onto a unit hypersphere, what theoretical justification guarantees that density ratios derived from unconstrained Euclidean Gaussians remain strictly valid after normalization? How about formulating an equivalent temperature relationship directly on the hypersphere using von Mises-Fisher (vMF) distributions or matrix Bingham distributions?
Yi et al. in [R3] have shown that the convergent optimal representations of multimodal contrastive learning follow a uniform distribution on a -dimensional unit hypersphere. Furthermore, Betser et al. (ICLR 2026) have shown that alignment leads to a constrained uniformity problem on the hypersphere, which, combined with norm concentration, yields Gaussian structure for both normalized (unit-norm) and unnormalized representations. Following these theoretical works, our assumption is valid even after normalization.
As discussed in Appendix M (Lines 1396–1398), there is a close connection between the uniform distribution on the unit hypersphere and the Gaussian distribution as the representation dimension increases. Additionally, Nicollier et al. in [R4] have shown that, for sufficiently large , the characteristic functions of the uniform distribution on the unit hypersphere and the Gaussian distribution are very similar, making the Gaussian approximation applicable when .
Formulating an equivalent temperature relationship directly on the unit hypersphere using directional statistics, as suggested by the reviewer, has already been discussed in detail in Appendix M (Lines 1389–1401) as a promising direction for future research. We believe that CorrCLR provides valuable insights into how to extend this framework to more complex distributions in directional statistics to establish relationships between temperature parameters and concentration parameters.
[R3] L. Yi, R. Douady, and C. Chen, "The convergent representation of contrastive vision-language models: Geometry, modality gap and shared space slignment," ICML, 2026.
[R4] L. Nicollier, M. Dunitz, M. Pic, P. Musé, E. Meinhardt-Llopis, and G. Facciolo, "SPHERE-JEPA: Spherical prediction with homogeneous embeddings", NeurIPS, 2026.
Response to Reviewer gKZU - Part 1
We thank the reviewer for their comments. Please find our responses below:
Q1. Dimensional isotropy assumption: How does the performance degrade if cross-modal correlations are highly anisotropic across embedding dimensions, and has the empirical variance across channels been inspected in larger vision-language models?
We would like to emphasize that the considered covariance structure is a practical and simplifying assumption that performs well across different settings, as demonstrated by our performance evaluation. Furthermore, this assumption is examined and justified through our analysis in Appendix B.
Making such practical and simplifying assumptions is common in self-supervised learning. For example, well-known approaches such as Barlow Twins [R1] and VICReg [R2] assume an isotropic covariance structure. Following the same reasoning, one could also question the choice of the number of temperature parameters used in InfoNCE and the Muscle loss function. This consideration motivated approaches such as iSogCLR (ICML 2023), which extends the InfoNCE formulation by introducing distinct temperature parameters for different samples.
The question raised by the reviewer has already been discussed in detail in Appendix M (Lines 1402–1415) as a promising direction for future research inspired by our newly proposed contrastive loss formulations. In particular, our formulation suggests that, in a bimodal setting, the number of correlation coefficients (equivalently, temperature parameters) can scale with the dimensionality of the representations d. This offers a fundamentally different perspective from previous works, such as iSogCLR, which suggests scaling the number of temperature parameters with the number of training samples.
[R1] J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny, "Barlow Twins: Self-supervised learning via redundancy reduction," ICML, 2021.
[R2] A. Bardes, J. Ponce, and Y. LeCun, "VICReg: Variance-invariance-covariance regularization for self-supervised learnin", ICLR, 2022.
Q2. Computational scalability on web-scale data: How does CorrCLR scale to web-scale datasets (e.g., DataComp/LAION with millions of pairs)? Does running external forward passes to compute every few epochs introduce memory/caching bottlenecks compared to standard in-batch contrastive estimators?
CorrCLR introduces periodic forward passes to estimate the cross-modal correlation matrix. However, these passes are performed only every T epochs and can operate on a subset containing a fraction r of the training data.
As already analyzed in Appendix E, the additional computational cost relative to standard contrastive training is approximately 3Tr. Thus, the overhead can be controlled through the sampling ratio and update interval. For example, with r=0.2 and T=5, the additional FLOPs are approximately 1.33%. This illustrates that correlation estimation does not necessarily require a substantial computational overhead, even when the training dataset is large.
Regarding memory consumption, the implementation described in Appendix E temporarily stores the representations of the selected samples to compute feature-wise means and standard deviations, which are then used to compute the correlation matrix. We acknowledge that caching all selected representations could introduce a memory bottleneck when processing millions of samples, particularly if the representations are retained in GPU memory. Importantly, however, this is an implementation choice rather than an intrinsic requirement of CorrCLR. A more memory-efficient implementation can estimate the feature-wise statistics incrementally and accumulate the cross-modal correlation statistics in a subsequent pass, avoiding the need to retain the entire set of representations simultaneously.
The computational analysis and the flexibility provided by r and T suggest that CorrCLR can be adapted to larger datasets without requiring correlation estimation over the entire training set at every epoch.
Q3. Comparison with learnable temperature: Why was a learnable temperature baseline (as used in CLIP/OpenShape) not included across all experimental settings?
In the 3D point-image-text setting, we compare against ULIP and OpenShape, both of which employ pairwise InfoNCE with learnable temperature parameters. We included these baselines in this setting because learnable temperatures were investigated and evaluated in the original ULIP and OpenShape papers. Furthermore, iSogCLR (ICML 2023) is another learnable-temperature baseline that has already been included in all experimental settings.