acceptodds
Withdrawn from ICLR 2027

CorrCLR: Correlation-based Contrastive Learning of Representations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Contrastive learning (CL) is a central self-supervised paradigm that has driven major advances in multimodal representation learning. However, its performance is highly sensitive to the temperature hyperparameter in loss functions such as InfoNCE, and the literature lacks a systematic approach that generalizes across diverse settings. In this work, we establish a fundamental relationship between the temperature hyperparameter and the correlation coefficient of multimodal representations. Leveraging this insight, we reformulate contrastive loss functions and introduce their correlation-based counterparts. While temperature hyperparameters are difficult to tune, correlation coefficients provide an interpretable and measurable alternative. Consequently, we propose CorrCLR, a novel algorithm that learns multimodal representations by dynamically updating correlation coefficients as the representations evolve. Experiments across diverse multimodal settings, including 3D point-image-text, video-audio-text, and mmWave-WiFi-RFID, demonstrate that CorrCLR delivers robust performance improvements over state-of-the-art baselines across various downstream tasks, such as zero-shot classification and retrieval.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Contrastive learning, correlation coefficient, InfoNCE, multimodal representation alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gKZUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses the difficulty of tuning temperature hyperparameters in multimodal contrastive learning (CL). By modeling the joint representation distribution as a multivariate Gaussian, the authors establish a mathematical link connecting contrastive temperature to cross-modal correlation coefficients: τ=(1−ρ2)/ρ\tau = (1-\rho^2)/\rho for two modalities, and elements of the inverse correlation matrix P−1P^{-1} for M>2M>2 modalities. Based on this, they introduce ρ\rho-InfoNCE and ρ\rho-Muscle, along with CorrCLR, an alternating optimization algorithm that iteratively estimates empirical correlation matrices and updates modality encoders. Validations on 3D point-image-text, video-audio-text, and RF sensing (mmWave-WiFi-RFID) demonstrate improvements over fixed/adaptive temperature baselines and pairwise alignment methods.

Critical strengths
  1. Bridging contrastive density ratios under a Gaussian prior to representation correlation coefficients turns an uninterpretable heuristic hyperparameter (τ\tau) into a physically meaningful, measurable quantity (ρ\rho).

  2. Effective scaling to multi-modality (M>2M > 2): Joint multimodal losses (like Muscle loss) suffer from a quadratic explosion of temperature parameters. CorrCLR bypasses grid searches over (M−1)2(M-1)^2 temperature combinations via empirical matrix inversion while theoretically guaranteeing positive definiteness and numerical invertibility.

Critical weaknesses
  1. Strong assumptions on representation distribution: The derivations strictly rely on representations following a zero-mean multivariate Gaussian with an isotropic block covariance Σ=P⊗Id\Sigma = P \otimes I_d. However, CL embeddings normalized to the unit hypersphere typically follow directional distributions (e.g., von Mises-Fisher), and assuming uniform correlation across all dd dimensions is a heavy simplification.

  2. Computational overhead in frequent representation caching: Updating the empirical correlation matrix requires periodic forward passes across training subsets (rNrN samples) and feature standardization. Although the paper argues that low rr works, the forward caching still breaks the standard end-to-end streaming training pipeline of modern large-scale contrastive learning (e.g., CLIP-scale pre-training).

Questions for authors
  1. Dimensional isotropy assumption: How does the performance degrade if cross-modal correlations are highly anisotropic across embedding dimensions, and has the empirical variance across channels been inspected in larger vision-language models?

  2. Computational scalability on web-scale data: How does CorrCLR scale to web-scale datasets (e.g., DataComp/LAION with millions of pairs)? Does running external forward passes to compute P^\hat{P} every few epochs introduce memory/caching bottlenecks compared to standard in-batch contrastive estimators?

  3. Comparison with learnable temperature: Why was a learnable temperature baseline (as used in CLIP/OpenShape) not included across all experimental settings?

  4. Hypersphere vs. Gaussian geometry: Given that embeddings zz are ℓ2\ell_2-normalized onto a unit hypersphere, what theoretical justification guarantees that density ratios derived from unconstrained Euclidean Gaussians remain strictly valid after normalization? How about formulating an equivalent temperature relationship directly on the hypersphere using von Mises-Fisher (vMF) distributions or matrix Bingham distributions?

2 comments
AuthorsOpenReview ↗

Response to Reviewer gKZU - Part 1

We thank the reviewer for their comments. Please find our responses below:


Q1. Dimensional isotropy assumption: How does the performance degrade if cross-modal correlations are highly anisotropic across embedding dimensions, and has the empirical variance across channels been inspected in larger vision-language models?

We would like to emphasize that the considered covariance structure is a practical and simplifying assumption that performs well across different settings, as demonstrated by our performance evaluation. Furthermore, this assumption is examined and justified through our analysis in Appendix B.

Making such practical and simplifying assumptions is common in self-supervised learning. For example, well-known approaches such as Barlow Twins [R1] and VICReg [R2] assume an isotropic covariance structure. Following the same reasoning, one could also question the choice of the number of temperature parameters used in InfoNCE and the Muscle loss function. This consideration motivated approaches such as iSogCLR (ICML 2023), which extends the InfoNCE formulation by introducing distinct temperature parameters for different samples.

The question raised by the reviewer has already been discussed in detail in Appendix M (Lines 1402–1415) as a promising direction for future research inspired by our newly proposed contrastive loss formulations. In particular, our formulation suggests that, in a bimodal setting, the number of correlation coefficients (equivalently, temperature parameters) can scale with the dimensionality of the representations dd. This offers a fundamentally different perspective from previous works, such as iSogCLR, which suggests scaling the number of temperature parameters with the number of training samples.

[R1] J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny, "Barlow Twins: Self-supervised learning via redundancy reduction," ICML, 2021.

[R2] A. Bardes, J. Ponce, and Y. LeCun, "VICReg: Variance-invariance-covariance regularization for self-supervised learnin", ICLR, 2022.


Q2. Computational scalability on web-scale data: How does CorrCLR scale to web-scale datasets (e.g., DataComp/LAION with millions of pairs)? Does running external forward passes to compute every few epochs introduce memory/caching bottlenecks compared to standard in-batch contrastive estimators?

CorrCLR introduces periodic forward passes to estimate the cross-modal correlation matrix. However, these passes are performed only every TT epochs and can operate on a subset containing a fraction rr of the training data.

As already analyzed in Appendix E, the additional computational cost relative to standard contrastive training is approximately r3T\frac{r}{3T}. Thus, the overhead can be controlled through the sampling ratio and update interval. For example, with r=0.2r=0.2 and T=5T=5, the additional FLOPs are approximately 1.33%. This illustrates that correlation estimation does not necessarily require a substantial computational overhead, even when the training dataset is large.

Regarding memory consumption, the implementation described in Appendix E temporarily stores the representations of the selected samples to compute feature-wise means and standard deviations, which are then used to compute the correlation matrix. We acknowledge that caching all selected representations could introduce a memory bottleneck when processing millions of samples, particularly if the representations are retained in GPU memory. Importantly, however, this is an implementation choice rather than an intrinsic requirement of CorrCLR. A more memory-efficient implementation can estimate the feature-wise statistics incrementally and accumulate the cross-modal correlation statistics in a subsequent pass, avoiding the need to retain the entire set of representations simultaneously.

The computational analysis and the flexibility provided by rr and TT suggest that CorrCLR can be adapted to larger datasets without requiring correlation estimation over the entire training set at every epoch.


Q3. Comparison with learnable temperature: Why was a learnable temperature baseline (as used in CLIP/OpenShape) not included across all experimental settings?

In the 3D point-image-text setting, we compare against ULIP and OpenShape, both of which employ pairwise InfoNCE with learnable temperature parameters. We included these baselines in this setting because learnable temperatures were investigated and evaluated in the original ULIP and OpenShape papers. Furthermore, iSogCLR (ICML 2023) is another learnable-temperature baseline that has already been included in all experimental settings.


AuthorsOpenReview ↗

Response to Reviewer gKZU - Part 2

Q4. Hypersphere vs. Gaussian geometry: Given that embeddings are l2l_2-normalized onto a unit hypersphere, what theoretical justification guarantees that density ratios derived from unconstrained Euclidean Gaussians remain strictly valid after normalization? How about formulating an equivalent temperature relationship directly on the hypersphere using von Mises-Fisher (vMF) distributions or matrix Bingham distributions?

Yi et al. in [R3] have shown that the convergent optimal representations of multimodal contrastive learning follow a uniform distribution on a dd-dimensional unit hypersphere. Furthermore, Betser et al. (ICLR 2026) have shown that alignment leads to a constrained uniformity problem on the hypersphere, which, combined with norm concentration, yields Gaussian structure for both normalized (unit-norm) and unnormalized representations. Following these theoretical works, our assumption is valid even after normalization.

As discussed in Appendix M (Lines 1396–1398), there is a close connection between the uniform distribution on the unit hypersphere and the Gaussian distribution as the representation dimension dd increases. Additionally, Nicollier et al. in [R4] have shown that, for sufficiently large dd, the characteristic functions of the uniform distribution on the unit hypersphere and the Gaussian distribution are very similar, making the Gaussian approximation applicable when d>256d>256.

Formulating an equivalent temperature relationship directly on the unit hypersphere using directional statistics, as suggested by the reviewer, has already been discussed in detail in Appendix M (Lines 1389–1401) as a promising direction for future research. We believe that CorrCLR provides valuable insights into how to extend this framework to more complex distributions in directional statistics to establish relationships between temperature parameters and concentration parameters.

[R3] L. Yi, R. Douady, and C. Chen, "The convergent representation of contrastive vision-language models: Geometry, modality gap and shared space slignment," ICML, 2026.

[R4] L. Nicollier, M. Dunitz, M. Pic, P. Musé, E. Meinhardt-Llopis, and G. Facciolo, "SPHERE-JEPA: Spherical prediction with homogeneous embeddings", NeurIPS, 2026.

Reviewer_cSPwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem of tuning the temperature hyperparameter in contrastive learning by proposing a theoretical framework that replaces temperature with an interpretable correlation coefficient. Under a multivariate Gaussian assumption, the authors derive the relationship between the InfoNCE temperature and the correlation coefficient, τ=(1−ρ2)/ρ\tau=(1-\rho^2)/\rho, and introduce ρ\rho-InfoNCE for the two-modality case and ρ\rho-Muscle for the case of three or more modalities. They propose CorrCLR, an algorithm that iteratively estimates the correlation coefficients, and show that it outperforms existing methods across three heterogeneous multimodal settings.

Critical strengths
  • Prior work has treated the temperature parameter inconsistently across papers, with little discussion of the rationale behind choosing a fixed value, a learnable parameter, or an adaptive scheme. By formulating the temperature through the correlation coefficient, this paper provides a clear interpretation for a parameter that has largely relied on heuristics. In particular, the relationship τ=(1−ρ2)/ρ\tau=(1-\rho^2)/\rho gives a mathematical foundation to the intuition that higher correlation corresponds to lower temperature, which is a meaningful contribution.

  • The proposed method is evaluated on heterogeneous settings including 3D point-image-text, video-audio-text, mmWave-WiFi-RFID, and multiview, demonstrating generality that does not depend on a specific data type. Although the method is not the best on every individual task, it achieves the best results on the overall metric Δ\Delta across all multimodal settings. This provides important evidence that replacing temperature with the correlation coefficient yields consistent improvements across a wide range of situations, supporting the paper's central claim.

Critical weaknesses
  • The central claim of the paper is that the temperature parameter can be replaced by the correlation coefficient under a multivariate Gaussian assumption. However, it is not clearly demonstrated whether introducing this Gaussian approximation itself contributes to the performance gains. Although a comparison with iSogCLR, an optimization-based temperature tuning method, is provided, the paper lacks an analysis that isolates how much the Gaussian-based formulation benefits performance relative to directly optimizing the temperature.

  • The authors claim that the converged correlation coefficients are robust to the initial values P0P^0 and β\beta. However, in Fig. 3, while the converged value of ρMW\rho_{MW} is indeed stable, the converged values of ρMR\rho_{MR} and ρWR\rho_{WR} show non-negligible differences across initializations, which does not fully support the claim that they are "nearly identical." The paper does not discuss why robustness varies across modality pairs. As a result, it remains unclear whether the estimated correlation coefficients genuinely converge to data-intrinsic quantities, or whether they settle into local optima that depend on the initialization or the learned representations.

Questions for authors

The following questions are all relevant to my assessment of the paper.

  1. What is the essential benefit of replacing the temperature parameter with the correlation coefficient? I understand the gain in interpretability, but it is not clear whether the performance gains stem from the introduction of the correlation concept itself, or from implementation aspects such as the iterative estimation procedure and the adaptive updates. Is it possible to isolate the essential benefit brought by correlation modeling, as compared to approaches that directly optimize the temperature?

  2. Related to the above, would it be possible to run a controlled experiment that keeps the same CorrCLR update procedure but estimates the temperature parameter through the same alternating updates instead of the correlation coefficient? This could empirically verify whether the Gaussian-assumption modeling itself contributes to the improvements.

5 comments
AuthorsOpenReview ↗

Response to Reviewer cSPw

We thank the reviewer for their comments. Please find our responses below:


Q1. What is the essential benefit of replacing the temperature parameter with the correlation coefficient?

We agree that it is important to distinguish the benefits of correlation modeling from those of the iterative estimation procedure. Please note that iterative estimation is also involved when temperature parameters are learned using SGD. Therefore, the iterative estimation procedure should not be considered a feature specific to CorrCLR. We provide further details on the benefits of correlation modeling below:

  • For two modalities, under our Gaussian assumption, the relationship τnm=(1−ρnm2)/ρnm\tau_{nm}=(1-\rho_{nm}^{2})/\rho_{nm} is one-to-one for positive correlation coefficients. Therefore, replacing a freely optimized temperature parameter with a freely optimized correlation coefficient does not, by itself, increase the expressive capacity of the contrastive objective. However, the distinction becomes important when we consider that the correlation coefficients are statistical quantities estimated by solving a maximum likelihood estimation (MLE) problem to ensure consistency with the derived density-ratio formulation of the InfoNCE loss function.

  • The main advantage of correlation modeling becomes more apparent in multi-modal settings. Rather than independently optimizing a temperature for each modality pair, CorrCLR estimates a joint correlation matrix. This introduces consistency constraints among pairwise relationships through the positive definiteness of the matrix. Furthermore, the inverse correlation matrix provides a structured parameterization of the multi-modal contrastive objective, potentially reducing the number of independently tuned parameters.

  • The iterative estimation procedure is a separate component of CorrCLR that has also been used in approaches that learn temperature parameters using SGD. In CorrCLR, this procedure allows the correlation estimates to track changes in the representations during training, while the moving-average update promotes stability (See Lines 287-295).

  • Regarding the reviewer's question about convergence to local optima, which may be affected by the initialization of P0\mathrm{P}^0 and β\beta, we have already discussed this matter in Appendix N (Lines 1446–1455).


Q2. Would it be possible to run a controlled experiment that keeps the same CorrCLR update procedure but estimates the temperature parameter through the same alternating updates instead of the correlation coefficient?

As discussed in our response to Q1, we argue that the controlled experiment requested by the reviewer is comparable to the learnable-temperature approaches already included in our performance evaluations, namely, ULIP and OpenShape in Table 1, as well as iSogCLR in Tables 1, 2, 3, and 7. While the Gaussian-derived relationship provides a mapping between correlation coefficients and temperature parameters, CorrCLR differs from approaches that treat temperature as a learnable parameter in two main ways:

  1. Approaches that treat temperature as a learnable parameter update the temperature parameters by minimizing the contrastive loss function, which is also used to learn the encoder parameters. In contrast, CorrCLR solves a maximum likelihood estimation (MLE) problem to estimate the correlation coefficients of a multivariate Gaussian distribution. Thus, CorrCLR alternately optimizes two different objectives: the contrastive loss for updating the encoder parameters and the MLE objective for estimating the correlation coefficients.

  2. CorrCLR models the joint correlation structure across multiple modalities. In particular, the positive-definiteness constraint on the correlation matrix imposes consistency among pairwise correlations, which is not automatically guaranteed when temperature parameters are optimized independently.


I thank the authors for their detailed responses and appreciate the conceptual clarifications. However, since the responses are largely conceptual and no new empirical evidence has been provided, my main concerns remain.

Q1 and Q2. I understand the authors' view that the existing learnable-temperature baselines are comparable. However, these baselines learn the temperature via SGD, whereas CorrCLR estimates correlations through an alternating MLE procedure. Since these mechanisms differ in exactly the aspect I wished to isolate, the existing comparisons do not fully disentangle whether the improvements stem from the correlation/Gaussian modeling itself or from the alternating estimation procedure. I leave any further evidence to the authors' discretion, but my concern is not fully resolved.

Weakness 2. I acknowledge the discussion in Appendix N on the EM-like nature of CorrCLR. However, my concern regarding Fig. 3 remains: while ρMW\rho_{MW} is stable, ρMR\rho_{MR} and ρWR\rho_{WR} still show non-negligible differences across initializations, which does not fully support the claim that they are "nearly identical." It thus remains unclear whether the estimated correlations converge to data-intrinsic quantities or depend on the initialization.

Overall, I find the clarifications helpful, but in the absence of new evidence my concerns remain.

AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for taking the time to review our responses to their questions. Our replies to the remaining concerns are as follows:


Reviewer's Concern: The existing comparisons do not fully disentangle whether the improvements stem from the correlation/Gaussian modeling itself or from the alternating estimation procedure.

Authors' Response: Following the reviewer's suggestion, we are conducting an experiment in the 3D Point-Image-Text setting, replacing OpenShape's learnable temperature mechanism with an alternating temperature update mechanism. We aim to make the temperature update procedure as close as possible to CorrCLR's correlation coefficient update mechanism.

Specifically, we first fix the temperature and update the encoder parameters by minimizing the contrastive loss. We then fix the encoder parameters and obtain the empirical temperature by minimizing the contrastive loss. The updated temperature is computed as an exponential moving average of the previous temperature and the estimated empirical temperature, with momentum β\beta. Specifically, the update rule is given by τnew=βτprev+(1−β)τ^,\tau^{\text{new}} = \beta \tau^{\text{prev}} + (1-\beta)\hat{\tau},where τprev\tau^{\text{prev}} denotes the previous temperature and τ^\hat{\tau} denotes the estimated empirical temperature.

We will share the experimental results as soon as they become available.


Reviewer's Concern: It remains unclear whether the estimated correlations converge to data-intrinsic quantities or depend on the initialization.

Authors' Response: As pointed out by the reviewer, CorrCLR can be viewed as an EM-like algorithm, similar in spirit to k-means. Therefore, CorrCLR is expected to converge to a local optimum rather than the global optimum. This means that we cannot claim that CorrCLR necessarily converges to exact data-intrinsic quantities. As with other EM-like algorithms, such as k-means, the initialization can affect the local optimum to which the algorithm converges [R1]. Nevertheless, in our paper, we have conducted several experiments to demonstrate both the stability of CorrCLR with respect to initialization and its ability to converge to data-intrinsic relationships.

In addition to Fig. 3, the results presented in Figs. 10 and 11 demonstrate the robustness of the converged correlation coefficients across different initializations. We acknowledge that, in Fig. 3, ρMR\rho_{MR} and ρWR\rho_{WR} converge to non-identical values across different initializations. However, we believe that the consistently larger values of ρMW\rho_{MW}, together with the lower values of ρMR\rho_{MR} and ρWR\rho_{WR} and the close agreement between the latter two coefficients, may reflect underlying data-intrinsic relationships.

Regarding the connection to data-intrinsic quantities, we have also conducted the experiment presented in Appendix H using synthetically generated samples from well-known probability distributions. The results of this experiment demonstrate that, in this setting, CorrCLR is capable of recovering data-intrinsic quantities.

[R1] S. Bubeck, M. Meilă, and U. von Luxburg. "How the initialization affects the stability of the k-means algorithm," ESAIM: Probability and Statistics, Sept. 2012.


Reviewer_RaXtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CorrCLR, a correlation-based approach to adaptive temperature selection in multimodal contrastive learning. The authors establish a relationship between the temperature parameters of InfoNCE and Muscle loss and the correlation coefficients of multimodal representations under a multivariate Gaussian assumption. Specifically, they derive (\tau=(1-\rho^2)/\rho) for the two-modality case and extend the formulation to multiple modalities using the inverse correlation matrix.

Based on this formulation, CorrCLR alternates between estimating the correlation matrix from learned representations and updating modality encoders using the resulting correlation-based contrastive loss. The correlation estimates are stabilized through exponential moving averages.

The authors evaluate CorrCLR across 3D point-image-text, video-audio-text, and mmWave-WiFi-RFID settings, as well as an additional image multiview setting. The reported results show competitive performance on zero-shot classification, retrieval, and human action recognition tasks, with favorable aggregate performance across several benchmarks.

Critical strengths

1. A unified probabilistic perspective on temperature parameterization.

The paper provides a systematic formulation that connects correlation statistics with the coefficients used in contrastive objectives. The extension from the two-modality case to the multimodal setting through the inverse correlation matrix is particularly interesting.

This formulation offers a potentially interpretable alternative to independently tuning multiple temperature parameters. It is also compatible with existing contrastive learning frameworks rather than requiring substantial modifications to their encoder architectures.

2. Broad empirical evaluation and a practical adaptive estimation procedure.

The paper evaluates the proposed approach across several heterogeneous modality combinations and downstream tasks, including zero-shot classification, retrieval, and human action recognition.

The reported results generally demonstrate competitive performance, and CorrCLR achieves favorable aggregate scores in several settings. The experiments also investigate the sensitivity to the initial correlation matrix, momentum parameter, sampling ratio, and update interval.

These results suggest that correlation-based temperature control is a practically viable approach, even though its specific advantages over simpler alternatives remain unclear.

Critical weaknesses

1. The theoretical connection does not sufficiently justify why empirical correlation should determine the contrastive temperature.

The main motivation of the paper is to replace difficult-to-tune temperature parameters with measurable correlation coefficients. However, the theoretical derivation establishes a relationship between temperature and correlation only under a particular Gaussian distributional model with a highly structured covariance matrix.

There are two related concerns.

First, the derivation assumes a Gaussian distribution with covariance (Σ=P⊗Id\mathbf{\Sigma}=\mathbf{P} \otimes \mathbf{I}_d), while the simplification of the density ratio in Appendix A relies on unit-normalized representations. These assumptions are not directly compatible for non-degenerate, high-dimensional representations: a random vector with identity covariance cannot simultaneously have unit norm almost surely. If the Gaussian assumption applies to representations before normalization, the density ratio of the normalized representations generally requires a separate derivation. In particular, the effective similarity scale can depend on the representation dimension and normalization procedure. The Gaussianity diagnostics in Appendix B do not fully resolve this issue.

Second, even if the proposed correlation-temperature relationship is accepted under the assumed model, it does not necessarily follow that the empirical correlation between evolving representations yields an appropriate temperature for contrastive optimization. The temperature controls the concentration of the contrastive similarity distribution and the relative contributions of positive and negative examples to the learning gradients. It is unclear why these optimization properties should be determined solely by the global correlation between modalities.

Moreover, the correlations are themselves functions of the learned encoders, which are optimized using the temperatures derived from those correlations. This creates a feedback mechanism whose benefits and stability are not theoretically characterized.

The multimodal extension raises an additional interpretational question. The effective temperature parameters are defined through negative off-diagonal entries of the inverse correlation matrix, but these entries need not always correspond to positive temperatures for an arbitrary positive-definite correlation matrix. Thus, the proposed multimodal objective may introduce signed interactions that are qualitatively different from simply reparameterizing the original Muscle loss.

These issues are consequential because the claimed principled relationship between correlation and temperature is the central theoretical motivation of the paper. Without a clearer justification of its applicability and optimization implications, the proposed approach remains closer to a plausible adaptive temperature heuristic than a theoretically established alternative to temperature tuning.

2. Insufficient controlled comparisons to establish the necessity and specific benefits of correlation-based temperature adaptation.

The central practical question is whether empirical correlation-based temperature selection is genuinely preferable to simpler strategies for adapting temperature parameters.

Although the paper compares CorrCLR with several contrastive learning methods, many of these baselines employ different loss formulations or optimization mechanisms. Therefore, their performance differences do not isolate the effect of the proposed correlation-based temperature update.

In particular, the paper does not provide a sufficiently controlled comparison between the same underlying contrastive objective using fixed temperatures, directly learnable temperatures, and the proposed correlation-based updates. It is also unclear whether periodically updating the temperature according to empirical correlations is necessary, compared with using fixed correlation estimates or a simpler adaptive temperature schedule.

Such comparisons are especially important because the proposed method introduces additional design choices, including initialization, momentum, sampling ratio, and update frequency. Although the authors provide sensitivity studies, these experiments do not establish that the correlation-based mechanism itself is responsible for the performance improvements.

Furthermore, the empirical advantages are not uniform across individual tasks. For example, in Table 2, CorrCLR achieves the highest aggregate improvement on MSR-VTT, but TRIANGLE substantially outperforms it on text-video retrieval. Similarly, Table 7 shows that CorrCLR underperforms the original Muscle loss across all reported metrics in the three-view ImageNet setting. In Table 3, the large relative improvements in the cross-domain RF setting should also be interpreted carefully, since some reference accuracies are extremely low.

These observations do not invalidate the reported results, but they make it difficult to conclude that CorrCLR consistently provides a superior temperature selection mechanism. The aggregate metric can obscure task-specific trade-offs, and the limited multi-seed evaluation further complicates the assessment of smaller performance differences.

This weakness directly affects the paper's methodological contribution: the experiments demonstrate that CorrCLR can work well, but do not convincingly establish why correlation-based adaptation is necessary or whether it is better than simpler temperature adaptation strategies under controlled conditions.

Questions for authors

1. Applicability of the Gaussian derivation to normalized representations (clarification).

The theoretical derivation assumes Gaussian representations with covariance (Σ=P⊗Id\mathbf{\Sigma}=\mathbf{P} \otimes \mathbf{I}_d), whereas Appendix A uses unit-normalized representations to simplify the density ratio.

Could the authors clarify whether the Gaussian assumption applies before or after normalization? If it applies before normalization, can the authors justify the proposed temperature-correlation relationship for the normalized embeddings actually used in the contrastive loss?

In particular, does the relationship remain unchanged with respect to the embedding dimension and feature normalization procedure?

2. Why should correlation determine the optimal contrastive temperature? (clarification).

The proposed relationship follows from a specific density-ratio model, but temperature also affects contrastive optimization through gradient weighting, negative sample discrimination, and the sharpness of the similarity distribution.

What evidence supports the claim that the empirically estimated correlation provides a suitable temperature for optimizing representation quality, rather than merely being a statistic that can be mapped to a temperature?

Since the estimated correlations depend on the encoders being optimized, how do the authors justify the resulting feedback mechanism and its effects on optimization?

3. Necessity of correlation-based adaptation compared with simpler alternatives (additional evidence).

Could the authors provide a controlled comparison using the same contrastive objective and encoder architecture with:

  • Fixed, carefully tuned temperature parameters.
  • Directly learnable temperature parameters.
  • Fixed correlation estimates without dynamic updates.
  • The proposed dynamically updated correlation estimates.

Such an experiment would help determine whether the improvements arise specifically from correlation-based adaptation rather than from introducing an adaptive temperature schedule or using a different effective temperature range.

This comparison would materially affect my assessment of the methodological contribution.

4. Multimodal interpretation and robustness of the results (clarification and additional evidence).

In the multimodal formulation, the off-diagonal entries of the inverse correlation matrix can have different signs, even when all pairwise correlations are positive. How should negative effective temperature coefficients be interpreted relative to the original Muscle loss, which uses positive temperature parameters?

Additionally, given the variation in performance across individual tasks, could the authors provide further evidence that the improvements are robust across random seeds and are not driven primarily by the aggregate relative-improvement metric?

2 comments
AuthorsOpenReview ↗

Response to Reviewer RaXt - Part 1

We thank the reviewer for their comments. Please find our responses below:


Q1. Applicability of the Gaussian derivation to normalized representations (clarification).

Yi et al. in [R1] have shown that the convergent optimal representations of multimodal contrastive learning follow a uniform distribution on a dd-dimensional unit hypersphere. Furthermore, Betser et al. (ICLR 2026) have shown that alignment leads to a constrained uniformity problem on the hypersphere, which, combined with norm concentration, yields Gaussian structure for both normalized (unit-norm) and unnormalized representations. Following these theoretical works, we assume in our work that the Gaussian assumption applies after normalization.

As discussed in Appendix M (Lines 1396–1398), there is a close connection between the uniform distribution on the unit hypersphere and the Gaussian distribution as the representation dimension dd increases. Additionally, Nicollier et al. in [R2] have shown that, for sufficiently large dd, the characteristic functions of the uniform distribution on the unit hypersphere and the Gaussian distribution are very similar, making the Gaussian approximation applicable when d>256d>256.

[R1] L. Yi, R. Douady, and C. Chen, "The convergent representation of contrastive vision-language models: Geometry, modality gap and shared space slignment," ICML, 2026.

[R2] L. Nicollier, M. Dunitz, M. Pic, P. Musé, E. Meinhardt-Llopis, and G. Facciolo, "SPHERE-JEPA: Spherical prediction with homogeneous embeddings", NeurIPS, 2026.


Q2. Why should correlation determine the optimal contrastive temperature? (clarification).

As discussed in Lines 171–172 and 230–232, our motivation is to provide a formulation consistent with the density ratios derived by Oord et al. (2019) and Setayesh et al. (ICLR 2026). Under the assumed joint Gaussian model, the correlation coefficients characterize the statistical dependence between modality representations and determine the coefficients of the corresponding density ratio. This provides a principled alternative to manually selecting or independently learning pairwise temperatures, which do not account for such dependence between modality representations.

As discussed in Appendix N (Lines 1446–1451), CorrCLR can be viewed as an EM-like algorithm, similar in spirit to k-means. Specifically, CorrCLR first learns multimodal representations while keeping the correlation coefficients fixed. Then, given the learned representations, a maximum likelihood estimation (MLE) problem is solved to estimate the parameters (i.e., the correlation coefficients) of the multivariate Gaussian distribution. Thus, CorrCLR involves two optimization problems.

The first optimization problem minimizes the contrastive loss. Similar to temperature parameters, the fixed correlation coefficients affect gradient weighting, negative-sample discrimination, and the sharpness of the similarity distribution. The second optimization problem uses the learned representations to solve an MLE problem, thereby estimating the statistical quantities required by the derived density-ratio formulation. Consequently, the estimated correlations adapt to the evolving representation space.

Our experiments provide empirical evidence that the correlation estimates converge under the tested settings and that the method is reasonably robust to different initial correlation matrices, sampling ratios, and update intervals.


Q3. Necessity of correlation-based adaptation compared with simpler alternatives (additional evidence).

We argue that all the controlled comparisons requested by the reviewer have already been included in our performance evaluations. Fixed, carefully tuned temperature parameters and fixed correlation coefficients without dynamic updates can be considered comparable approaches, given the relationship between temperature parameters and correlation coefficients discussed in Lines 190 and 249–252. Pairwise Alignment in Table 2, Muscle in Table 3, and InfoNCE and Muscle in Table 7 are relevant to these baselines. Directly learnable temperature parameters have been employed in ULIP and OpenShape (Table 1), as well as in iSogCLR (Tables 1, 2, 3, and 7).


AuthorsOpenReview ↗

Response to Reviewer RaXt - Part 2

Q4. Multimodal interpretation and robustness of the results (clarification and additional evidence).

Regarding the signs of the inverse correlation matrix, we clarify that the coefficients appearing in the Gaussian density-ratio formulation are not ordinary positive temperature parameters as used in the original Muscle loss. The inverse correlation matrix represents the precision matrix of the joint Gaussian distribution, whose off-diagonal entries characterize conditional dependencies among modalities. Consequently, these entries can be negative even when all pairwise marginal correlations are positive. Such negative coefficients reflect the conditional dependence structure among modalities rather than negative marginal correlations. Thus, they should be understood as signed coefficients in the joint density-ratio formulation, rather than as conventional positive temperature parameters. We believe that allowing these coefficients to take negative values provides greater flexibility in optimizing representations and capturing their underlying dependencies.

As noted by the reviewer, CorrCLR may not achieve the best performance on a particular downstream task compared with some baselines. However, when considering overall performance across diverse tasks, CorrCLR achieves the best overall performance in terms of Δ\Delta. We use this well-established metric because Δ\Delta captures the relative improvement over a base method and provides a single measure for comparing performance across heterogeneous and diverse tasks.

For example, although TRIANGLE achieves better performance on text–video retrieval in Table 2, its performance is lower on other retrieval tasks. In contrast, CorrCLR achieves reasonably high performance across all downstream tasks. We also refer the reviewer to our discussion in Lines 502–505 regarding the robustness of the reported results.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.