acceptodds
Withdrawn from ICLR 2027

Contrastive Learning Across Samples and Features Through the Lens of Mutual Information

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Contrastive representation learning typically contrasts corresponding samples across two views, as in InfoNCE. However, the mutual information bound of sample-wise InfoNCE is limited by the number of negative samples and does not explicitly discourage redundancy across features. To address these issues, we study sample-wise and feature-wise contrastive learning through a common mutual information formulation. Our analysis of the feature axis reveals an additional penalty arising from the dependence and non-exchangeability of feature channels and motivates feature standardization before contrastive matching. The two objectives are complementary: the sample-wise objective distinguishes inputs, while the feature-wise objective promotes informative, non-redundant features. We prove that the mutual information captured along either axis lower-bounds the mutual information between the full embedding matrices, providing a principled basis for combining the two objectives. Our extensive experiments across image representation learning and cross-modal retrieval demonstrate that the combined objective improves over either objective alone and achieves superior performance compared with existing self-supervised learning methods.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Contrastive Learning, Self-Supervised Learning, Representation Learning, InfoNCE

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_AQghOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies sample-wise and feature-wise contrastive learning from a unified mutual information perspective. The authors extend the InfoNCE formulation to the feature dimension and derive a mutual information bound that includes an additional dependence penalty Δ, accounting for correlations and non-exchangeability among feature channels. They further show that the mutual information along either axis is bounded by the mutual information between the full embedding matrices, which motivates combining sample-wise and feature-wise contrastive objectives. The paper also discusses the connection to redundancy-reduction methods such as Barlow Twins. Experiments on ImageNet-1K and Flickr30K evaluate the combined objective for visual representation learning and cross-modal retrieval, with additional ablations on batch size, feature dimension, loss weighting, and temperature.

Critical strengths
  1. I think the mutual information analysis of feature-wise contrastive learning is the most interesting part of this work. The authors identify the dependence and non-exchangeability of feature channels as factors that are absent from the standard sample-wise InfoNCE formulation. The introduction of Δ provides a useful theoretical perspective, and the synthetic experiments help illustrate why this additional term matters.

  2. The paper offers a clear and coherent view of sample-wise and feature-wise contrastive learning within the same framework. I also find the discussion of the connection to Barlow Twins useful, as it helps explain the difference between relative feature matching and explicit redundancy reduction. The overall formulation is straightforward and does not require major architectural modifications.

  3. The empirical evaluation is reasonably comprehensive. The authors consider k-NN, linear evaluation, and full fine-tuning on ImageNet, along with cross-modal retrieval on Flickr30K. The ablations on batch size, feature dimension, loss weighting, and temperature are also helpful.

Critical weaknesses
  1. While I find the mutual information perspective interesting, the methodological contribution seems rather limited. The feature-wise contrastive objective is closely related to those in SimCSE++ [1] and DimCL [2], and the paper acknowledges this connection. The main novelty lies in the analysis of the dependence penalty Δ. Lemma 1 and the convex combination provide a valid theoretical interpretation, but they do not by themselves establish that the two objectives are complementary in practice. I would have liked to see a clearer demonstration of what the new analysis tells us beyond what is already known from existing dimensional contrastive methods.

  2. My main concern is that the theoretical analysis is not fully connected to the proposed optimization objective. The mutual information bound in Eq. (13) contains the representation-dependent penalty Δ, while the actual training objective only minimizes the combined contrastive loss. The authors also show in Appendix D that feature standardization does not guarantee a small Δ. As a result, a lower training loss does not necessarily correspond to a tighter mutual information bound. Since the dependence penalty is central to the paper's theoretical contribution, the lack of evidence about its behavior in learned representations makes the practical significance of the analysis less clear.

  3. The experimental results are promising for frozen representations, but they support a narrower conclusion than the paper's overall claims suggest. In Table 1, the combined objective performs well on k-NN and linear evaluation, while its advantage largely disappears under full fine-tuning. The cross-modal experiments are also limited to training linear projection heads on frozen encoders, and the improvement over VICReg on Flickr30K image-to-text R@1 is only 0.12 percentage points. The comparison with SimCLR+DimCL is particularly important given the similarity of the objectives, yet the current presentation does not make it easy to determine where the substantial performance gap comes from. I think the empirical evidence is strongest for improving frozen representation quality under the specific pretraining setup, rather than establishing a general advantage over existing SSL methods.

[1] SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives. EMNLP 2023.

[2] DimCL: Dimensional Contrastive Learning for Improving Self-Supervised Learning. IEEE Access, 11, 21534–21545.

Questions for authors
  1. I would appreciate more details on the implementations used for the closely related baselines in Table 1, especially SimCLR+DimCL. Are the projection heads, feature normalization, negative candidate sets, temperatures, and other training settings matched to the proposed method? In particular, what explains the difference between SimCLR and the proposed sample-wise objective, given that both are based on standard InfoNCE?

  2. Is there a way to examine the dependence penalty Δ on representations learned by the actual model? I understand that estimating it directly may be difficult, but some analysis of feature correlations, covariance structure, or other tractable proxies would be helpful. I am particularly interested in whether the feature-wise loss actually reduces the dependencies described by the theory.

  3. I noticed a potential issue in Appendix C.4. Under a uniform random permutation of all feature channels, conditioning on the positive position V does not seem to determine the ordering of the remaining negatives. Would Eq. (46) and Eq. (52) still hold as equalities, or should they be inequalities in the general case? It would be helpful to clarify the permutation construction and any additional assumptions needed.

  4. Could the authors clarify how the Gaussian assumptions relate to the actual implementation? The analysis assumes i.i.d. rows and a non-degenerate covariance matrix, while training-time BatchNorm introduces dependencies across samples. The final 2048-to-8192 linear projection also raises questions about covariance rank under fixed affine normalization. I would like to understand which theoretical conclusions are intended to apply to the practical model. It would also help to specify the exact encoder checkpoints, optimizer, and learning rate used in the Flickr30K experiments.

Reviewer_xJEmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies contrastive learning along both axes of the embedding matrices Z,Z^∈RB×DZ, \hat Z \in \mathbb{R}^{B \times D}. Besides standard sample-wise InfoNCE (whose MI bound is capped at log⁡B\log B), the authors apply InfoNCE to the DD standardized feature columns. They show that the resulting bound, Ifeat≥log⁡D−Δ−E[Lfeat]I_{\text{feat}} \ge \log D - \Delta - \mathbb{E}[\mathcal{L}_{\text{feat}}], includes a penalty Δ\Delta arising from the dependence and non-exchangeability of feature channels, which they analyze under a Gaussian model. Since both axis-wise MI quantities lower-bound the joint MI I(Z;Z^)I(Z;\hat Z), they propose a convex combination of the two losses and relate the feature-wise loss to Barlow Twins. Experiments on ImageNet (ResNet-50, 100 epochs, B=256B = 256) and Flickr30K retrieval show that the combined objective improves k-NN and linear-probe accuracy over either loss alone, with ablations over BB, DD, π\pi and τ\tau.

Critical strengths
  1. The paper sets up a symmetric row/column view of InfoNCE, and the theoretical study of the column (feature) view is genuinely interesting. It shows that, along the feature axis, the bound has an extra term Δ\Delta due to the dependence and non-exchangeability of channels. Although Δ\Delta is hard to estimate, it remains a quantity of interest: whenever it is kept small, Lfeat\mathcal{L}_{\text{feat}} yields a meaningful lower bound on the column-wise MI.
  2. The method is simple and cheap. The combined objective improves k-NN and linear-probe accuracy over either loss alone, with only roughly 7% extra training time.
  3. The ablations over BB, DD and π\pi support the complementarity of the two axes and suggest good small-batch behavior.
Critical weaknesses
  • The feature-wise loss is essentially the dimension-wise InfoNCE of DimCL and SimCSE++, as the paper acknowledges. Combining sample-wise and feature-wise terms was previously done in SimCLR+DimCL (which is in Table 1) and in Zero-CL (Zhang et al., 2022b; cited but not compared). The contribution therefore rests mainly on the MI analysis.
  • The only quantitative control on Δ\Delta is the Gaussian upper bound in Eq. 8, which contains the term −B2log⁡det⁡Σz^-\frac{B}{2}\log\det\Sigma_{\hat z}, where Σz^\Sigma_{\hat z} is the covariance of the DD-dimensional output embedding. Per Appendix B, this embedding is produced by a linear layer from 20482048 to D=8192D = 8192 dimensions followed by BatchNorm. The output therefore lies in a subspace of dimension at most 20482048: at least D−2048=6144D - 2048 = 6144 directions have zero variance, so Σz^\Sigma_{\hat z} has zero eigenvalues and det⁡Σz^=0\det\Sigma_{\hat z} = 0. Consequently −B2log⁡det⁡Σz^=+∞-\frac{B}{2}\log\det\Sigma_{\hat z} = +\infty, and Eq. 8 only states Δ≤+∞\Delta \le +\infty, which places no constraint on Δ\Delta. Since Δ\Delta is unconstrained, Eq. 10 no longer guarantees that minimizing L_feat increases I_feat.
  • To my understanding, Eq. 8 requires an isotropic covariance (whitening), not just per-channel standardization. Appendix D itself shows standardization does not guarantee a small Δ\Delta. The experiments lacks ablation with versus without standardization (or with full whitening).
  • Only one setting is tested (ResNet-50, 100 epochs, B=256, one seed, no error bars), while the gaps between methods are 1–3 points. A ViT backbone, a longer schedule, or larger batches would show whether the conclusions generalize.
  • There is no transfer evaluation (detection, segmentation, other datasets).
  • The batch-size study stops at B=256, so it remains open whether L_feat helps once L_sam has large batches.
  • The abstract claims superior performance over existing SSL methods, but under fine-tuning L_comb (73.2) is below BYOL (74.5), MoCo (74.3) and SimCLR (73.9).
  • The setup trains linear heads on frozen ViT-L/14 and BERT-base features using Flickr30K pairs only. Which ViT-L/14 checkpoint is used (a CLIP image tower would already be language-aligned) ? The only external baseline is VICReg, other baselines would strengthen this result.
Questions for authors
  1. Beyond the MI interpretation, what distinguishes Lcomb\mathcal{L}_{\text{comb}} from SimCLR+DimCL and Zero-CL in practice? Could you compare against Zero-CL under the same setting?
  2. Given that rank⁡(Σz^)≤2048<D\operatorname{rank}(\Sigma_{\hat z}) \le 2048 < D, is the bound in Eq. 8 finite for the trained models? Could you report an estimate of Δ\Delta (or of its Gaussian bound) for the learned representations, e.g., with D≤2048D \le 2048?
  3. Could you ablate the feature standardization step (none vs. per-channel standardization vs. full whitening)?
  4. Do your results hold at larger batch sizes (e.g., B≥1024B \ge 1024), and could you report mean and standard deviation over several seeds for the main ImageNet results?
  5. For Flickr30K, which ViT-L/14 checkpoint is used? Could you add a standard InfoNCE/CLIP-style baseline?
Reviewer_XrPvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies contrastive learning with a new approach that contrasts features lather than only contrasting samples. They categorizes two contrasting approaches: 1) sample-wise; and 2) feature-wise contrastive learning. The former is the conventional contrastive learning method, whereas the latter one tries to distinguish features by applying InfoNCE like objective over features. With those two approaches, some analysis of the proposed method is also provided, especially in terms of mutual information.

Critical strengths

The proposed features-wise contrastive learning looks somewhat interesting as I have not seen such method. Also, experimental results show their strong performance over baselines.

Critical weaknesses

Although the concept of feature-wise contrastive learning is interesting, I have several concern about the paper:

  1. I could not understand a clear motivation or rationale to apply feature-wise contrasting. Sample-wise contrastive learning has been well understood through several theories and insight. However, I cannot find why we need feature-wise contrastive learning. Do we really need it? If so, for what reasons and what applications?
  2. Several analysis are very difficult to follow. There are undefined notation too. For example, what is I(xi;x^i)I(x_i; \hat{x}_i) (in Eq. (3)) with a fixed anchor sample ziz_i? I thought it is Shannon mutual information but mutual information requires distributions, not sample. The whole analysis in the paper should be checked and polished.
  3. Most of the analysis for InfoNCE with mutual information is already well known in the literature. The new things are only for the feature-wise InfoNCE, but it provides less information. For example, Δ\Delta just expresses left terms after making the MI bound as in Eq (10).
  4. The DeltaDelta is also problematic. As discussed after Eq (8), making Δ\Delta small is difficult in general, meaning that the bound would be loose, which implies that optimizing feature-wise infoNCE has less meaning. Moreover, as I noted in the first point, I could not find any reason for having orthogonal features.
  5. As written in line 257, unavailbility of a tractable quantity of Δ\Delta is problematic.
Questions for authors

I have some questions as follows:

  1. Why does the standardization help the model reduce Δ\Delta?
  2. which quantity in Eq (11) gives max in general? Can the second term exceed the first term?
  3. what benefits can we obtain by distingushing features?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.