- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies sample-wise and feature-wise contrastive learning from a unified mutual information perspective. The authors extend the InfoNCE formulation to the feature dimension and derive a mutual information bound that includes an additional dependence penalty Δ, accounting for correlations and non-exchangeability among feature channels. They further show that the mutual information along either axis is bounded by the mutual information between the full embedding matrices, which motivates combining sample-wise and feature-wise contrastive objectives. The paper also discusses the connection to redundancy-reduction methods such as Barlow Twins. Experiments on ImageNet-1K and Flickr30K evaluate the combined objective for visual representation learning and cross-modal retrieval, with additional ablations on batch size, feature dimension, loss weighting, and temperature.
Critical strengths
-
I think the mutual information analysis of feature-wise contrastive learning is the most interesting part of this work. The authors identify the dependence and non-exchangeability of feature channels as factors that are absent from the standard sample-wise InfoNCE formulation. The introduction of Δ provides a useful theoretical perspective, and the synthetic experiments help illustrate why this additional term matters.
-
The paper offers a clear and coherent view of sample-wise and feature-wise contrastive learning within the same framework. I also find the discussion of the connection to Barlow Twins useful, as it helps explain the difference between relative feature matching and explicit redundancy reduction. The overall formulation is straightforward and does not require major architectural modifications.
-
The empirical evaluation is reasonably comprehensive. The authors consider k-NN, linear evaluation, and full fine-tuning on ImageNet, along with cross-modal retrieval on Flickr30K. The ablations on batch size, feature dimension, loss weighting, and temperature are also helpful.
Critical weaknesses
-
While I find the mutual information perspective interesting, the methodological contribution seems rather limited. The feature-wise contrastive objective is closely related to those in SimCSE++ [1] and DimCL [2], and the paper acknowledges this connection. The main novelty lies in the analysis of the dependence penalty Δ. Lemma 1 and the convex combination provide a valid theoretical interpretation, but they do not by themselves establish that the two objectives are complementary in practice. I would have liked to see a clearer demonstration of what the new analysis tells us beyond what is already known from existing dimensional contrastive methods.
-
My main concern is that the theoretical analysis is not fully connected to the proposed optimization objective. The mutual information bound in Eq. (13) contains the representation-dependent penalty Δ, while the actual training objective only minimizes the combined contrastive loss. The authors also show in Appendix D that feature standardization does not guarantee a small Δ. As a result, a lower training loss does not necessarily correspond to a tighter mutual information bound. Since the dependence penalty is central to the paper's theoretical contribution, the lack of evidence about its behavior in learned representations makes the practical significance of the analysis less clear.
-
The experimental results are promising for frozen representations, but they support a narrower conclusion than the paper's overall claims suggest. In Table 1, the combined objective performs well on k-NN and linear evaluation, while its advantage largely disappears under full fine-tuning. The cross-modal experiments are also limited to training linear projection heads on frozen encoders, and the improvement over VICReg on Flickr30K image-to-text R@1 is only 0.12 percentage points. The comparison with SimCLR+DimCL is particularly important given the similarity of the objectives, yet the current presentation does not make it easy to determine where the substantial performance gap comes from. I think the empirical evidence is strongest for improving frozen representation quality under the specific pretraining setup, rather than establishing a general advantage over existing SSL methods.
[1] SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives. EMNLP 2023.
[2] DimCL: Dimensional Contrastive Learning for Improving Self-Supervised Learning. IEEE Access, 11, 21534–21545.
Questions for authors
-
I would appreciate more details on the implementations used for the closely related baselines in Table 1, especially SimCLR+DimCL. Are the projection heads, feature normalization, negative candidate sets, temperatures, and other training settings matched to the proposed method? In particular, what explains the difference between SimCLR and the proposed sample-wise objective, given that both are based on standard InfoNCE?
-
Is there a way to examine the dependence penalty Δ on representations learned by the actual model? I understand that estimating it directly may be difficult, but some analysis of feature correlations, covariance structure, or other tractable proxies would be helpful. I am particularly interested in whether the feature-wise loss actually reduces the dependencies described by the theory.
-
I noticed a potential issue in Appendix C.4. Under a uniform random permutation of all feature channels, conditioning on the positive position V does not seem to determine the ordering of the remaining negatives. Would Eq. (46) and Eq. (52) still hold as equalities, or should they be inequalities in the general case? It would be helpful to clarify the permutation construction and any additional assumptions needed.
-
Could the authors clarify how the Gaussian assumptions relate to the actual implementation? The analysis assumes i.i.d. rows and a non-degenerate covariance matrix, while training-time BatchNorm introduces dependencies across samples. The final 2048-to-8192 linear projection also raises questions about covariance rank under fixed affine normalization. I would like to understand which theoretical conclusions are intended to apply to the practical model. It would also help to specify the exact encoder checkpoints, optimizer, and learning rate used in the Flickr30K experiments.