- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates how different training objectives affect the predictive information retained in representations under limited capacity, providing theoretical analyses using binary and Gaussian models to show that contrastive learning can sometimes discard useful predictive information. The authors further validate their findings through controlled synthetic experiments and experiments on real-world datasets, including DeepFashion, CUB, and OCR-VQA.
Critical strengths
-
The paper provides theoretical analyses showing that improving contrastive matching performance does not necessarily improve the predictive information retained in learned representations. I think this is an interesting finding that could help us better understand the limitations of contrastive learning.
-
The authors investigate the problem from several perspectives, including finite-bit representations, Gaussian models, and different divergence-based objectives. I think the theoretical analysis looks comprehensive and provides some insights into how training objectives affect representation learning.
Critical weaknesses
-
Although the paper provides extensive theoretical analysis and controlled experiments, the results on real-world datasets are not very convincing. I think stronger experiments on modern contrastive learning frameworks would make the paper much more convincing.
-
The paper covers many different theoretical settings, including binary codes, Gaussian models, and power divergences. While each analysis might be interesting, I find it difficult to identify the main contribution and understand how these different results connect to each other. I think the authors should better organize the paper and more clearly highlight the key findings.
-
Although the theoretical results show that contrastive learning can lose useful predictive information, it is not clear how often this problem occurs in practical applications or how these findings could help improve existing contrastive learning methods. I think the authors should provide a clearer discussion of the practical implications.
Questions for authors
-
Could you provide stronger experiments on modern contrastive learning frameworks, such as SimCLR or CLIP, with end-to-end representation learning? I think this would help demonstrate whether the theoretical findings are relevant to practical applications.
-
Could you make the paper easier to follow and better highlight the main contributions? I find it difficult to understand how the different theoretical analyses connect to the central research question.
-
Could you clarify the practical value of this work? In particular, how could these theoretical findings help us design better training objectives or avoid losing useful predictive information?
Thank you for asking for a test on real images and for a concrete example of how the theory could guide training.
COCO. During discussion we trained encoders that compress frozen CLIP features into eight bytes per image. In each run, we trained two copies of the same starting encoder, one to match captions to images and one to predict captions from images. We evaluated both tasks from each encoder's saved bytes on 11,578 images. The table compares the matching-trained code with the prediction-trained code at the original fitting budget.
The differences were small, and each paired 95% interval excluded zero for these fixed fits. On the same saved codes, stronger caption predictors reduced the caption gaps. The matching advantage disappeared when we refitted the matching scorers and selected them on calibration data. This is why we would refit both the predictor and the matching scorer before attributing a gap to lost information.
Other image experiments. On CUB we fine-tuned the full SigLIP2 image encoder for bird-attribute prediction. Direct training did better on both measures in one selected pair. Both objectives selected the starting code in the other. This was supervised fine-tuning. End-to-end SimCLR or CLIP pretraining remains untested.
In a follow-up to the submitted one-bit DeepFashion study, we could compute the lowest prediction loss possible from the matching code and the requested attribute. On 47,389 images from held-out filename groups, a fitted predictor using the direct code beat that minimum by 0.0136 nats per request. The matching-score difference there was inconclusive.
What the theory adds. The binary model shows when a high matching score forces extra prediction loss even with the best decoder. The Gaussian model identifies which directions an optimal linear encoder keeps. The divergence examples show that objectives without a finite candidate list can also disagree with prediction.
Section 6 gives one way to align the objectives in the binary model. Match each requested bit against an independent fair bit, given the preceding requested bits and the same later information. The bits agree half the time. When they differ, choosing the correct candidate is exactly predicting the bit. With a shared scoring function, expected matching loss equals a constant plus half the prediction loss, so both objectives have the same minimizers.
We would be glad to expand on the experiments or the objective design if helpful.