acceptodds
Under review as a conference paper at ICLR 2027

How Training Objectives Allocate Predictive Information

Abstract

A representation with a fixed number of bits or a fixed linear rank must allocate information among the predictions it will support. We study how contrastive matching makes this choice when the prediction context arrives after encoding. A finite candidate list caps the gain from any one request. Matching can therefore favor information for harder requests even when prediction values it less. For two independent uniform bit strings with later complementary information, we solve prediction over all randomized encoders within one string's bit length and bound unavoidable predictive regret from attained matching scores. In two of six controlled neural settings, matching training loses information from a supplied prediction-optimal code. Simultaneous risk bounds at confidence for the fitted models show that a smoothed direct predictor outperforms every predictor of the paired matching code. Withholding complementary information reverses the minimum-risk ordering for the same parity-initialized pairs. We then characterize every optimal linear matching subspace in a Gaussian block model and quantify how predictive cost depends on optimization accuracy. Uncapped power divergences can favor a less frequent, more reliably observed string. Bounds over all randomized fixed-bit encoders distinguish bounded from growing regret as divergence order approaches one and string length grows. These results separate objective-induced information loss from errors in fitting a predictor.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, contrastive learning, information theory, predictive information, capacity constraints, variational objectives

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KTawOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how different training objectives affect the predictive information retained in representations under limited capacity, providing theoretical analyses using binary and Gaussian models to show that contrastive learning can sometimes discard useful predictive information. The authors further validate their findings through controlled synthetic experiments and experiments on real-world datasets, including DeepFashion, CUB, and OCR-VQA.

Critical strengths
  1. The paper provides theoretical analyses showing that improving contrastive matching performance does not necessarily improve the predictive information retained in learned representations. I think this is an interesting finding that could help us better understand the limitations of contrastive learning.

  2. The authors investigate the problem from several perspectives, including finite-bit representations, Gaussian models, and different divergence-based objectives. I think the theoretical analysis looks comprehensive and provides some insights into how training objectives affect representation learning.

Critical weaknesses
  1. Although the paper provides extensive theoretical analysis and controlled experiments, the results on real-world datasets are not very convincing. I think stronger experiments on modern contrastive learning frameworks would make the paper much more convincing.

  2. The paper covers many different theoretical settings, including binary codes, Gaussian models, and power divergences. While each analysis might be interesting, I find it difficult to identify the main contribution and understand how these different results connect to each other. I think the authors should better organize the paper and more clearly highlight the key findings.

  3. Although the theoretical results show that contrastive learning can lose useful predictive information, it is not clear how often this problem occurs in practical applications or how these findings could help improve existing contrastive learning methods. I think the authors should provide a clearer discussion of the practical implications.

Questions for authors
  1. Could you provide stronger experiments on modern contrastive learning frameworks, such as SimCLR or CLIP, with end-to-end representation learning? I think this would help demonstrate whether the theoretical findings are relevant to practical applications.

  2. Could you make the paper easier to follow and better highlight the main contributions? I find it difficult to understand how the different theoretical analyses connect to the central research question.

  3. Could you clarify the practical value of this work? In particular, how could these theoretical findings help us design better training objectives or avoid losing useful predictive information?

1 comment

Thank you for asking for a test on real images and for a concrete example of how the theory could guide training.

COCO. During discussion we trained encoders that compress frozen CLIP features into eight bytes per image. In each run, we trained two copies of the same starting encoder, one to match captions to images and one to predict captions from images. We evaluated both tasks from each encoder's saved bytes on 11,578 images. The table compares the matching-trained code with the prediction-trained code at the original fitting budget.

RunTwo-candidate matching loss (nats/query)Caption prediction loss (nats/caption)
1Lower by 0.00066Higher by 0.068
2Lower by 0.00051Higher by 0.083

The differences were small, and each paired 95% interval excluded zero for these fixed fits. On the same saved codes, stronger caption predictors reduced the caption gaps. The matching advantage disappeared when we refitted the matching scorers and selected them on calibration data. This is why we would refit both the predictor and the matching scorer before attributing a gap to lost information.

Other image experiments. On CUB we fine-tuned the full SigLIP2 image encoder for bird-attribute prediction. Direct training did better on both measures in one selected pair. Both objectives selected the starting code in the other. This was supervised fine-tuning. End-to-end SimCLR or CLIP pretraining remains untested.

In a follow-up to the submitted one-bit DeepFashion study, we could compute the lowest prediction loss possible from the matching code and the requested attribute. On 47,389 images from held-out filename groups, a fitted predictor using the direct code beat that minimum by 0.0136 nats per request. The matching-score difference there was inconclusive.

What the theory adds. The binary model shows when a high matching score forces extra prediction loss even with the best decoder. The Gaussian model identifies which directions an optimal linear encoder keeps. The divergence examples show that objectives without a finite candidate list can also disagree with prediction.

Section 6 gives one way to align the objectives in the binary model. Match each requested bit against an independent fair bit, given the preceding requested bits and the same later information. The bits agree half the time. When they differ, choosing the correct candidate is exactly predicting the bit. With a shared scoring function, expected matching loss equals a constant plus half the prediction loss, so both objectives have the same minimizers.

We would be glad to expand on the experiments or the objective design if helpful.

Reviewer_iNozOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how contrastive training objectives allocate information in a rate-limited representation, and the implications for downstream prediction. In the settings considered, training on a contrastive matching objective may lose information useful for the predictor, and direct prediction may perform better. The main contributions include lower bounds on the regret for prediction implied by attaining a certain matching score in the given settings, and an analysis of the mechanism causing these losses including special cases where the two objectives (matching and prediction) agree. Validation is mostly through a pair of studies on synthetic binary and Gaussian sources, which are analyzed in detail. A smaller set of experiments tests the hypothesis on natural-data features from DeepFashion, OCR-VQA and CUB, with the conclusions being less clear in this setting.

Critical strengths
  1. The paper attempts to address an interesting question, namely how contrastive matching objectives interact with a downstream prediction objective. Given the popularity of contrastive objectives for learning representations, e.g., though CLIP, which often need to be transferred later to other downstream tasks, a theoretical characterization of these interactions and potential information losses is interesting and relevant. However, prior work has already studied such questions under various lenses, such as Chen et al., (2021), Robinson et al., (2021), Xue et al., (2023). While some of these works were more empirical in flavor, I am still of the opinion that the current paper could better differentiate the question it is asking and its conclusions and insights from this existing literature.
  2. For the binary and Gaussian models, the analysis and results are comprehensive and convincing.
Critical weaknesses
  1. While much interesting mathematics has been done, the settings and consequently the validation of the claims seem limited. In particular, the binary fixed-bit matching experiments concern a very restrictive setting where there are only two possible bit-strings with fixed request probabilities. Proposition 2, establishing the lower bound on the predictive regret, further requires request and retention probabilities exactly p=3/4p=3/4, r∈{1/2,9/10}r \in \{ 1/2, 9/10 \}. The Gaussian source is also limited to independent, isotropic Gaussians. It is unclear how these settings are expected to provide insights which generalize to practical contrastive learning workflows, and the paper leaves this question open in the conclusion.
  2. In a similar vein, the results of the natural-data experiments in Section 6 do not appear to strongly support the idea that the observations from the synthetic setting carry over to the natural data case. While the paragraph from line 470 onwards describes the experiment's setting in detail, it does not establish the same conclusions concerning the predictive regret, indeed, the paper states that none of the 42 comparisons shows the direct predictor universally beating the matching code. This raises additional concerns, from my point of view, about the importance of the results in the preceding sections.
Questions for authors
  1. With reference to point (1) in the Strengths section, could the authors clarify how their contribution fits into the existing literature on the quality of representations learned via contrastive learning, and their suitability for downstream prediction?
  2. The conclusion states that "how often these losses occur in natural representations remains open". In view of the mixed results on natural data in Section 6, this seems a substantial limitation of the work. Do the authors have any ideas on how this broader question could be resolved either empirically or via a more general theory?
  3. Why does Proposition 2 require p=3/4p=3/4 and r∈{1/2,9/10}r \in \{ 1/2, 9/10 \} specifically?
  4. Some terms, such as "parity-initialized pairs", "candidate codes", "controlled neural fits", read somewhat oddly to me and could do with more explanation when they are first used. In particular, the Abstract introduces many of these terms and concepts without first explaining their meaning or significance adequately, which makes the beginning of the paper quite difficult to read.
2 comments

Thank you for the careful reading. We should have explained why Proposition 2 uses those probabilities.

Why those values? pp is the chance of requesting the first string, and rr is the chance of also receiving the other string. We chose r=0.5r=0.5 and r=0.9r=0.9 on opposite sides of p=0.75p=0.75. Storing the first string is optimal for prediction at r=0.5r=0.5. At r=0.9r=0.9, storing their bitwise XOR is optimal because the XOR and either string together recover the other.

For example, take p=2/3p=2/3 and r=4/5r=4/5, and suppose the full 8-bit string is requested (a=1a=1 in the paper). Saving four bits from each of two independent uniform 8-bit strings rules out a random alternative 15 times out of 16. With two candidates, this gives 15/1615/16 bits of matching gain over guessing. Any 8-bit code reaching that gain loses at least 11/8011/80 bits per request relative to the best 8-bit prediction code. At that score, the penalty stays positive throughout 1/2<p<r<15/161/2<p<r<15/16. This example follows from Appendix A.1 and Eq. (4) in A.2 of the submission.

Prior work. Chen et al. link finite-batch saturation to competition between features. Robinson et al. show that contrastive optima can suppress features. Xue et al. study which features a minimum-norm solution keeps under a dimension limit. Our result asks what we can conclude from a matching score that a code actually reaches. In the binary model, a sufficiently high score gives a lower bound on the extra prediction loss even its best decoder must incur.

In Section 3, the “controlled neural fits” are encoders trained on strings sampled from the model. Both encoders in a “parity-initialized pair” start from the same XOR code. In the Gaussian experiment, the scorer chooses which of several stored codes goes with one target. These are the “candidate codes.” The theorem allows signal strength to vary within a block, though it assumes independent Gaussian blocks and a linear encoder.

Natural data. In the 42 comparisons you mention, no fitted direct predictor beat the best possible predictor of its paired matching code. In eight pairs, however, the matching code matched better but carried less information for prediction on the evaluated images (Table 9, p. 48). The direct code could support better predictions in those eight pairs than its trained predictor delivered. The 42 comparisons reuse fits and images.

A separate one-bit DeepFashion study did pass that stronger test (Section 6, p. 9). Each predictor receives its own one-bit image code and the same request to predict the denim or floral label. The table gives prediction log loss on the evaluated images, in nats per request. Lower is better.

DeepFashion evaluationDirect code (fitted predictor)Matching code (best possible predictor)
Submitted split (27,221 images)0.17190.1786
Follow-up with held-out filename groups (47,389 images)0.18110.1947

Every evaluation filename group in the submitted split also appeared in training. The direct predictor’s advantage persisted when we held out entire groups. The small matching advantage changed sign, and its 95% interval on the new split included zero. To estimate how often both effects occur together, we would choose tasks and storage budgets in advance and use fresh data to bound the best prediction loss each code can achieve.

We would be glad to expand on the calculation if helpful.

Thank you for the clarifications. I have read them and decided to maintain my score and assessment of the paper.

Reviewer_2eVnOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is very poorly writen, and it is impossible as a human reader to understand what they are doing so I will not be providing any summary. I detail my concerns in the weaknesses section below

Critical strengths

None

Critical weaknesses

I’m having a hard time trying to understand what the paper is doing because it is written so poorly.

Even reading the first line of the abstract: “A representation with a fixed number of bits or a fixed linear rank must allocate info among the predictors it will support” what does this mean? Are you saying if we compress the representation under a given budget it can’t always be lossless?

Next line: we study how contrastive matching makes this choice when the prediction context arrives after encoding? What prediction context, what encoding? There’s some mention of a finite candidate list, for what exactly?

“For two independent uniform bit strings with later complementary information, we prove…” sorry what does this even mean, how does this fit in this context? I could say the same for nearly every line of the abstract written after it

The introduction is also poorly written, first para talks about what contrastive learning does, then talks about reuse in downstream tasks (which might not always be great, is what i understand from their writing)

Next is so confusing: “when a representation cannot retain everything useful, this becomes a question of allocation” allocation of what? This seems to be a big leap of faith what if the representations have the information but it’s a matter of decoding them properly. Why have we turned the question of poor reuse in downstream tasks into a matter of information allocation? This is very vague.

So many sentences just read as gibberish: “the value of retained information also depends on what the predictor will know later” The paragraph then talks about matching and prediction, why? Where did this come from? What is representation constraint, and what do you mean later source observation?

Next, the paper seems to give an example about two 8-bit strings and prediction and matching losses, but without writing what they mean it’s hard to make sense of.

After reading the introduction, I have very little idea of what the paper is trying to do, given how poorly it’s written.

This doesn’t stop at the introduction. The first line of section 2 (which is supposedly the setup). I thought the paper is linked to contrastive losses. The way it starts “an encoder observes Y and stores Z before the context V of a later request arrives” sorry what encoder Y, what is this supposed to denote, how is this linked to the problem you tried to motivate in the introduction?

Then the paper defines bayes risk over all predictors minimizing expected loss over what? What is the problem at hand, why is this problem being studied?

Reading any of the next 4-5 paragraphs it just reads as words written by some LLM in a compressed format. Policy allows it, sure, but why are we discussing these things? There is little connection or flow in writing. I can’t make any sense of it.

Overall the way the paper is written now, maybe there is some hidden value in there, but it’s next to impossible to parse I have to recommend a reject.

Questions for authors

The problem being studied, the motivation everything is super unclear which makes it hard to ask any meaningful questions during the review process. In my opinion, it has to be re-written in a way that makes it readable, only then one can ask some questions or evaluate the significance/relevance of the work

1 comment

Thank you for pointing out where the introduction became hard to follow. As we developed the paper through discussions with LLM agents, we let the technical details get ahead of the motivation. In the next draft, we'll use Figure 1 to explain the two tasks before introducing the notation.

An encoder saves a description of an input, called a code. A decoder reads it later. Suppose the code for a picture says only "dog present." That can help you choose between "a dog catching a ball" and "a cat on a sofa." It leaves open what the dog is doing. Can training to choose from a short list make it harder to predict the answer without that list?

In Figure 1, the encoder sees two independent, uniformly random eight-bit strings, AA and BB. Together they form the input YY. The encoder saves eight bits total as its code ZZ. Only after those bits are saved does the decoder learn which string to predict: AA on 75% of requests and BB on the remaining 25%. The requested string is XX. On 90% of requests, the decoder also receives the other string as a clue. The request and any clue form the context VV.

For prediction, the decoder assigns probabilities to all 256 possible answers. For matching, it also sees two strings in random order, the requested string and an independent random draw. The loss is minus the log probability assigned to the correct string in prediction and to the correct position in matching.

Could a better decoder recover the information? A weak decoder can miss useful information. Here we can count the bits even the best decoder has to guess. An XOR bit records whether the strings differ at one position. Given either string and the eight XOR bits, the decoder recovers the other.

Eight bits savedAnswer bits unknown with the clue (90%)Without the clue (10%)
Eight XOR bits08
First bit of one string, second bit of the other, six remaining XOR bits17

Each unknown bit costs one bit of prediction log loss here. The best possible loss, averaged over the strings and requests, goes from 0.8 to 1.6 bits per request. This minimum is the Bayes risk. Matching goes the other way. The one known bit available without the clue rules out half the random alternatives, while seven known bits with the clue usually settle the match. The best matching loss falls from 0.104 to 0.057 bits per list. Proposition 2 shows when a high matching score forces extra prediction loss for any code of the allowed size.

We saw this in training too (p. 4). In two of the six settings, all three runs got better at matching while losing information needed for prediction. In both settings, the starting code was already optimal for prediction.

We'd be glad to walk through either calculation or clarify another part of the setup.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.