- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors study which features JEPA encoders retain when representation capacity is restricted. The paper argues that selection depends on the joint predictive value of feature sets, and not necessarily on each feature’s individual predictability. Under a fixed latent distribution and Gaussian assumptions, the authors reformulate optimized squared prediction error as maximizing mutual information between consecutive representations. The objective is then expressed in terms of the complete set of features recoverable from an encoder.
Critical strengths
In my view, a nice contribution of this work is that a feature can be worth representing because it helps predict other features, even when its own future is unpredictable. The traffic-light experiment is well designed for establishing this. The next light is independently sampled, while the usefulness of the current light for predicting image persistence is varied through . Figure 2 shows that the light becomes substantially more decodable as this usefulness increases. I like that the manipulation changes the light’s predictive role without making its future more predictable.
The paper is also clear about several limitations, e.g., regularization sensitivity, non-monotonic effects, etc.
Critical weaknesses
I would like to ask the authors about the feature set theorem. The definition of feasibility is existential, where a set is feasible when it is the complete represented feature set of some encoder. Appendix A.2 then establishes information preservation by noting that scaled latent-coordinate functions themselves belong to . This makes the equality between latent mutual information and complete-feature-set mutual information valid by construction. However, it seems to me that there is a gap between
Selecting an optimal encoder is equivalent to selecting its complete recoverable feature set.
and:
Given a collection of identifiable observation features, the characterization predicts which ones training will retain.
I do not see how the current formulation independently characterizes which combinations of named features are feasible, or how approximate recoverability should enter the optimization, nor which compressed combinations of features are competitive alternatives. These difficulties largely seem to be absorbed into the definition of the feasible family. Would I be missing something?
Another thing to note is that the experiments do not calculate the proposed objective for competing representations or derive a quantitative switching condition from it. They demonstrate effects consistent with the characterization, but do not distinguish it from the more general explanation that a shared restricted representation allocates resources toward information useful for prediction.
Questions for authors
What additional explanatory or predictive insight does the feature-set formulation provide beyond the latent mutual-information objective? And how do the experiments establish that additional value?
2 comments
Thank you for your explanation. Yes, it does resolve my comments, and I will keep my positive score.
Turning “information useful for prediction” into predictions about features
Thank you for the generally positive review, we are glad that you found our experiments well designed. Thank you also for engaging so closely with the formalization. Your questions point to places where the paper should be clearer, and the revised PDF that we uploaded clarifies them: what our characterization does and does not claim about named features, why we do not calculate the objective, what our experiments distinguish predictive closure from, and how approximate recoverability enters.
To clarify our claim: we do not claim to predict which features training will retain given only a collection of identifiable observation features. That would require the full list of features and their relations, which is intractable even in our very simple data. For the same reason, we do not calculate the objective for competing representations or a switching condition, and the formulation constrains which combinations of named features are feasible only through the latent's entropy. The implications section now states this at its opening, just before we present the predictions (line 236).
What the characterization does provide is predictions that are conditional on which other features are represented, and these predictions remain useful without the full map of features and their relations. For example, if the images are represented, the traffic light adds between zero (q=0, when it does not determine which image will be resampled) and one bit (q=1, when it decides which image will be resampled each time) of predictive information while always taking up one bit of the latent's capacity, so a larger q can only favor representing the light. This prediction is novel, contradicts predictions made by earlier views (e.g., Pendharkar, 2026; Zhang et al., 2026), and is validated by our traffic-light experiment. The implications section now states the prediction in this conditional form (lines 266-269).
This was not clear enough in the paper, and we have now clarified it. In our formalization each feature is either represented or not. An image, however, has many features, and a probe can identify the image from the latent only if some of them are represented. The probe's accuracy therefore approximates the representation of the image's features, in the sense of how much information can be extracted from the features that are represented. A rise in accuracy means that more of the image's features are represented, not that a single feature is represented more accurately or is more likely to be in the selected set. We revised the wording for this across the entire paper, including the abstract and the introduction, and the experimental setup now explains what the probe's accuracy approximates (lines 313-318).
Predictive closure agrees with this explanation and adds to it. Like the latent mutual-information objective you mention, the general explanation speaks of information useful for prediction, but it is not about features, so it is unintuitive to use it to reason about which features will be represented. Predictive closure expresses the same objective over features, which allows us to understand feature selection and to make the conditional predictions above (lines 216-230). Our experiments distinguish predictive closure from recent work which also makes claims about features (e.g., Pendharkar, 2026; Zhang et al., 2026): under these works, the light would not be represented no matter what its predictive relations are. We revised the paper to make this clearer: the traffic-light results section now states what our experiments distinguish predictive closure from (lines 357-358).
Please let us know if this resolves your concerns.