acceptodds
Under review as a conference paper at ICLR 2027

Characterizing How JEPA Constructs Representations

Abstract

In joint-embedding predictive architectures (JEPA), an encoder and a predictor are trained together to produce a representation that minimizes the error in predicting the representation of the next observation. Because the representations of both the current and the next observation are learned, it is unclear which features they will include. Earlier works portray JEPA as selecting features based on their individual predictability, raising the concern that control-relevant features may be omitted if they are hard to predict. To address this concern and better understand how JEPA selects which features to encode, we propose the predictive closure characterization. Based on a reformulation of the JEPA training objective, we show that selection depends on joint properties of features. We support this characterization with experiments showing that unpredictable features can become better represented when they help predict other features, and that changing a feature's predictive relationships can affect how well others are represented. We also demonstrate implications for steering feature selection: adding an auxiliary objective targeting one feature can strengthen the representation of others, and supplying a feature as side information to the predictor can change how well other features are represented. We hope this work helps practitioners understand and steer JEPA representations and guides researchers in improving how these systems learn.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Joint-embedding predictive architectures, JEPA, world models, predictive closure, representation learning, feature selection, self-supervised learning.

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_UAnoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors study which features JEPA encoders retain when representation capacity is restricted. The paper argues that selection depends on the joint predictive value of feature sets, and not necessarily on each feature’s individual predictability. Under a fixed latent distribution and Gaussian assumptions, the authors reformulate optimized squared prediction error as maximizing mutual information between consecutive representations. The objective is then expressed in terms of the complete set of features recoverable from an encoder.

Critical strengths

In my view, a nice contribution of this work is that a feature can be worth representing because it helps predict other features, even when its own future is unpredictable. The traffic-light experiment is well designed for establishing this. The next light is independently sampled, while the usefulness of the current light for predicting image persistence is varied through qq. Figure 2 shows that the light becomes substantially more decodable as this usefulness increases. I like that the manipulation changes the light’s predictive role without making its future more predictable.

The paper is also clear about several limitations, e.g., regularization sensitivity, non-monotonic effects, etc.

Critical weaknesses

I would like to ask the authors about the feature set theorem. The definition of feasibility is existential, where a set is feasible when it is the complete represented feature set S(f)S(f) of some encoder. Appendix A.2 then establishes information preservation by noting that scaled latent-coordinate functions themselves belong to S(f)S(f). This makes the equality between latent mutual information and complete-feature-set mutual information valid by construction. However, it seems to me that there is a gap between

Selecting an optimal encoder is equivalent to selecting its complete recoverable feature set.

and:

Given a collection of identifiable observation features, the characterization predicts which ones training will retain.

I do not see how the current formulation independently characterizes which combinations of named features are feasible, or how approximate recoverability should enter the optimization, nor which compressed combinations of features are competitive alternatives. These difficulties largely seem to be absorbed into the definition of the feasible family. Would I be missing something?

Another thing to note is that the experiments do not calculate the proposed objective for competing representations or derive a quantitative switching condition from it. They demonstrate effects consistent with the characterization, but do not distinguish it from the more general explanation that a shared restricted representation allocates resources toward information useful for prediction.

Questions for authors

What additional explanatory or predictive insight does the feature-set formulation provide beyond the latent mutual-information objective? And how do the experiments establish that additional value?

2 comments
AuthorsOpenReview ↗

Turning “information useful for prediction” into predictions about features

Thank you for the generally positive review, we are glad that you found our experiments well designed. Thank you also for engaging so closely with the formalization. Your questions point to places where the paper should be clearer, and the revised PDF that we uploaded clarifies them: what our characterization does and does not claim about named features, why we do not calculate the objective, what our experiments distinguish predictive closure from, and how approximate recoverability enters.

it seems to me that there is a gap between "Selecting an optimal encoder is equivalent to selecting its complete recoverable feature set." and: "Given a collection of identifiable observation features, the characterization predicts which ones training will retain." I do not see how the current formulation independently characterizes which combinations of named features are feasible [...] the experiments do not calculate the proposed objective for competing representations or derive a quantitative switching condition from it.

To clarify our claim: we do not claim to predict which features training will retain given only a collection of identifiable observation features. That would require the full list of features and their relations, which is intractable even in our very simple data. For the same reason, we do not calculate the objective for competing representations or a switching condition, and the formulation constrains which combinations of named features are feasible only through the latent's entropy. The implications section now states this at its opening, just before we present the predictions (line 236).

What the characterization does provide is predictions that are conditional on which other features are represented, and these predictions remain useful without the full map of features and their relations. For example, if the images are represented, the traffic light adds between zero (q=0, when it does not determine which image will be resampled) and one bit (q=1, when it decides which image will be resampled each time) of predictive information while always taking up one bit of the latent's capacity, so a larger q can only favor representing the light. This prediction is novel, contradicts predictions made by earlier views (e.g., Pendharkar, 2026; Zhang et al., 2026), and is validated by our traffic-light experiment. The implications section now states the prediction in this conditional form (lines 266-269).

or how approximate recoverability should enter the optimization

This was not clear enough in the paper, and we have now clarified it. In our formalization each feature is either represented or not. An image, however, has many features, and a probe can identify the image from the latent only if some of them are represented. The probe's accuracy therefore approximates the representation of the image's features, in the sense of how much information can be extracted from the features that are represented. A rise in accuracy means that more of the image's features are represented, not that a single feature is represented more accurately or is more likely to be in the selected set. We revised the wording for this across the entire paper, including the abstract and the introduction, and the experimental setup now explains what the probe's accuracy approximates (lines 313-318).

They demonstrate effects consistent with the characterization, but do not distinguish it from the more general explanation that a shared restricted representation allocates resources toward information useful for prediction.

Predictive closure agrees with this explanation and adds to it. Like the latent mutual-information objective you mention, the general explanation speaks of information useful for prediction, but it is not about features, so it is unintuitive to use it to reason about which features will be represented. Predictive closure expresses the same objective over features, which allows us to understand feature selection and to make the conditional predictions above (lines 216-230). Our experiments distinguish predictive closure from recent work which also makes claims about features (e.g., Pendharkar, 2026; Zhang et al., 2026): under these works, the light would not be represented no matter what its predictive relations are. We revised the paper to make this clearer: the traffic-light results section now states what our experiments distinguish predictive closure from (lines 357-358).

Please let us know if this resolves your concerns.

Thank you for your explanation. Yes, it does resolve my comments, and I will keep my positive score.

Reviewer_k5ucOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes a new characterization of JEPA loss called Predictive Closure (PC) that advocates at looking at features sets as opposed to representation which is just a feature in itself. The paper challenges current understanding of JEPA architectures that advocate features are learned based on their predictability. Instead PC states that certain features are predictive while others that are predicted from predictive features are what is learned by JEPAs. Experiments with toy examples or synthetic data are provided as empirical support for PC.

Critical strengths

The paper attempts to provide an answer to the question of what information is really being captured in JEPA latents. This is a critical question as JEPA has emerged as the go-to method to solve forward dynamics or world modeling. JEPA latnets are typically unstructured and it is important for us to develop an understanding of what constitutes a good latent.

Critical weaknesses
  • The predictions made by PC characterization are qualitative but does not provide any guidance on where an optimum or cross-over points be for the first three experiments. Is there a way to make the predictions precise?

  • Experiments based on synthetic data are interesting and in general appear to test the predictions made by predictive closure. But these experiments are conducted on one latent size (3 components) to control capacity. This makes the reader wonder if the results shown in the paper are for one particular setting or general.

  • The datasets used to construct experiments are described in text but an illustration could be more helpful to the reader.

  • There are no real-world/realistic examples that helps the reader relate to what is observed in practice vs what PC predicts. If this were available, then that would add significant value to the paper.

Questions for authors

Please see weaknesses section. In my opinion as it stands this is a borderline paper that would be of interest to ICLR. What would really help me update my understanding and appreciation for the paper is to have precise predictions and perhaps a real/realistic example that can link practice to theory.

2 comments
AuthorsOpenReview ↗

Predictive closure explains a real-robot failure and holds across latent sizes

Thank you for the clear review and for the concrete suggestions, and we are glad you also found this question a critical one to address. Following your comments, we tested other latent sizes and uploaded a revised PDF that explains why our predictions are directional and adds these results, a real-world example, and example frames of each sequence.

The predictions made by PC characterization are qualitative but does not provide any guidance on where an optimum or cross-over points be for the first three experiments. Is there a way to make the predictions precise?

It is true that our predictions are in some cases approximate and directional. Making them precise would require the full list of features and their relations, which is intractable even in our simple setup. Still, even a directional prediction can separate our view from earlier ones: earlier views imply that the traffic light in our running example will not be represented (e.g., Pendharkar, 2026), while predictive closure predicts that it can be if it helps predict other features, as our traffic-light experiment shows. We added this explanation at the opening of the implications section, just before we present the predictions (line 236). The traffic-light results now also state this contrast with earlier views (lines 357-358).

these experiments are conducted on one latent size (3 components) to control capacity. This makes the reader wonder if the results shown in the paper are for one particular setting or general.

We repeated our experiments with other latent sizes. The results match what predictive closure implies. The latent size changes the entropy that the regularizer forces on the latent, and with it which sets of features fit. For example, a small set of mutually predictive features disappears from the latent when the latent grows, because the larger latent now fits a larger and more predictive competing set. We added these results to the paper (lines 414-416; see also this figure).

The datasets used to construct experiments are described in text but an illustration could be more helpful to the reader.

We agree, and added a figure with example frames of each sequence (Figure 7).

There are no real-world/realistic examples that helps the reader relate to what is observed in practice vs what PC predicts.

We added such an example (in the section that explains earlier findings, lines 248-250) from the very recent study by Zhang et al. (2026), who trained an action-conditioned JEPA world model for planning on DROID, a dataset of real robot-manipulation videos. Without an additional loss, the encoder kept the scene and discarded the robot, and planning failed; adding an inverse-dynamics loss (steering the latents to represent the repercussions of actions) restored it. Predictive closure explains both the failure and the fix (lines 1374-1389). The scene stays fixed within an episode, so its features have high predictive information and fill the latent's limited capacity, keeping the robot's state out. The inverse-dynamics loss requires the latent to reveal the action, and, like the reconstruction loss in our steering experiment, it also brings into the selected set the features that are predictively related to the steered ones.

Please let us know if this resolves your concerns.

I thank the authors for their thorough rebuttal. This is a good paper

Reviewer_A5utOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

in this work, the authors present a framework by which we can understand what kind of features are captured by JEPA algorithms. this framework, referred to as predictive closure, is based on the observation that a JEPA algorithm can be thought of as identifying a set of features that maximizes the mutual information between these features of consecutive frames in a video (or a stream of images.)

under this framework, the authors demonstrate empirically (as predicted by their own explanation) that JEPA approaches can capture features that are not temporally predictive (that is, predictable from previous frames or predictive of future frames) as well as they are predictive of other features within the same frame. in hindsight, this is somewhat obvious, since there is no explicit pressure on learning in JEPA for representation to be maximally succinct. if there is a set of features that are predictive of future frames’ features either directly or indirectly, they will likely be captured by the encoder.

of course, being obvious in hindsight is a great feature of this work. experiments are simple yet well designed with focus, and i enjoyed reading this paper, although for others who are more experienced with JEPA and other representation learning literature, findings in this work may be considered obvious not only in hindsight but also in foresight.

Critical strengths

see my answer to the first question.

Critical weaknesses

see my answer to the first question.

Questions for authors

see my answer to the first question.

1 comment
AuthorsOpenReview ↗

Recent descriptions of JEPA imply the opposite of our results

Thank you for the positive review, and we are glad that you enjoyed our paper and found the experiments focused and well designed. Following your comment, we uploaded a revised PDF that addresses the issue you raised: it cites the recent works below in our background section (line 150) and explains where their per-feature view agrees with ours (when features evolve independently) and where it differs (lines 200-209).

for others who are more experienced with JEPA and other representation learning literature, findings in this work may be considered obvious not only in hindsight but also in foresight.

We agree that the result can seem natural once the objective is written over sets of features. However, we believe this finding is non-obvious to many of the people currently working on JEPA. For example, see how some recent work describes JEPA's feature selection as being done based on each individual feature’s predictability:

  • H-JEPA (Zhang et al., 2026, arXiv 2610.06805): "the objective selects by predictability."
  • V-JEPA 2 (Assran et al., 2025, arXiv 2506.09985): JEPA learns the predictable aspects of a scene while "ignoring unpredictable details".

These descriptions do not match the results of our experiments and imply that the prevalent view of people that are currently working on JEPA is different from what our paper suggests. For example, they imply that the traffic light in our running example would be dropped, and that an auxiliary loss on one image would not help represent another. Predictive closure predicts the opposite in both cases, and our traffic-light and reconstruction experiments show both. The revised version now states this contrast in our traffic-light results (lines 357-358).

If the revision addresses the issues that prevented you from giving a score of 4, we would appreciate it if you would update it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.