acceptodds
Under review as a conference paper at ICLR 2027

Frozen in a Frame: The Velocity Blind Spot in JEPA World Models

21 views

Abstract

Joint-embedding predictive architectures for world modeling train an encoder so that a predictor can map a current frame embedding and an action to the embedding of the next frame. The target for this prediction always comes from a single rendered frame. We show that this choice has a structural blind spot. Every renderer without motion blur is instantaneous: it draws a scene from its configuration alone and leaves no trace of how fast that configuration is changing. So a single-frame embedding is a deterministic function of configuration alone, and it carries no information about velocity. This holds for the optimal predictor under any encoder, however large, including the official released LeWM weights. We confirm this on official checkpoints across four real benchmarks, PushT, Reacher, Cube, and TwoRoom, where every linear velocity probe sits at or below its own chance floor while position probes reach . We then introduce RateIdent, a three-stage diagnostic protocol, and TI-JEPA, a lightweight fix that splits the latent into a pose code and an explicit finite-difference motion code and predicts both. Across three physically grounded environments, TI-JEPA gives a statistically significant, seed-robust improvement on a stop-at-goal planning task over a matched-memory baseline, for example a 55% lower final distance on Pendulum () and 64% lower on CartPole (). We reproduce the theory's central prediction at official ViT-Tiny plus AdaLN-transformer scale on two environments, then push the same recipe onto real dm_control Reacher photographs trained entirely from scratch, where TI-JEPA's kill-experiment branch separation exceeds the memory-having baseline's by roughly , the largest margin in the paper. We also compare directly against a recurrent RSSM-style predictor at the same footprint. Its opaque hidden state matches or beats TI-JEPA's raw rollout accuracy on two of three environments, but it cannot be probed for pose and motion separately the way TI-JEPA's explicit split can, and it loses outright on the environment with the most coupled dynamics. Together, a checkable formal argument and six evaluated environments show that single-frame targets are the wrong object to predict when velocity matters, and that a small, interpretable structural change fixes it without any privileged supervision.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
world models, joint-embedding predictive architecture, identifiability, self-supervised representation learning, video prediction, latent dynamics, velocity estimation, model-based planning, recurrent state-space models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_TaUdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper identify a "problem" for jepa based model like LeWM. These models map a current frame embedding and action to the embedding of a future frame, and velocity information is missing in this paradigm. The authors introduce RateIdent, a three-stage diagnostic protocal to probe the problem, and TI-JEPA, a a lightweight fix that splits the latent into a pose code and an explicit finite-difference motion code, and predict both infomation together.

Critical strengths

None

Critical weaknesses

W1:(main issue) The authors claim that "single-frame targets are the wrong object to predict when velocity matters". However, if historical information is included in prediction, namely, if model have a "context window" for last few frames, velocity and other dynamics can be learned by predicting single frames. That is because these information can be encoded in context window and learned by model. As most world models, including LeWM(mainly used in paper), actually have context window, the claim is not valid. This is a fundamental flaw in the paper.

Even the authors' experiment faithfull reflect this issue. LeWM has 3 frames of history, and in table 2, baseline k = 3 have the ability to predict velocity, even better than TI-JEPA in InertiaBall and Pendulum env. Why are authors comparing with "single frame" LeWM, or memoryless LeWM? This not a practical case. And TI-JEPA also use past 3 frames information, the comparison for single frame baseline is not fair.

W2: Assume the authors' method is somehow helpful, can add some velocity information and surpass memory-having baselines. The authors claim that "TI-JEPA’s kill-experiment branch separation exceeds the memory-having baseline’s by roughly 38×". I read through chapter 7.6, and here are my questions: Why the rest of paper use official LeWM checkpoint, but here a trained-from-scratch ckpt is used? How is the performance of this trained-from-scratch ckpt on other tasks? How is the performance of offical ckpt? And branch separation is a strange metric, in my understanding, it meansures the end pos of arm. TI-JEPA achieves 2.166, but what does greater than 1 means? Does greater means better? Why for rollout position MSE, the baseline is better? All these lack further analysis.

Unfortunately, unless the authors can provide explanation and more convincing evidence, this paper do not meet the standard of acceptance.

Questions for authors

See weakness.

Reviewer_mBvVOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper focuses on the JEPA architectures, and argues that JEPA encoders fail to encode velocity due to the single frame target, preventing world models from accurately model dynamics. They propose a new architecture, TI JEPA, which builds on LeWM (JEPA based world model) and adds context frames as input to the encoder, so as to predict velocity alongside a pose representation. Both are then fed to the predictor, which explicitly predicts next frame velocity alongside the next observation. Authors evaluate their method with respect to LeWM and recurrent approaches using three tests: velocity probes, predictions with opposite velocity and an inertia test on simple environments such as cartpole and pendulum.

Critical strengths
  • As JEPA models become more popular, it is interesting to dive into the potential limitations of this approach. Understanding the impact of relying on single frame encoders (similarly to image vs video VAEs) would provide a lot of useful insights.
  • Authors have made efforts to mathematically ground their work and justify their claim.
  • Authors have carried out a large scale of experiments, considering multiple settings and architectures. They have also made efforts to design evaluation protocols to tests different aspects of velocity modelling.
Critical weaknesses
  • The presentation of the paper is very poor: The text is very terse, with important details needed to understand the paper in the appendix, or not present at all. For example, tables 1, 2 and 4 are never referenced to in the text and their results are never discussed or contextualised, it is not clearly stated what Table 1 reports, the explanations of protocol B and C are very confusing and the concrete set up is not discussed in the main paper, authors should clarify what all variables mean, such as \sigma, H and o_g. Importantly, 14 references are missing and are replaced by question marks in the text. There is also a poorly formatted reference as “Fast-lewm”. A simple proof reading would have caught this.
    The method is barely explained in the main paper. Author only provide a high level overview and do not provide details on the base architecture, how velocity is learned, how this is connected to the predictor, and how the model is trained. For example, what are loss functions? What is the training signal used for velocity prediction? Is the encoder trained separately, or are encoder and predictor trained jointly? If it is the latter, how does that impact the quality of the encoding? A figure should not replace a concrete description of the work.
  • Authors run unfair comparisons with baselines and claim to have an edge. TI JEPA sees three frames of context in the encoding stage, yet several experiments compare to LeWM variants that only see one frame of history. While some comparisons are carried out with 3 frames, the number of context frames arbitrarily changes in protocol C for example. The core question we should be answering is whether moving context from the predictor to the encoder, and explicitly modelling velocity makes a difference. Results do not show clear evidence that one is consistently better at the world modelling task.
    More concerning, the authors claim that adding three frames of context to the predictor was “risky” and led to “false negative” results because velocity information was part of the context. Rather than a false negative, this highlights that the method shows some redundancy with the way LeWM is designed, and that existing models are capable of extracting velocity information. This should be explored instead of being called a false negative. An interesting question would be to find out whether explicitly modelling velocity is better than a large transformer extracting the information implicitly.
  • All experiments focus on situations where one object is moving on a simple static background. How would this approach transfer to more general situations where complex articulated objects are moving in dynamic scenes? In such settings, wouldn’t implicit learning of velocity from the context be more helpful.
Questions for authors
  • Authors repeatedly put the emphasis on rendered environments without motion blur. How crucial is the lack of motion blur? How do finding translate in real world data that comprises motion blur? Have you trained the model on more complex datasets?
  • What were obtained results when using three frames of context in the predictor for protocol C, and for the T1 JEPA model itself? What were the results that the authors refer to as false negative? Can authors expand on why they call this a false negative?
  • What are the missing references throughout the paper?
  • Why was the number of baseline context frames set to 1 in Protocol C experiments?
Reviewer_ACfYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether single-frame JEPA representations can capture velocity. It shows that, when images depend only on position and configuration, a single-frame embedding cannot uniquely identify velocity. The paper introduces RateIdent, an evaluation protocol based on velocity probes, opposite-velocity rollouts, and planning. It also proposes TI-JEPA, which separates pose features from motion features computed from consecutive frames. Experiments cover released LeWM checkpoints, controlled physics environments, and simulated Reacher images. TI-JEPA improves planning over a single-frame baseline in three controlled environments. Results against history-based and recurrent baselines are mixed.

Critical strengths
  • The opposite-velocity experiment gives a clear example: two identical images can require different future predictions. Combining this test with probes and planning is more informative than reporting representation quality alone.

  • The paper evaluates released checkpoints, uses episode-level probe splits, and includes history-based and recurrent baselines. It also reports the PushT planning failure and the weak velocity results on Reacher. These results help define where the proposed method is useful.

Critical weaknesses
  • Proposition 1 shows that velocity cannot be uniquely recovered from one frame under the stated assumptions. It does not show that the embedding contains no statistical information about velocity. It also does not show that a predictor using multiple frames cannot predict futures that depend on velocity.
  • Corollary 1 considers only predictors of the form pred(zt,at)pred(z_t, a_t), but the standard baseline uses a history of frames. In addition, Appendix A.5 needs clearer conditions for its claim that finite differences allow velocity to be uniquely recovered. A finite difference usually approximates instantaneous velocity; it does not guarantee exact recovery. These gaps make it difficult to tell which limitations come from the observations, the predictor inputs, or the prediction target.
  • The main planning results in Table 3 compare TI-JEPA, whose motion code uses several frames, with a baseline that receives only one frame. This comparison cannot isolate the benefit of the proposed structure from the benefit of additional observations. Comparisons with history-based and recurrent models are more informative, but TI-JEPA has lower planning success than both on InertiaBall and Pendulum in Table 5. The Reacher result also needs a more careful interpretation. The reported 38× gain concerns separation magnitude, while sign accuracy is near chance at 0.525 and rollout error is higher than the history baseline’s, at 0.230 versus 0.041. Together with the unresolved sign convention issue in Appendix F, these results leave uncertainty about whether the larger separation reflects more accurate motion prediction.
Questions for authors
  • Which conclusions apply to single-frame embeddings, and which also apply to predictors that use frame history? Proposition 1 rules out unique recovery of velocity, but does it rule out statistical information about velocity? What prevents a history-based predictor trained on single-frame targets from learning velocity-dependent transitions?

  • Under what assumptions do the constructions in Appendix A.5 allow velocity to be uniquely recovered? Please clarify which claims concern exact instantaneous velocity, finite-difference approximations, or linear decoding.

  • How do the planning comparisons account for TI-JEPA using several frames to construct its motion code? Given the mixed results against history-based and recurrent baselines in Tables 5 and 11, what evidence separates the benefit of the proposed structure from the benefit of additional observations?

  • On Reacher, what does the 38× increase in branch separation demonstrate when sign accuracy is near chance and rollout error is higher than the history baseline’s? Could the unresolved sign convention issue in Appendix F affect this interpretation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.