acceptodds
Under review as a conference paper at ICLR 2027

Learning Visuomotor Policies from JEPA Latent Transition Models

Abstract

Video pretraining offers more than visual representations: predictive models also learn how visual states evolve. The success of recent generative world-action models (WAMs) suggests that pretrained transition models can themselves provide a transferable foundation for policy learning. We ask whether this principle extends to latent transition models learned through joint-embedding predictive architectures (JEPAs): beyond transferring the visual encoder, can a predictor that consumes robot actions be repurposed into a visuomotor policy that produces them? We introduce LTM-Policy, which adapts the latent transition model through two complementary routes. A policy route employs learnable action queries to generate robot actions, while a world route retains action-conditioned prediction of future representations from demonstrated transitions. At deployment, the resulting policy generates actions without pixel reconstruction, future-video rollout, or model-predictive search. Controlled experiments show that pretrained transition priors remain beneficial after repurposing the model for action generation. Retaining future-latent prediction during adaptation provides an additional 5.4-point benefit, and the transfer advantage persists in low-data regimes with limited robot demonstrations. Across the four LIBERO suites, LTM-Policy achieves 97.4% average success using only a single forward pass at deployment, without pixel reconstruction or future-video rollout. We further evaluate the method on LIBERO-Plus and real-world manipulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Visuomotor Policy, Robot Policy, JEPA

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NQ6wOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether a pretrained JEPA transition predictor can serve as a direct visuomotor policy backbone. LTM-Policy combines an action-query policy route with an action-conditioned future-prediction route, using a frozen visual encoder and a flow-matching action head. It reports 97.4% average success on LIBERO, controlled initialization and future-supervision ablations, and evaluations on LIBERO-Plus and three real-robot tasks.

Critical strengths
  1. The paper asks a useful, focused question beyond visual encoder transfer. With architecture and observations fixed, pretrained predictor weights improve LIBERO-Long from 83.2% to 94.4% under the same future-supervision objective (Table 3).

  2. The initialization-by-objective comparison is informative. Pretraining also helps with action-only training, while future supervision adds another 5.4 points to the pretrained model on Long. This separates two plausible sources of improvement.

  3. The policy avoids video rollout and search at deployment. The low-data results and physical demonstrations make the transfer idea more convincing than benchmark accuracy alone.

Critical weaknesses
  1. Table 3 removes future supervision as a whole, without separating the policy-future and world-future losses. It therefore does not establish whether the proposed dual-route design is needed beyond a simpler policy with one auxiliary prediction loss.

  2. The benchmark comparisons mix task-ID conditioning with language-conditioned policies. The paper acknowledges this, but Table 2 still includes an unaffected language category in the 67.4% headline average; the non-language score is 62.6%. Matched comparisons would better support the robustness claim.

  3. Uncertainty around the gains is not reported across independent training runs. The real-robot study has only 20 trials per task and no comparison policy, so it establishes feasibility more clearly than a transfer advantage on hardware.

Questions for authors
  1. Can you separate the two future losses while keeping action-supervised updates and training compute matched? Which route accounts for the improvement on Long?

  2. How does LTM-Policy compare with a strong baseline under the same task conditioning and camera inputs, particularly on the non-language LIBERO-Plus categories?

  3. How stable are the Table 3 gains across training seeds, and is there a matched action-only or random-initialization baseline for the physical tasks?

Reviewer_1qKgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper repurposes a latent transition model learned through video pretraining within the JEPA framework into a visuomotor policy for downstream control. At deployment, the resulting policy generates actions without pixel reconstruction, future-video rollout, or model-predictive search.

Critical strengths
  1. The paper has a clear motivation, and the method is simple and easy to understand. The experimental results in Table 3 show that initialization with pretrained latent transition model weights is more beneficial for downstream policy learning than random initialization.
  2. The experimental evaluation is relatively comprehensive, covering LIBERO and LIBERO-Plus with perturbations, as well as three real-world tasks. These experiments demonstrate the method’s value for practical deployment.
Critical weaknesses
  1. The paper notes that existing world models (WAMs) have already explored repurposing pretrained transition models into policies. This work mainly extends this transfer idea to the JEPA architecture, resulting in a relatively limited conceptual advance over existing methods. The authors should clarify what unique value the proposed method offers beyond adopting the JEPA architecture compared with these existing WAM approaches. Furthermore, during downstream learning, the original action inputs to the pretrained Latent Transition Model are replaced with action queries, thereby changing the model’s input conditioning. Under this change, can the pretrained latent transition model still reliably predict the future, and can the policy route still reliably leverage the pretrained transition priors to repurpose the model into a policy?
  2. The current experimental results do not appear to establish that transition-prior knowledge is indeed transferred from the pretrained Latent Transition Model to the downstream policy. The results in Table 3 support the benefit of initializing from the pretrained Latent Transition Model over random initialization, but do not demonstrate that the downstream policy actually uses the pretrained transition-prior information. It is also possible that pretraining simply provides a better parameter initialization than random initialization.
Questions for authors
  1. Given that WAM has already demonstrated how to repurpose pretrained transition models into policies, please clarify what distinct value this work offers beyond extending this approach to JEPA architectures.
  2. Please clarify whether the Latent Transition Model remains reliable when its conditioning inputs change from actual actions during pretraining to action queries during policy learning, and whether the policy route can still reliably leverage the pretrained Latent Transition Model under this change.
  3. Please provide additional controlled experiments demonstrating that transition priors are indeed transferred from the pretrained Latent Transition Model and contribute to downstream policy learning. For example, a comparison with pretraining without action conditioning, or other experiments that provide evidence of the transfer of transition priors, would help address this concern.
3 comments

Thank you for the clarifications and the additional experiments. The response explains the practical advantages of deployment without video generation or planning. However, beyond extending the idea of transfer learning to the JEPA framework, it remains unclear what substantive conceptual advance this work makes over existing WAM approaches. The additional experiments show that the benefit of pretrained initialization does not depend on downstream future prediction supervision, but they still do not establish that the improvement in downstream performance is due to action-conditioned transition knowledge. I therefore maintain my score.

Part 1: Conceptual advance beyond existing WAMs

We appreciate that the reviewer acknowledges the practical advantages of our work, particularly that LTM-Policy avoids pixel generation and planning, enjoying greater training and inference efficiency.

We would like to further clarify that our conceptual contribution is not simply to replace a video-generation WAM backbone with JEPA. Our central perspective is that existing WAMs can be understood more generally as transition-model prior transfer: a video generation model is, at its core, a state-transition model in pixel space, and its learned predictive structure can be repurposed for control. From this perspective, the transferable object is not necessarily the ability to generate pixels, but the learned structure of how states evolve.

This reinterpretation motivates an important question: does transition-model transfer still work when prediction occurs entirely in latent space, without pixel generation? This question also connects to the broader community interest in pixel-space versus latent world models. Existing WAMs establish predictive transfer through pixel-generative models; LTM-Policy shows that the same principle extends to a latent transition model, without pixel reconstruction, future-video rollout, or planning. We therefore view the contribution as testing a broader transfer principle, rather than simply substituting one architecture for another.

More broadly, this abstraction is meaningful as it opens a larger design space for policy transfer. A video generation model can be viewed as a transition model in pixel space, a JEPA predictor as one in latent space, and other predictive models, such as image-editing or transformation models, may encode transition structure in yet other forms. We believe this broader perspective is an important conceptual value of the work: it reframes WAMs as one instance of a more general family of transferable transition models and inspires broader future works.


Part 2: Transition knowledge versus action conditioning

We would also like to clarify a distinction in the reviewer’s second concern. Our claim is about the transfer of state-transition knowledge, rather than about establishing action conditioning itself as the source of the downstream gain.

Action conditioning is one particular mechanism through which the V-JEPA2-AC predictor learns transition structure, but it is not the defining property of the transition prior studied in this work. A transition model may be action-conditioned or action-free; the central question of our paper is whether predictive knowledge about how states evolve can be reused for downstream policy learning.

We believe the concern about whether the gain can be specifically attributed to action-conditioned transition knowledge conflates two distinct questions: whether pretrained transition knowledge transfers to policy learning, which is the hypothesis studied in this paper, and whether action conditioning during pretraining is itself the causal source of that transfer, which is not a claim we make. Our experiments directly target the former: compared with the same architecture trained from random initialization, the pretrained transition model provides a consistent downstream advantage; this advantage persists after downstream future-prediction supervision is removed and remains stable across independent training runs.

Therefore, requiring the downstream gain to be specifically attributed to action conditioning imposes a stronger and different claim than the one advanced by our work. Our contribution is to establish transition-model prior transfer as the broader principle.


We hope this clarification better conveys both the intended scope and the broader conceptual contribution of the paper, and we respectfully invite the reviewer to reconsider the work from this perspective.

Dear Reviewer 1qKg,

Thank you again for your time and thoughtful feedback. Since the discussion period is closing soon, we wanted to follow up on our latest response and ask whether it addresses your remaining concerns regarding the conceptual contribution and transition-prior transfer.

If our clarification has helped resolve these concerns, we would greatly appreciate it if you could reconsider your assessment. We would also be happy to clarify any remaining points before the discussion closes.

Reviewer_e8yEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes LTM-Policy, which uses a pretrained V-JEPA 2-AC predictor as the backbone for learning a policy. The proposed method feeds images into a V-JEPA 2 encoder, then feeds visual tokens, learned action and future queries into a V-JEPA 2-AC predictor, which predicts action and future features. Action features go into a policy head for imitation learning, and future features are compared against the future ground-truth for a world model prediction objective. Overall the approach makes sense. The experiments and empirical analysis support the main claim that a pretrained transition predictor can be a useful policy trunk, and ablations show that future prediction can improve policy performance.

Critical strengths
  • The idea of using a pretrained latent transition model as a policy trunk overall makes sense.
  • The paper puts out a clearly scoped claim and supports it well. The resulting policy achieves decent performance on the LIBERO eval suites, and works on a real robot.
  • Extensive baselines and evaluations in simulation.
  • The pretrained vs. random predictor ablation and the action vs. action + future ablations show that pretraining and the future prediction objective both helps with downstream policy performance in this setting.
Critical weaknesses
  • Missing a regular behavior cloning baseline/ablation with V-JEPA 2 directly going into the policy head. This would tell us how much the policy trunk itself matters.
  • Real evaluations have no baselines.
Questions for authors
  • Ablation/baseline: how well does V-JEPA 2 + action head vanilla imitation learning do, without the trunk?

Comments on the paper

2 comments
Area_Chair_LCXxOpenReview ↗

Discussion with the authors

Dear reviewers,

the authors have provided responses for your reviews. Reviewer 1qKg, thank you for letting the authors know that you assessment remains unchanged. Can the others please let them know if your comments have been addressed?

Thank you, Your AC

AuthorsOpenReview ↗

Summary of rebuttal at the end of the discussion period

Additional evidence added

Dear Area Chair and Reviewers,

Thank you for coordinating the discussion and for the reviewers’ careful evaluation. Across the reviews, there was broad agreement that the paper investigates a clear and focused question, that pretrained transition-model initialization benefits downstream policy learning, and that the evaluation is broad. The main concerns involved isolating the gains, decomposing the objectives, evaluation fairness, and result stability. We added targeted experiments addressing each point.

  • Beyond visual encoder transfer. We added an encoder-only BC baseline using the same frozen V-JEPA 2 encoder and policy inputs. LTM-Policy improves by 4.4 points on LIBERO-Goal and 25.0 points on LIBERO-Long, showing clear benefit beyond visual representation transfer.

  • Complete objective ablation. We independently ablate action prediction, policy-future prediction, and world-future prediction under matched action supervision and training compute:

SettingActionPolicy-futureWorld-futureLIBERO-Long
Action only✓––89.0
+ Policy-future✓✓–92.4
+ World-future✓–✓90.0
Full LTM-Policy✓✓✓94.4

Policy-future gives a clear gain, world-future adds a complementary signal, and their combination performs best.

  • Stability across runs. We repeated all four core initialization/objective settings with three seeds:
Predictor Init.ObjectiveLIBERO-Long
RandomAction only82.20 ± 1.56
RandomAction + future82.73 ± 1.81
PretrainedAction only89.40 ± 1.83
PretrainedAction + future92.47 ± 2.23

Pretraining improves LIBERO-Long by 7.20 points with action-only training and 9.74 points with future supervision.

  • Stronger real-world evidence. We added matched internal baselines and OpenVLA-OFT on the same three tasks:
MethodBlockPlugSequential
Random-init12/2011/2012/20
Action-only14/2011/2013/20
Full LTM-Policy14/2012/2015/20
OpenVLA-OFT14/2010/2013/20

The results show a consistent Random-init → Action-only → Full progression, while LTM-Policy matches or exceeds OpenVLA-OFT on all three tasks.

  • Matched robustness analysis. On LIBERO-Plus Goal, using identical task-ID conditioning and dual-view inputs, Full LTM-Policy achieves 62.4% non-language robustness, versus 59.47% for Pretrained Action-only and 57.08% for Random-init Full.

Remaining reservations and scope

These additions strengthen the empirical support for the paper. Reviewer NQ6w acknowledged that the additional ablations, matched controls, and three-seed results addressed several concerns, while retaining reservations about the breadth of the matched robustness evaluation and the 20-trial real-world study. We believe these remaining reservations concern the desired breadth of additional validation rather than the validity of the central claims: the core conclusions are supported by extensive controlled simulation experiments and multi-seed results, while the real-world study serves as supporting evidence of deployment feasibility. We are willing to extend the real-world evaluation to 50 trials per task in the revision. Given the limited rebuttal time, we prioritized adding matched baselines and an external baseline over simply increasing repeated trials, and we would be happy to expand the physical evaluation in the revised version.

Reviewer 1qKg’s remaining concerns include two points: whether the work offers a conceptual advance beyond extending WAM, and whether the downstream gains can be specifically attributed to action-conditioned transition knowledge. We clarified that our contribution is broader than an architectural substitution: we reinterpret existing WAMs as instances of transition-model prior transfer and ask whether this principle still holds when prediction occurs entirely in latent space, without pixel generation, rollout, or planning. We also clarified that the paper’s claim concerns the transfer of state-transition knowledge in general, rather than identifying action conditioning as the sole causal source of the gain. Our controlled comparisons directly support this intended claim, while the remaining reservation may refer to a narrower causal attribution than the claim advanced in the paper.

We therefore view the remaining reservations as reflecting differences in the expected breadth of validation and scope of causal attribution, rather than specific unresolved weaknesses in the reported experiments or central claims. We hope this summary helps consolidate both the evidence during rebuttal and the remaining points of discussion. We sincerely appreciate the Area Chair’s and reviewers’ time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.