acceptodds
Under review as a conference paper at ICLR 2027

V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents

19 views

Abstract

World-action models (WAMs) couple future visual-state prediction with action generation. By adapting video generators or image-editing models pretrained at scale, a prominent line of recent WAMs inherits both predictive knowledge and the models in which it was learned. We ask whether a predictive visual latent space induced by large-scale predictive pretraining can instead provide a sufficient foundation for effective WAM learning without inheriting a complete pretrained visual generative model. To answer this question, we introduce V-JEPA Policy, a simple framework that builds a WAM on the latent space of a frozen V-JEPA 2.1 encoder. An instruction-conditioned future-latent predictor and a flow-matching action expert are jointly learned from scratch in a single downstream stage, with the predictor's future-informed context key–value states conditioning action generation. With 0.9B total parameters, of which 0.6B are trainable, V-JEPA Policy achieves competitive performance with representative WAM and vision-language-action baselines across LIBERO, LIBERO-Plus, and RoboCasa-GR1. Comparing visual foundations under the same downstream framework and training budget identifies V-JEPA latents as more effective than the discriminative, reconstructive, and video-understanding-oriented alternatives, particularly under distribution shifts. Beyond task-specific learning, the same latent space supports acquiring transferable future-modeling knowledge from diverse in-the-wild instruction-annotated videos. Pretraining the predictor on DROID video–instruction pairs without action labels and adapting it into a WAM within our framework yields substantial gains in downstream control and out-of-distribution generalization. Together, these findings establish predictive visual latents as a foundation for learning effective WAMs, supporting both direct learning from task-specific demonstrations and the transfer of future-modeling knowledge acquired through predictor pretraining on broader in-the-wild videos.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World-action-models, Language-conditioned robotic manipulation, JEPA, Single-stage method

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WyyAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes V-JEPA Policy, a world-action model built on the latent space of a frozen V-JEPA 2.1 visual encoder, without inheriting a pretrained visual generator. Its central contribution is a framework that jointly trains an instruction-conditioned future-latent predictor and a flow-matching action expert from scratch. The predictor processes observed context and learnable future queries through bidirectional attention, while its layer-wise context key–value states condition action generation. The paper also investigates whether predictor-only pretraining on DROID videos, using language and proprioceptive inputs but no action labels, provides transferable future-modeling knowledge.

Validation covers LIBERO, LIBERO-Plus, RoboCasa-GR1, and two real-world bimanual tasks. Controlled encoder comparisons favor predictive visual representations over discriminative, reconstructive, and video-understanding alternatives, particularly under distribution shifts. Ablations support the contributions of both future queries and explicit future-prediction supervision. Predictor pretraining improves downstream performance, notably raising LIBERO-Plus success from 79.25% to 91.50%, exceeding the gains from extended downstream training. Deployment profiling also shows lower memory usage and latency than FastWAM.

Critical strengths
  1. The paper shows that a frozen predictive visual encoder can support effective world-action modeling without a pretrained visual generator. The model achieves competitive results with larger WAMs across simulation and real-world tasks. Encoder comparisons and ablations support the approach, making it a useful alternative to adapting large generative backbones.

  2. Predictor pretraining without action labels improves performance across all evaluated benchmarks and real-world tasks, particularly on LIBERO-Plus (79.25% to 91.50%). These gains exceed those from longer downstream training, suggesting that the predictor learns transferable knowledge beyond the task demonstrations. This provides a promising way to use additional video data for robot learning.

Critical weaknesses
  1. The pretraining results show positive transfer, but do not isolate the transfer of future-modeling knowledge. DROID pretraining also exposes the predictor to additional scenes, instructions, and robot states, while the extended-training baseline only reuses downstream demonstrations. A comparison using the same DROID data and a non-future-predictive pretraining objective would help establish whether the gains specifically come from learning to predict future states. Without this control, the results support the usefulness of predictor pretraining more clearly than the claimed source of its transfer benefits.
Questions for authors
  1. Could the authors clarify the relative strengths of V-JEPA Policy and WAMs? Both approaches inherit useful representations from video pretraining, so it would be helpful to explain what is gained or lost by using a frozen predictive latent space instead of a pretrained generative backbone. Are there domains or tasks where retaining the visual generator offers a clear advantage, or may even be necessary?

  2. V-JEPA Policy learns language conditioning during predictor and policy training, whereas text-conditioned diffusion backbones already have pretrained text–visual alignment. Could this difference affect generalization to unseen instructions or novel task compositions? It would be helpful to discuss or evaluate this beyond the language perturbations in LIBERO-Plus.

6 comments

Dear Reviewer WyyA,

We sincerely appreciate your precious time and constructive comments. In the following, we would like to answer your concerns separately.

[W1] Isolating the transfer of future-modeling knowledge during pretraining.

Response: Thank you for highlighting this distinction. We agree that DROID pretraining may expose the predictor to additional tabletop scenes, language instructions, and especially the same Franka Panda's robot states. To verify how much the downstream gains stem exactly from learning to predict future states, we conducted pre-training with a non-future-predictive objective on the same dataset. Here we adopt the spatial masking denoising objective same as what I-JEPA[1] does. We also conducted a complementary experiment on human videos to examine transfer beyond robot-specific pretraining data, since the embodiment involved in DROID is same as that in LIBERO.

All pretrained variants use 100k optimizer updates, followed by the same downstream LIBERO training and LIBERO/LIBERO-Plus evaluation recipe.

  1. Setup 1: Current-context spatial masking denoising pretraining on DROID. Following the spatial masking denoising principle of I-JEPA[1], we use the exact same DROID video clips, dual camera views, T5 instructions ℓ\ell, and robot states qt\mathbf{q}_t, but predict masked patches within the current context tubelet (Δt=0\Delta t = 0) instead of future tubelets (0<δ1<⋯<δK0 < \delta_1 < \dots < \delta_K). We randomly mask spatial patches across both current views, feed the visible tokens and learnable mask queries (preserving 3D RoPE) into the predictor, and optimize the identical ℓ1\ell_1 regression loss (Eq. 7) against frozen V-JEPA 2.1 targets. The prediction task therefore concerns current spatial content rather than future tubelets at 0<δ1<⋯<δK0 < \delta_1 < \dots < \delta_K.
  2. Setup 2: Future prediction pre-training on human egocentric video dataset EgoVerse. We pretrain the predictor with Lfuture\mathcal{L}_{\text{future}} (Eq. 7) on human egocentric videos paired with text instructions. The state branch is omitted entirely, and pretraining uses neither robot proprioception nor action labels.
Pretraining ConfigurationPretraining DataEmbodimentPretraining conditioningMasked Prediction TargetLIBERO (%)LIBERO-Plus (%)Gain on Plus vs. Scratch
From ScratchNone---97.2579.25-
Setup 1: Spatial Masking Denoising on DROIDDROIDFranka PandaInstructions + robot statesCurrent Spatial Patches98.1584.81+5.56 pp
Setup 2: Future Prediction on EgoVerseEgoVerseHumanInstruction-onlyFuture Tubelets98.1582.92+3.67 pp
Ours: Future Prediction on DROIDDROIDFranka PandaInstructions + robot statesFuture Tubelets98.7092.22+12.97 pp

We conclude three observations from the results.

  1. Non-future-predictive pretraining also provides useful transfer. DROID spatial prediction improves LIBERO-Plus success from 79.25% to 84.81% (+5.56 pp). This establishes that useful transfer can arise even when the predictor’s pretraining task does not forecast future observations.

  2. Future prediction provides a substantial additional benefit on the same data. Replacing current-context spatial prediction with future prediction raises LIBERO-Plus success from 84.81% to 92.22%, an improvement of approximately 7.4 pp. With the DROID dataset, conditioning information, predictor architecture, and pretraining budget matched, this comparison supports the additional value of learning future prediction for downstream control.

  3. Such transfer also extends to human egocentric video–language data even though there is a significant embodiment gap between human and robotic arm. Instruction-only EgoVerse pretraining reaches 82.92% (+3.67 pp over scratch). Its lower score than DROID spatial pretraining is consistent with differences in data alignment: DROID shares LIBERO’s Franka Panda embodiment and provides external and wrist-camera views together with proprioception, whereas EgoVerse provides a single human egocentric view without robot states. The within-DROID comparison tests the benefit of the pretraining objective, while the EgoVerse experiment shows that our predictive pretraining framework can also benefit from human videos across these embodiment and observation differences.

These controls are reported in Section 5.4 and Appendices F.2–F.3 (Tables 15).

[Q1] Relative strengths of V-JEPA Policy vs. generator-based WAMs, and when retaining a visual generator is advantageous.

Response: Thank you for this question. Both approaches benefit from large-scale visual pretraining. Our central question is whether the resulting predictive representations can provide a sufficient foundation for effective WAM learning without inheriting a complete pretrained visual generator. In our opinions, there could exist the following benefits and trade-offs.

  1. Gains from a frozen predictive latent space.

Decoupling the representations that entail world-modeling knowledge from heavy generative backbones. Our 0.9B policy combines a frozen V-JEPA 2.1 encoder with a single-pass predictor and a lightweight action expert, with only 0.6B trainable parameters. The base policy lies on the empirical parameter–performance Pareto frontier in all five evaluated settings among the compared methods (Fig. 6, Appendix H). Under the matched profiling protocol in Table 8, it requires 4.66 GiB peak VRAM, compared with 12.74 GiB for FastWAM, a 63.4% reduction.

Predictive features for robust control. V-JEPA learns through latent feature self-prediction, emphasizing spatial and temporal structure without requiring pixel reconstruction. This provides an effective foundation for control under appearance changes. When used as frozen visual foundations under the same downstream recipe, V-JEPA 2.1 achieves 79.25% on LIBERO-Plus, compared with 46.43% for WAN2.2 VAE, with particularly large advantages under texture (80.02% vs. 10.13%) and lighting (97.20% vs. 32.05%) perturbations (Table 5).

  1. Trade-offs and advantages retained by visual generators.

Pretrained vision–language grounding. Text-conditioned generative backbones inherit cross-modal associations learned from large-scale paired data. V-JEPA is pretrained on vision alone. Its feature are instead aligned with pretrained T5 embeddings during predictor and policy training, using substantially less paired visual–language data. Our policy may therefore underperform these backbones on novel instructions or complex paraphrases, particularly when trained only on downstream demonstrations. As shown in Q2, predictor pretraining on DROID video–instruction pairs substantially improves language robustness and also improves novel-task generalization.

Explicit visual synthesis. Retaining a generator is useful when predicted RGB frames are needed for the inspection of anticipated outcomes, VLM-based scoring of imagined rollouts, or visual subgoal generation. V-JEPA Policy predicts future latents and actions, so supporting these applications would require an additional decoder or generative component.

We discuss these trade-offs in Appendix J.

[Q2] Impact of language conditioning learned during predictor/policy training on unseen instructions and novel task compositions.

Response: This is an excellent point that directly connects to [Q1] above. As V-JEPA Policy uses pretrained T5 embeddings, but learns their association with visual latents during predictor and policy training rather than inheriting a text-aligned diffusion backbone, we observe a clear distinction between from-scratch training and action-free predictor pretraining.

MethodVisual FoundationVision–Language Alignment SourceLIBERO-Plus: Language (%)LIBERO-Pro: Novel Task (%)
FastWAM (6.0B)WAN Video DiffusionWeb-scale Text–Video + Downstream68.914.90$^\dagger$
ImageWAM (4.5B)Image-Editing BackboneWeb-scale Text–Image + Downstream91.419.05$^\dagger$
V-JEPA Policy (From Scratch)Frozen V-JEPA 2.1Downstream Demos Only64.917.95
V-JEPA Policy (Pretrained Predictor)Frozen V-JEPA 2.1DROID Video–Text + Downstream94.622.25 (+4.30 pp)

†^\dagger denotes that task scores are borrowed from the corresponding entries in Table 12 in WAVE-0[3]

  1. Language robustness improves through video–instruction pretraining. When trained from scratch on 1,693 LIBERO demonstrations, V-JEPA Policy achieves 64.9% on Language perturbations, trailing text-pretrained ImageWAM (91.4%, Table 1). However, pretraining the predictor on DROID video–instruction pairs yields our largest single-axis gain (+29.73 pp, Fig. 2b), reaching 94.6% and surpassing ImageWAM (91.4%) and π0.5\pi_{0.5} (88.5%).
  1. The improvement extends to novel task goals. We additionally evaluated our policies on the LIBERO-Pro[2] Task axis using 2,000 rollouts. In the official implementation, Task perturbations change the instruction together with the relevant objects and symbolic success conditions. They therefore require completing new task goals, such as opening a different drawer, rather than following an alternative description of the same goal. Predictor pretraining improves novel-task success from 17.95% to 22.25% (+4.30 pp), exceeding the corresponding reported results of FastWAM and ImageWAM. These results show that the benefits extend beyond linguistic perturbations to executing changed task goals.

That said, novel-task generalization remains challenging, but the observed gains show that a frozen predictive visual foundation can support improvements in both capabilities through predictor pretraining.

These evaluations are included in Section 5.2 and Appendix A.1 (Table 2).

References

[1] Assran, Mahmoud, et al. "Self-supervised learning from images with a joint-embedding predictive architecture." 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023.

[2] Zhou, Xueyang, et al. "LIBERO-Pro: Towards robust and fair evaluation of vision-language-action models beyond memorization." arXiv preprint arXiv:2510.03827 (2025).

[3] Wang, Haomin, et al. "WAVE-0: Bridging VLA Policies and WAMs with Lightweight Latent Future Prediction." arXiv preprint arXiv:2610.18568 (2026).

Reviewer_uYkaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes V-JEPA Policy, a world-action model built on frozen V-JEPA 2.1 visual representations. An instruction-conditioned future-latent predictor and a flow-matching action expert are jointly trained from scratch, with future-informed representations conditioning action generation. The paper further explores action-free predictor pretraining on DROID and demonstrates transfer to downstream robot control.

Critical strengths
  • The paper shows that training the predictor and action expert from scratch on frozen V-JEPA representations can achieve competitive performance without relying on a large pretrained visual generator.

  • The experiments are comprehensive and carefully controlled, covering standard simulation benchmarks, OOD generalization, and real-world manipulation.

  • Pretraining the future predictor without action labels provides substantial improvements, particularly under distribution shifts, suggesting that useful dynamics knowledge can be transferred from video future modeling to robot control.

Critical weaknesses
  • Limited methodological novelty. The overall framework overlaps conceptually with recent V-JEPA-based latent WAM approaches and existing MoT/MMDiT-style action-generation architectures. The proposed future-informed KV coupling appears useful, but is more of an architectural variation than a fundamentally new paradigm.

  • The necessity of joint WAM training is unclear. Table 9 establishes that explicit future-prediction supervision is important, but does not show that jointly optimizing future prediction and action generation is necessary. A key missing baseline is to independently pretrain a generic predictor on frozen V-JEPA features, freeze it, and train only an action expert conditioned on its intermediate features/KV states. If this simpler two-stage approach performs similarly, the gains may primarily come from dynamics-aware representations learned through video prediction.

Questions for authors
  • What is the performance if the DROID-pretrained predictor is frozen during downstream training and only the action expert is optimized?

  • The current predictor pretraining uses DROID trajectories, including robot videos, instructions, and proprioceptive states, although no action labels are used. Could similar gains be obtained from more general video-language data without robot proprioception?

7 comments

Dear Reviewer uYka,

We sincerely appreciate your precious time and constructive comments. In the following, we would like to answer your concerns separately.

[W1] Methodological novelty and design philosophy.

Response: Thank you for raising this point. We would like to clarify that our primary goal of this work is to answer a fundamental question in world-action modeling: whether a predictive visual latent space learned at scale can provide a sufficient foundation for effective WAM learning without inheriting a complete pretrained visual generative model. We further investigate whether the same frozen latent space supports efficient acquisition and transfer of general future-modeling knowledge from in-the-wild videos. We intentionally use established JEPA-style predictor and Mixture-of-Transformers (MoT) components to investigate these questions within a simple framework.

  1. A simple architecture for prediction and control. Since V-JEPA 2.1 is pretrained via masked latent prediction using a ViT-based predictor with 3D RoPE, directly adapting the native V-JEPA predictor architecture and equipping it with instruction and state conditioning provides a natural interface to forecast future latents. We further connect it to a flow-matching action expert through a well-established MoT architecture. Interactions between context tokens and future queries produce future-informed context K/V states, which are cached and reused throughout action sampling. This connects predictive modeling to control while requiring only one predictor pass per decision step.

  2. A clean design space enabling rigorous controlled study of the visual foundation. This is one of the central contributions of our work. Rigorous visual foundation comparison demands a standardized model architecture and training pipeline. Therefore, avoiding intricate multi-stage alignment pipelines allows us to cleanly isolate the contribution of the visual foundation itself across discriminative, reconstructive, video-understanding, and predictive encoders.

  3. A simple training recipe with competitive performance. Regarding recent closely related methods, VLA-JEPA[1] learns latent actions through future V-JEPA latent prediction within a pretrained VLM framework, combining human-video and action-supervised robot pretraining before downstream adaptation. JEPA-WAM[2] emphasizes a spatially structured joint current–future target and a shared Qwen-based predictor, and requires extra vision–language alignment training before robot-policy learning. StageWAM[3] first trains Stage-JEPA to predict the next task stage, then freezes it to guide a Motus-based WAM that retains a video-generation expert. Different from these multi-stage approaches, our default recipe instead jointly learns the predictor and action expert from scratch in a single downstream stage. With 0.9B total parameters (0.6B trainable), it achieves 97.25% on LIBERO and 79.25% on LIBERO-Plus, exceeding JEPA-WAM’s scores and remaining competitive with VLA-JEPA in the comparison reported in Table 1. With an additional predictor-only pretraining phase over 331 hours of DROID videos without action labels, our V-JEPA Policy achieves 98.70% on LIBERO and 91.50% on LIBERO-Plus, yielding substantially higher performance than both methods.

In the end, we would like to highlight again that our contribution combines a direct and simple WAM learning recipe on frozen predictive visual latents with controlled findings on the role of the visual foundation and the transfer of predictive knowledge.

[W2, Q1] Freezing the DROID-pretrained predictor versus joint adaptation

Response: Thanks for the constructive suggestion. We directly evaluated the proposed strategy of freezing the pretrained predictor and training only the action expert. This baseline achieves 64.18% on LIBERO-Plus, compared with 92.22% for full joint downstream optimization.

To examine the roles of predictor adaptation and the two training objectives, we compare four downstream strategies. All variants initialize from the same DROID-pretrained predictor checkpoint, train a randomly initialized action expert on LIBERO, and keep the visual encoder frozen. We reran the full joint-optimization baseline using the same training seed and hardware as the three ablated variants. Variant A below reports this new run.

The variants differ in how the predictor is updated:

  1. A - Full joint optimization (Ours): The predictor receives both future-prediction supervision Lfuture\mathcal{L}_{\text{future}} and action flow matching loss Laction\mathcal{L}_{\text{action}} gradients through its context key–value interface.

  2. B - Freezing predictor: Only the action expert is optimized via Laction\mathcal{L}_{\text{action}}. The predictor receives no parameter updates.

  1. C - Blocking action gradients to predictor: Both Laction\mathcal{L}_{\text{action}} and Lfuture\mathcal{L}_{\text{future}} remain active, but we apply stop-gradient sg(Cϕ)\text{sg}(\mathcal{C}_{\phi}) on the context K/V interface so that Laction\mathcal{L}_{\text{action}} gradients cannot flow into the predictor.

  2. D - Removing Lfuture\mathcal{L}_{\text{future}} after pretraining: We set λfuture=0\lambda_{\text{future}} = 0 during downstream training so the pretrained predictor is updated solely by Laction\mathcal{L}_{\text{action}} gradients through Cϕ\mathcal{C}_{\phi}.

Downstream Adaptation StrategyPredictor TrainableAction Grad →\rightarrow PredictorEnabling Future LossLIBERO (%)LIBERO-Plus (%)
B: Freezing predictor×\times×\times×\times91.5564.18
C: Blocking action gradients✓\checkmark×\times✓\checkmark98.6087.97
D: Removing future loss✓\checkmark✓\checkmark×\times98.2588.94
A: Full joint optimization (Ours)✓\checkmark✓\checkmark✓\checkmark98.7092.22

As shown in the table above, full joint optimization outperforms other three variants on LIBERO-Plus by +28.04, +4.25, and +3.28 percentage points respectively, while Variants A, C, and D perform similarly on in-distribution LIBERO (98.25%–98.70%). This reveals three key findings on how predictor pretraining transfers to downstream control:

  1. Adapting the pretrained predictor matters substantially in this setting. During action-free pretraining, the predictor is only supervised at its final output layer to forecast future tubelets Z^t+\hat{\mathbf{Z}}_{t}^{+} on real-world camera views in DROID dataset. Its intermediate layer-wise context K/V states Cϕ\mathcal{C}_{\phi} have never been trained to serve as a readout interface for an action expert. Freezing the predictor leaves both the visual domain gap (real-world DROID vs. simulated LIBERO) and the intermediate control interface unadapted, degrading both in-distribution and out-of-distribution performance.

  2. Pretraining provides a strong initialization, but future supervision remains useful after pretraining. Under identical action-only downstream supervision (λfuture=0\lambda_{\text{future}}=0), initializing from the DROID-pretrained predictor (Variant D, 88.94%) outperforms random initialization (Table 13 row 2, 68.81%) by +20.13 pp on LIBERO-Plus, confirming that pretraining embeds rich future-modeling dynamics priors into the predictor weights. However, removing future supervision in Variant D gives 88.94%, approximately 3.3 pp below full joint adaptation, supporting that predictive supervision continues to provide an additional benefit even when the predictor has already undergone pretraining on much larger in-the-wild dataset.

  3. Action gradients contribute beyond domain-adapted future prediction. Variant C retains future supervision and allows the predictor to adapt to downstream observations, but blocks the action objective from changing its representations. Enabling this gradient path in Variant A further improves LIBERO-Plus success from 87.97% to 92.22%. This comparison shows that through this gradient path, action supervision steers the predictor’s context interface Cϕ\mathcal{C}_\phi so that it not only supports future modeling but also better serve downstream action generation.

These comparisons are included in Section 5.4 and Appendix F.1 (Table 14).

[Q2] Predictor pretraining on general video–language data without robot proprioception.

Response: Thanks for your insightful suggestion. We tested this directly by pretraining the future-latent predictor on human egocentric manipulation video dataset EgoVerse[4], without robot proprioception or action labels.

We pretrained only the predictor for 100k optimizer updates using the future-prediction objective in Eq. 7, while keeping the visual encoder frozen and omitting the state branch entirely. Visual observations serve as predictor inputs, and T5 instruction tokens provide the only cross-attention conditioning. A newly initialized state encoder is introduced during downstream LIBERO training alongside the action expert. All variants follow the same downstream training and evaluation recipe.

Predictor initializationPretraining videosPretraining conditioningLIBERO (%)LIBERO-Plus (%)Δ LIBERO-Plus (pp)
From scratch——97.2579.25—
DROID (100k)Robot manipulationInstructions + robot proprioception98.7092.22+12.97
EgoVerse (100k)Human egocentric manipulationInstructions only98.1582.92+3.67

These results reveal two complementary findings:

  1. Human video–language pretraining provides useful transfer without robot states. Instruction-only EgoVerse pretraining improves LIBERO-Plus success from 79.25% to 82.92% (+3.67 pp). The gains are particularly pronounced under language perturbations (64.87% → 82.17%, +17.30 pp), background texture changes (80.02% → 90.71%, +10.69 pp), and object layout changes (79.08% → 85.51%, +6.43 pp). These improvements show that our predictive pretraining framework can benefit from human videos without robot-specific state supervision.

  2. The transfer pattern helps explain the remaining gap to DROID. The gains are not uniform: success under robot initial-state perturbations decreases from 80.65% to 75.29%, and success under camera-viewpoint perturbations decreases from 70.04% to 67.98%. DROID pretraining achieves 93.94% and 79.24% on these two axes, respectively. We attribute the smaller transfer gains from EgoVerse in part to two differences between the pretraining settings. EgoVerse only provides a single human egocentric view, whereas DROID provides robot manipulation observations with external and wrist-camera views, together with proprioception, closely matching LIBERO’s observation configuration. Moreover, transferring knowledge from human manipulation in EgoVerse to LIBERO’s Franka Panda robot involves a substantial human–robot embodiment gap, whereas DROID and LIBERO share the same robot embodiment. DROID therefore aligns more closely with LIBERO. The stronger DROID results on robot-state and camera perturbations are consistent with this alignment.

These results and protocols are included in Section 5.4 and Appendices F.2–F.3 (Tables 15 and 16).

References

[1] Sun, Jingwen, et al. "Vla-jepa: Enhancing vision-language-action model with latent world model." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.

[2] Lin, Yihan, et al. "JEPA-WAM: Learning vision-language-action policies with joint-embedding world modeling." arXiv preprint arXiv:2608.09381 (2026).

[3] Liu, Xiao, et al. "StageWAM: Joint-embedding stage prediction for world-action models in robot manipulation." arXiv preprint arXiv:2608.10780 (2026).

[4] Punamiya, Ryan, et al. "Egoverse: An egocentric human dataset for robot learning from around the world." arXiv preprint arXiv:2604.07607 (2026).

Reviewer_K7FeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces V-JEPA Policy, coupling a future-latent predictor with a flow-matching action expert on frozen V-JEPA features. Both modules can be jointly trained from scratch without inheriting a pretrained visual generator. Evaluation spans three simulation benchmarks and two real-world bimanual tasks. Action-free predictor pretraining on DROID further improves downstream performance, notably raising LIBERO-Plus success from 79.25% to 91.50%.

Critical strengths
  1. A simple and practically useful alternative to generator-based WAMs. The default recipe jointly trains a future predictor and an action expert from scratch on frozen V-JEPA features, demonstrating effective control without inheriting a complete pretrained visual generator. A single predictor pass supplies cached, layer-wise context key–value states for action generation, avoiding iterative visual generation at deployment.

  2. Broad empirical validation and substantial transfer benefits. Evaluation spans LIBERO, LIBERO-Plus, RoboCasa-GR1, and physical bimanual manipulation, covering controlled distribution shifts and different robot embodiments. Under matched downstream data and training budgets, predictor pretraining on DROID raises LIBERO-Plus success from 79.25% to 91.50% and average success on the two physical tasks from 45% to 80%. These improvements demonstrate that predictor pretraining without action labels can benefit both robustness in simulation and physical control, making the transfer result practically consequential.

Critical weaknesses
  1. Analysis of the DROID-pretrained setting remains limited. The paper motivates joint optimization and demonstrates the value of future-prediction supervision when training from scratch, but does not examine whether these findings and the encoder-scaling conclusions extend to the pretrained setting. In particular, it lacks comparisons with freezing the predictor, blocking action-loss gradients to the predictor, or removing the future-prediction loss after pretraining, leaving the roles of different downstream adaptation strategies unclear.

  2. The real-world performance gap to PRTS is not analyzed. PRTS achieves the highest success rates on both physical tasks—100% and 95%, compared with 75% and 85% for the proposed pretrained model. The discussion does not examine the task stages or failure modes underlying this gap, leaving the approach’s practical limitations insufficiently characterized.

Questions for authors
  1. After DROID predictor pretraining, what evidence supports full joint optimization over freezing the predictor, blocking action-loss gradients to it, or removing continued future-prediction supervision?

  2. Which task stages account for the real-world gap to PRTS? Can existing rollout observations or known training differences help explain its stronger performance?

  3. How much DROID data is needed to obtain the reported transfer gains? Is there evidence from data subsets or intermediate pretraining checkpoints that increasing pretraining exposure continues to improve downstream generalization, or that the gains are approaching saturation?

  4. How stable is the 0.88-point LIBERO-Plus difference between V-JEPA 2.1 ViT-L and ViT-G across independent training runs? Does the encoder ranking persist after DROID predictor pretraining under matched downstream settings?

7 comments

Dear Reviewer K7Fe,

We deeply appreciate your insightful comments and constructive feedback in reviewing our manuscript. In the following, we would like to answer your concerns separately.

[W1, Q1] Downstream adaptation strategies after DROID predictor pretraining.

Response: Thanks for the insightful suggestion. First, we would like to clarify our design philosophy regarding joint optimization: following established WAM formulations such as Cosmos Policy[1], DreamZero[2], FastWAM[3], ImageWAM[4], jointly optimizing future prediction and action generation is the standard paradigm for coupling world modeling with control. We intentionally adopt this established joint formulation to cleanly investigate our central research question: whether a frozen predictive visual latent space can serve both as a sufficient foundation for direct single-stage WAM learning and as a reliable substrate for efficiently acquiring and transferring general future-modeling knowledge from in-the-wild data.

Secondly, in Section 5.3 and Table 13, we showed the necessity of explicit future-latent supervision when learning from scratch. To examine how these findings extend to the DROID-pretrained setting, we additionally evaluate three downstream adaptation strategies following the reviewer's suggestion. All four variants inherit the predictor pretrained on DROID video-instruction pairs for initialization, and the action expert is learned during downstream LIBERO training, and the visual encoder remains frozen. Note that we reran the full joint-optimization baseline using the same training seed and hardware as the three ablated variants to ensure a strictly controlled comparison. Variant A below reports this new downstream run.

The variants differ in how the predictor is updated:

  1. A - Full joint optimization (Ours): The predictor receives both future-prediction supervision Lfuture\mathcal{L}_{\mathrm{future}} and action flow matching loss Laction\mathcal{L}_{\mathrm{action}} gradients through its context key–value interface.

  2. B - Freezing predictor: Only the action expert is optimized via Laction\mathcal{L}_{\mathrm{action}}. The predictor receives no parameter updates.

  3. C - Blocking action gradients to predictor: Both Laction\mathcal{L}_{\mathrm{action}} and Lfuture\mathcal{L}_{\mathrm{future}} remain active, but we apply stop-gradient sg⁡(Cϕ)\operatorname{sg}(\mathcal{C}_{\phi}) on the context KV interface so that Laction\mathcal{L}_{\mathrm{action}} gradients cannot flow into the predictor.

  4. D - Removing Lfuture\mathcal{L}_{\mathrm{future}} after pretraining: We set λfuture=0\lambda_{\mathrm{future}}=0 during downstream training so the pretrained predictor is updated solely by Laction\mathcal{L}_{\mathrm{action}} gradients through Cϕ\mathcal{C}_{\phi}.

We report results as follows:

Downstream Adaptation StrategyPredictor TrainableAction Grad →\rightarrow PredictorEnabling Future LossLIBERO (%)LIBERO-Plus (%)
B: Freezing predictor×\times×\times×\times91.5564.18
C: Blocking action gradients✓\checkmark×\times✓\checkmark98.6087.97
D: Removing future loss✓\checkmark✓\checkmark×\times98.2588.94
A: Full joint optimization (Ours)✓\checkmark✓\checkmark✓\checkmark98.7092.22

As shown in the table above, full joint optimization outperforms other three variants on LIBERO-Plus by +28.04, +4.25, and +3.28 percentage points respectively, while Variants A, C, and D perform similarly on in-distribution LIBERO (98.25%–98.70%). This reveals three key findings on how predictor pretraining transfers to downstream control:

  1. Adapting the pretrained predictor matters substantially in this setting. During action-free pretraining, the predictor is only supervised at its final output layer to forecast future tubelets Z^t+\hat{\mathbf{Z}}_{t}^{+} on real-world camera views in DROID dataset. Its intermediate layer-wise context K/V states Cϕ\mathcal{C}_{\phi} have never been trained to serve as a readout interface for an action expert. Freezing the predictor leaves both the visual domain gap (real-world DROID vs. simulated LIBERO) and the intermediate control interface unadapted, degrading both in-distribution and out-of-distribution performance.
  1. Pretraining provides a strong initialization, but future supervision remains useful after pretraining. Under identical action-only downstream supervision (λfuture=0\lambda_{\text{future}}=0), initializing from the DROID-pretrained predictor (Variant D, 88.94%) outperforms random initialization (Table 13 row 2, 68.81%) by +20.13 pp on LIBERO-Plus, confirming that pretraining embeds rich future-modeling dynamics priors into the predictor weights. However, removing future supervision in Variant D gives 88.94%, approximately 3.3 pp below full joint adaptation, supporting that predictive supervision continues to provide an additional benefit even when the predictor has already undergone pretraining on much larger in-the-wild dataset.

  2. Action gradients contribute beyond domain-adapted future prediction. Variant C retains future supervision and allows the predictor to adapt to downstream observations, but blocks the action objective from changing its representations. Enabling this gradient path in Variant A further improves LIBERO-Plus success from 87.97% to 92.22%. This comparison shows that through this gradient path, action supervision steers the predictor’s context interface Cϕ\mathcal{C}_\phi so that it not only supports future modeling but also better serves downstream action generation.

We have added these results and discussions to Section 5.4 and Appendix F.1 (Table 14).

[W1, Q4] Whether encoder ranking conclusion persists in predictor-only pretraining setting and how stable is the performance difference on LIBERO-Plus between V-JEPA 2.1 ViT-L and ViT-G.

Response: To examine how our Section 5.3 conclusions on representation quality (encoder generation) and encoder scale extend to the predictor-only pretraining setting, we performed 100k-optimizer-step predictor pretraining on DROID dataset followed by identical 10-epoch downstream LIBERO adaptation across V-JEPA 2 ViT-L, V-JEPA 2.1 ViT-L, and V-JEPA 2.1 ViT-G:

Visual foundationPretraining Predictor on DROIDScaleLIBERO (%)LIBERO-Plus (%)
V-JEPA 2✓\checkmarkViT-L98.5086.18
V-JEPA 2.1✓\checkmarkViT-L98.7092.22
V-JEPA 2.1✓\checkmarkViT-G99.1590.87

Encoder scale trade-off (ViT-G > ViT-L in-distribution, but ViT-L > ViT-G under OOD shifts) persists in pretraining setting. In our from-scratch results (Table 5), we noted that scaling V-JEPA 2.1 from ViT-L (304M) to ViT-G (1.01B) slightly improves in-distribution LIBERO (97.25% →\rightarrow 97.70%, +0.45%) while yielding slightly lower OOD performance on LIBERO-Plus (79.25% →\rightarrow 78.37%, -0.88%). Remarkably, this exact pattern reproduces after DROID 100k pretraining: V-JEPA 2.1 ViT-G achieves a higher in-distribution LIBERO score than ViT-L (99.15% vs. 98.70%), whereas ViT-L achieves a higher OOD score on LIBERO-Plus (92.22% vs. 90.87%).

Encoder generation ranking (V-JEPA 2.1 > V-JEPA 2 on both benchmarks) also persists in pretraining setting. At the matched ViT-L scale, V-JEPA 2.1 outperforms V-JEPA 2 on both in-distribution LIBERO (98.70% vs. 98.50%) and OOD LIBERO-Plus (92.22% vs. 86.18%), consistent with the from-scratch ordering (79.25% vs. 74.15%) in Table 5.

The performance difference on LIBERO-Plus is stable across multiple independent runs. For each encoder scale, V-JEPA 2.1 ViT-L and ViT-G, we conducted three independent runs of V-JEPA Policy training on LIBERO from scratch. We evaluated all resulting policies on LIBERO and LIBERO-Plus using the same evaluation seed. As shown below, ViT-L continues to achieve a higher mean success rate on LIBERO-Plus than ViT-G, consistent with the ordering reported in Table 5.

Visual foundationLIBERO-Plus (%)
V-JEPA Policy w/ V-JEPA 2.1 ViT-L (from scratch)79.05 ± 0.50
V-JEPA Policy w/ V-JEPA 2.1 ViT-G (from scratch)77.79 ± 0.56

Results are reported as mean ± sample standard deviation across three independent training runs.

These results are reported in Appendices C.2–C.3.

[W2, Q2] Real-world performance gap to PRTS and failure analysis.

Response: Thanks for the constructive comment. To examine the concrete behavioral difference, we additionally evaluate PRTS[5], V-JEPA Policy (Pretrained Predictor) and FastWAM on Table Cleanup task using the same starting debris layout, and record their rollout for direct behavior decision comparison. We provide the accompanying filmstrip illustration in Appendix B.3 (Fig. 5). All three policies successfully placed the bowl in the drying rack, accomplishing the initial bowl racking sub-task. However, their behaviors diverged during the subsequent paper removal phase:

  • PRTS used its left arm to move the far-left piece onto the tray, clearing all visible debris.
  • FastWAM stalled while attempting to grasp the rightmost paper piece on the tray, leaving it uncleared at timeout.
  • V-JEPA Policy (Pretrained) successfully discarded one paper piece, but subsequently reached across with its right gripper toward the far-left piece (outside the tray) instead of switching to a left-arm transfer. It attempts repeated retries, leaving the far-left piece uncleared at timeout.

Training differences help interpret the observation. First of all, PRTS explicitly optimizes a language-conditioned contrastive objective that encodes goal reachability, which inject the temporal-progress awareness over a longer horizon that helps understanding sequential subgoals into the representation space of the generalist policy. In contrast, V-JEPA Policy's future-latent prediction trains the predictor to forecast spatiotemporal visual transitions over a fixed limited horizon, which does not take goal reachability into account and thus does not induce structured representation like PRTS. Secondly, PRTS perform action-supervised pre-training with a 5B architecture on about 4,000 hours of action-labeled data together with over 10 million visual questioning-answering pairs. In contrast, V-JEPA Policy is a compact 0.9B model whose predictor is pretrained on only about 300 hours of action-free single-arm DROID videos. This substantial difference in action-supervised data scale and model capacity contributes to PRTS's higher precision in multi-stage bimanual coordination.

[Q3] DROID pretraining exposure and intermediate pretrained checkpoints.

Response: Thanks for the question.

  1. We curated the official DROID dataset by excluding failed demonstrations and trimming idle-action segments, yielding 74,344 episodes with a total duration of 331 hours.

  2. To examine whether downstream generalization continues to improve with pretraining exposure or approaches saturation, we evaluated intermediate checkpoints from our DROID predictor pretraining run after 20k, 50k, 70k, and 100k optimizer updates. Each checkpoint was adapted to LIBERO using the same 10-epoch downstream post-training recipe and evaluated on both LIBERO and LIBERO-Plus.

DROID Pretraining Optimizer StepsFraction of Pre-training BudgetLIBERO Avg. (%)LIBERO-Plus Avg. (%)Gain on LIBERO-Plus vs. Scratch (pp)
0 (From scratch)0%97.2579.25—
20k steps20%99.0092.11+12.86
50k steps50%99.0592.33+13.08
70k steps70%99.2592.06+12.81
100k steps (Default)100%98.7092.22+12.97

These results highlight two properties of predictor pretraining on frozen V-JEPA latents:

  1. Rapid acquisition of future-modeling prior. After just 20k updates, corresponding to 20% of the default pretraining budget, downstream success increases from 97.25% to 99.00% on LIBERO and from 79.25% to 92.11% on LIBERO-Plus (+12.86 pp). The latter is already within 0.11 pp of the default 100k checkpoint, showing that most of the transfer benefit can be obtained with one-fifth of the pretraining updates. This early improvement is consistent with our design: since the frozen V-JEPA 2.1 encoder already provides a predictive spatiotemporal latent space, the lightweight 0.5B predictor can rapidly acquire instruction-conditioned future transition priors early in pretraining.

  2. Downstream transfer gain approaches an early plateau. Between 20k and 100k updates, LIBERO success ranges from 98.70% to 99.25%, peaking at 70k updates. LIBERO-Plus success remains within 92.06%–92.33%, a range of only 0.27 pp, with the highest score at 50k updates. Thus, extending pretraining beyond 20k updates yields small fluctuations rather than a consistent improvement in OOD performance.

The data curation and checkpoint results are documented in Appendices E and F.4 (Table 17).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (Part 1)

Dear PCs, SACs, ACs and Reviewers,

We sincerely thank the reviewers for their time and thoughtful feedback. As the author–reviewer discussion draws to a close, we would like to summarize the additional experiments, clarifications, and manuscript revisions we have made in response to the reviews.

Upon the release of the initial reviews, we are encouraged that all three reviewers recognized the effectiveness of learning WAMs directly on frozen predictive visual representations without inheriting a pretrained visual generator. Across the reviews, they highlighted the broad empirical validation across simulation and real-world manipulation and the substantial downstream benefits of predictor pretraining without action labels, particularly under distribution shifts. We appreciate this shared recognition of the framework’s practical value and the promise of using additional video experience to improve robot control.

Design Motivation over Joint Optimization

Regarding the shared concern on joint optimization raised by Reviewers K7Fe and uYka, we would like to clarify the rationale for joint optimization further right befor summarizing the additional experiments. Our central question is whether a predictive visual latent space learned at scale is enough for directly learning an effective WAM by playing a role as the visual latent foundation, instead of inheriting a complete pretrained visual generator. A natural way to investigate this question is to adopt the standard training objective of WAM, i.e., learning future prediction and action generation together in the training phase, but do it from scratch in a single stage on the same frozen visual foundation. This motivates our single-stage joint optimization design. The common architecture and training recipe also enable clean ablations of the training objectives and controlled comparisons across different visual foundations.

Additional Experiments during Rebuttal Stage

Following the reviewers’ suggestions, we have dedicated the first week in the discussion stage to conducting extensive additional experiments and analyses to thoroughly address the constructive feedback from all reviewers.

We summarize the key updates and additional experiments included in our revised manuscript and individual responses below:

CategoryReviewers’ ConcernsOur Additional Experiments & Responses
Joint Training and Downstream AdaptationK7Fe and uYka: Whether joint adaptation is needed after predictor pretraining, particularly compared with freezing the predictor.We evaluated four strategies beginning from the same DROID-pretrained initialization. Full joint adaptation achieves 92.22% on LIBERO-Plus in a controlled rerun, compared with 64.18% for freezing the predictor, 87.97% for blocking action gradients, and 88.94% for removing future supervision. These comparisons establish the benefits of predictor adaptation and of retaining both objectives. Details are provided in Section 5.4, Appendix F.1, Table 14.
Rigorously Identitying Pretraining TransferWyyA: Whether the gains arise specifically from future prediction or from additional scenes and robot states in DROID data.We perform a controlled experiment by using non-future-predictive pre-training objective with the same DROID data, conditioning, predictor architecture, and pretraining updates, keeping the visual encoder frozen in both settings. Future prediction improves LIBERO-Plus from 84.81% to 92.22% (+7.41 pp), demonstrating a substantial additional transfer benefit from in-the-wild general future modeling knowledge. Details are provided in Section 5.4, Appendix F.2, Table 15.
Transfer from Human VideosuYka and WyyA: Whether useful transfer extends beyond robot videos and proprioceptive conditioning.We pretrained the predictor on EgoVerse human egocentric videos paired with instructions, omitting the state branch entirely. Without robot states or action labels, this improves LIBERO-Plus from 79.25% to 82.92% (+3.67 pp), showing useful transfer despite substantial embodiment and viewpoint differences. Details are provided in Section 5.4, Appendix F.3.
Evaluation Reliability and Encoder RankingK7Fe: Stability of the ViT-L/ViT-G difference and whether encoder rankings persist after predictor pretraining.Across three independent training runs, ViT-L retains its LIBERO-Plus advantage: 79.05 ± 0.50% versus 77.79 ± 0.56% for ViT-G. Additional DROID-pretraining experiments also reproduce the observed encoder-generation and encoder-scale rankings. Details are provided in Appendices C.2–C.3, Tables 11–12.
AuthorsOpenReview ↗

General Response (Part 2)

CategoryReviewers’ ConcernsOur Additional Experiments and Responses
Pretraining Exposure and EfficiencyK7Fe: How much pretraining is needed and whether downstream gains continue to improve or approach saturation.We evaluated checkpoints at 20k, 50k, 70k, and 100k updates. At 20k updates, LIBERO-Plus already reaches 92.11%, within 0.11 pp of the default 100k checkpoint, showing that most of the transfer benefit is obtained with one-fifth of the updates. Details are provided in Appendices E and F.4, Table 17.
Novel-Task GeneralizationWyyA: How V-JEPA Policy generalizes to unseen instructions and novel task compositions without inheriting a pretrained text-aligned generative backbone.We evaluated V-JEPA Policy on the LIBERO-Pro Task axis using 2,000 rollouts per policy, testing changed task goals and success conditions. The from-scratch and pretrained-predictor models achieve 17.95% and 22.25%, respectively. The latter exceeds the reported results of FastWAM (14.90%) and ImageWAM (19.05%). Details are provided in Section 5.2, Table 2, Appendix A.1.
Real-World Failure AnalysisK7Fe: Which behaviors help explain the performance gap to PRTS.We added a qualitative Table Cleanup rollout comparison with PRTS and FastWAM under the same initial object layout. The analysis identifies repeated unsuccessful reaching and failure to switch arms during debris removal, and discusses relevant differences in training objectives and pretraining dataset scale. Details are provided in Appendix B.3, Figure 5.
Predictive versus Generative FoundationsWyyA: What is gained or lost by using predictive latents instead of retaining a pretrained visual generator.We expanded the discussion of parameter and deployment efficiency, and vision–language grounding. We also identify applications that benefit from explicit visual synthesis, including inspecting anticipated outcomes, scoring imagined rollouts with VLMs, and rendering visual subgoals. Discussion details are provided in Appendix J.

These additional experiments and clarifications have substantially improved the empirical rigor, clarity, and completeness of our work, and we are grateful for these valuable questions and constructive suggestions. The revision of our manuscript was submitted alongside the submission of our rebuttal, with changes highlighted in red.

With a few hours remaining in the discussion period, we would welcome any final feedback on whether the additional evidence addresses your concerns. If any points remain unresolved or would benefit from further explanation, we would be happy to clarify them right before the discussion closes. We hope these additional results and revisions address your concerns and support a more favorable final assessment of our work.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.