- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes V-JEPA Policy, a world-action model built on the latent space of a frozen V-JEPA 2.1 visual encoder, without inheriting a pretrained visual generator. Its central contribution is a framework that jointly trains an instruction-conditioned future-latent predictor and a flow-matching action expert from scratch. The predictor processes observed context and learnable future queries through bidirectional attention, while its layer-wise context key–value states condition action generation. The paper also investigates whether predictor-only pretraining on DROID videos, using language and proprioceptive inputs but no action labels, provides transferable future-modeling knowledge.
Validation covers LIBERO, LIBERO-Plus, RoboCasa-GR1, and two real-world bimanual tasks. Controlled encoder comparisons favor predictive visual representations over discriminative, reconstructive, and video-understanding alternatives, particularly under distribution shifts. Ablations support the contributions of both future queries and explicit future-prediction supervision. Predictor pretraining improves downstream performance, notably raising LIBERO-Plus success from 79.25% to 91.50%, exceeding the gains from extended downstream training. Deployment profiling also shows lower memory usage and latency than FastWAM.
Critical strengths
-
The paper shows that a frozen predictive visual encoder can support effective world-action modeling without a pretrained visual generator. The model achieves competitive results with larger WAMs across simulation and real-world tasks. Encoder comparisons and ablations support the approach, making it a useful alternative to adapting large generative backbones.
-
Predictor pretraining without action labels improves performance across all evaluated benchmarks and real-world tasks, particularly on LIBERO-Plus (79.25% to 91.50%). These gains exceed those from longer downstream training, suggesting that the predictor learns transferable knowledge beyond the task demonstrations. This provides a promising way to use additional video data for robot learning.
Critical weaknesses
- The pretraining results show positive transfer, but do not isolate the transfer of future-modeling knowledge. DROID pretraining also exposes the predictor to additional scenes, instructions, and robot states, while the extended-training baseline only reuses downstream demonstrations. A comparison using the same DROID data and a non-future-predictive pretraining objective would help establish whether the gains specifically come from learning to predict future states. Without this control, the results support the usefulness of predictor pretraining more clearly than the claimed source of its transfer benefits.
Questions for authors
-
Could the authors clarify the relative strengths of V-JEPA Policy and WAMs? Both approaches inherit useful representations from video pretraining, so it would be helpful to explain what is gained or lost by using a frozen predictive latent space instead of a pretrained generative backbone. Are there domains or tasks where retaining the visual generator offers a clear advantage, or may even be necessary?
-
V-JEPA Policy learns language conditioning during predictor and policy training, whereas text-conditioned diffusion backbones already have pretrained text–visual alignment. Could this difference affect generalization to unseen instructions or novel task compositions? It would be helpful to discuss or evaluate this beyond the language perturbations in LIBERO-Plus.
6 comments
Part2
[Q1] Relative strengths of V-JEPA Policy vs. generator-based WAMs, and when retaining a visual generator is advantageous.
Response: Thank you for this question. Both approaches benefit from large-scale visual pretraining. Our central question is whether the resulting predictive representations can provide a sufficient foundation for effective WAM learning without inheriting a complete pretrained visual generator. In our opinions, there could exist the following benefits and trade-offs.
- Gains from a frozen predictive latent space.
Decoupling the representations that entail world-modeling knowledge from heavy generative backbones. Our 0.9B policy combines a frozen V-JEPA 2.1 encoder with a single-pass predictor and a lightweight action expert, with only 0.6B trainable parameters. The base policy lies on the empirical parameter–performance Pareto frontier in all five evaluated settings among the compared methods (Fig. 6, Appendix H). Under the matched profiling protocol in Table 8, it requires 4.66 GiB peak VRAM, compared with 12.74 GiB for FastWAM, a 63.4% reduction.
Predictive features for robust control. V-JEPA learns through latent feature self-prediction, emphasizing spatial and temporal structure without requiring pixel reconstruction. This provides an effective foundation for control under appearance changes. When used as frozen visual foundations under the same downstream recipe, V-JEPA 2.1 achieves 79.25% on LIBERO-Plus, compared with 46.43% for WAN2.2 VAE, with particularly large advantages under texture (80.02% vs. 10.13%) and lighting (97.20% vs. 32.05%) perturbations (Table 5).
- Trade-offs and advantages retained by visual generators.
Pretrained vision–language grounding. Text-conditioned generative backbones inherit cross-modal associations learned from large-scale paired data. V-JEPA is pretrained on vision alone. Its feature are instead aligned with pretrained T5 embeddings during predictor and policy training, using substantially less paired visual–language data. Our policy may therefore underperform these backbones on novel instructions or complex paraphrases, particularly when trained only on downstream demonstrations. As shown in Q2, predictor pretraining on DROID video–instruction pairs substantially improves language robustness and also improves novel-task generalization.
Explicit visual synthesis. Retaining a generator is useful when predicted RGB frames are needed for the inspection of anticipated outcomes, VLM-based scoring of imagined rollouts, or visual subgoal generation. V-JEPA Policy predicts future latents and actions, so supporting these applications would require an additional decoder or generative component.
We discuss these trade-offs in Appendix J.
[Q2] Impact of language conditioning learned during predictor/policy training on unseen instructions and novel task compositions.
Response: This is an excellent point that directly connects to [Q1] above. As V-JEPA Policy uses pretrained T5 embeddings, but learns their association with visual latents during predictor and policy training rather than inheriting a text-aligned diffusion backbone, we observe a clear distinction between from-scratch training and action-free predictor pretraining.
| Method | Visual Foundation | Vision–Language Alignment Source | LIBERO-Plus: Language (%) | LIBERO-Pro: Novel Task (%) |
|---|---|---|---|---|
| FastWAM (6.0B) | WAN Video Diffusion | Web-scale Text–Video + Downstream | 68.9 | 14.90$^\dagger$ |
| ImageWAM (4.5B) | Image-Editing Backbone | Web-scale Text–Image + Downstream | 91.4 | 19.05$^\dagger$ |
| V-JEPA Policy (From Scratch) | Frozen V-JEPA 2.1 | Downstream Demos Only | 64.9 | 17.95 |
| V-JEPA Policy (Pretrained Predictor) | Frozen V-JEPA 2.1 | DROID Video–Text + Downstream | 94.6 | 22.25 (+4.30 pp) |
denotes that task scores are borrowed from the corresponding entries in Table 12 in WAVE-0[3]
- Language robustness improves through video–instruction pretraining. When trained from scratch on 1,693 LIBERO demonstrations, V-JEPA Policy achieves 64.9% on Language perturbations, trailing text-pretrained ImageWAM (91.4%, Table 1). However, pretraining the predictor on DROID video–instruction pairs yields our largest single-axis gain (+29.73 pp, Fig. 2b), reaching 94.6% and surpassing ImageWAM (91.4%) and (88.5%).
Part3
- The improvement extends to novel task goals. We additionally evaluated our policies on the LIBERO-Pro[2] Task axis using 2,000 rollouts. In the official implementation, Task perturbations change the instruction together with the relevant objects and symbolic success conditions. They therefore require completing new task goals, such as opening a different drawer, rather than following an alternative description of the same goal. Predictor pretraining improves novel-task success from 17.95% to 22.25% (+4.30 pp), exceeding the corresponding reported results of FastWAM and ImageWAM. These results show that the benefits extend beyond linguistic perturbations to executing changed task goals.
That said, novel-task generalization remains challenging, but the observed gains show that a frozen predictive visual foundation can support improvements in both capabilities through predictor pretraining.
These evaluations are included in Section 5.2 and Appendix A.1 (Table 2).
References
[1] Assran, Mahmoud, et al. "Self-supervised learning from images with a joint-embedding predictive architecture." 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023.
[2] Zhou, Xueyang, et al. "LIBERO-Pro: Towards robust and fair evaluation of vision-language-action models beyond memorization." arXiv preprint arXiv:2510.03827 (2025).
[3] Wang, Haomin, et al. "WAVE-0: Bridging VLA Policies and WAMs with Lightweight Latent Future Prediction." arXiv preprint arXiv:2610.18568 (2026).
Part1
Dear Reviewer WyyA,
We sincerely appreciate your precious time and constructive comments. In the following, we would like to answer your concerns separately.
[W1] Isolating the transfer of future-modeling knowledge during pretraining.
Response: Thank you for highlighting this distinction. We agree that DROID pretraining may expose the predictor to additional tabletop scenes, language instructions, and especially the same Franka Panda's robot states. To verify how much the downstream gains stem exactly from learning to predict future states, we conducted pre-training with a non-future-predictive objective on the same dataset. Here we adopt the spatial masking denoising objective same as what I-JEPA[1] does. We also conducted a complementary experiment on human videos to examine transfer beyond robot-specific pretraining data, since the embodiment involved in DROID is same as that in LIBERO.
All pretrained variants use 100k optimizer updates, followed by the same downstream LIBERO training and LIBERO/LIBERO-Plus evaluation recipe.
We conclude three observations from the results.
Non-future-predictive pretraining also provides useful transfer. DROID spatial prediction improves LIBERO-Plus success from 79.25% to 84.81% (+5.56 pp). This establishes that useful transfer can arise even when the predictor’s pretraining task does not forecast future observations.
Future prediction provides a substantial additional benefit on the same data. Replacing current-context spatial prediction with future prediction raises LIBERO-Plus success from 84.81% to 92.22%, an improvement of approximately 7.4 pp. With the DROID dataset, conditioning information, predictor architecture, and pretraining budget matched, this comparison supports the additional value of learning future prediction for downstream control.
Such transfer also extends to human egocentric video–language data even though there is a significant embodiment gap between human and robotic arm. Instruction-only EgoVerse pretraining reaches 82.92% (+3.67 pp over scratch). Its lower score than DROID spatial pretraining is consistent with differences in data alignment: DROID shares LIBERO’s Franka Panda embodiment and provides external and wrist-camera views together with proprioception, whereas EgoVerse provides a single human egocentric view without robot states. The within-DROID comparison tests the benefit of the pretraining objective, while the EgoVerse experiment shows that our predictive pretraining framework can also benefit from human videos across these embodiment and observation differences.
These controls are reported in Section 5.4 and Appendices F.2–F.3 (Tables 15).