- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces Latent Action JEPA (La-JEPA), a world-action model that has both world model DiT (future state predictor) and action DiT (robot action generator) jointly trained from scratch in a pretrained V-JEPA2.1 latent space. The model jointly predicts future states and actions using flow matching during the training. The action tokens are designed to attend to the current observation tokens, so that during inference the future state tokens can be dropped. The proposed method performs well over multiple simulation and real-world benchmarks like LIBERO-Plus and RoboTwin.
Critical strengths
The paper is clearly presented and easy to follow, the selected experiments are reasonable and the authors provided helpful details. Table 4 is informative with important ablations on action future prediction, latent representations and so on.
Critical weaknesses
The contributions of this paper need further clarification. Given the current state, it could be easily regarded as a natural extension of Fast-WAM and Faster-WAM as mentioned in the paper, or VLA-JEPA[1]. The claim of a compact, effective world-action policy, together with evidence to guide its design mainly comes from the large scale pretraining of V-JEPA and the jointly training of future prediction and action generation, neither of them is from this paper. I would appreciate further clarifications on this if I get it wrong.
[1] Sun, Jingwen, et al. "Vla-jepa: Enhancing vision-language-action model with latent world model." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.
Questions for authors
No
3 comments
I appreciate the authors' rebuttal, especially the new clarification on the contribution of this paper. I understand the paper suggests we should adapt latent space rather than work with large generative backbones, and V-JEPA is a good example. The experiment of V-JEPA features → Wan2.2 VAE latents supports that the latent space of V-JEPA is better than Wan2.2 VAE.
However, the current formulation of the paper looks more like a tech report, instead of a research paper. If the claim is to show the advantages of the latent space over generative backbones, I would expect to see a deeper analysis on the property of the latent space, e.g. what makes such a good latent space, what makes it different than a generative latent and so on. Though there are strong results shown by the current paper, answering these questions should make the paper more insightful to the community and inspire further research.
Considering all these, I would like to maintain my current score.
We thank the reviewer for their reply and for acknowledging the strength of our results. We realize our claim was not stated clearly enough and will clarify it in the final version: we do not oppose adapted latent spaces to pretrained backbones. Training from scratch is not integral to our method, but it is central to our claim: tailoring the policy's design, including its latent space, matters more than the knowledge transferred from a heavily pretrained generative backbone.
Which latent spaces suit world modeling is studied by dedicated work: V-JEPA 2.1 reports the state of the art among frozen encoders on prediction and world-model planning (Mur-Labadia et al., 2026), and a recent comparison of six encoders under a fixed world-model protocol ranks semantic encoders ahead of reconstruction ones, with "V-JEPA 2.1 (strongest overall on policy)" [1]. Our contribution is to show how far a policy can go by picking the best currently available space and designing the policy around it from scratch. Our feature-space comparison confirms this choice: V-JEPA features outperform Wan2.2 VAE latents in all four evaluations (Section 4.1, Figure 3B), with the largest LIBERO-Plus gap under background textures (63.0% vs. 36.0%, reply to Reviewer QK7p) and visual perturbations.
More generally, we understand the concern that the paper reads as a technical report: the text mostly puts forward the strength of our results, which obscures the novelty. It should emphasize two points:
First, most recent strong WAM contributions focus on scaling the video backbone and its data, but ignore the choice of latent space, as they are constrained by existing backbones made for video. We take the opposing view: the choice of latent space is just as critical, and even small video backbones trained from scratch can be competitive. This does not exclude pretraining as future work (limitations, Section 6, "Large-scale pretraining", p. 9).
Second, training the video backbone natively in that space is important: LeapBot-WA aligns the Fast-WAM architecture with V-JEPA 2.1, our space, and we still significantly outperform it. Previous non-VAE policies under-perform VAE-based baselines, sending mixed signals about the viability of moving away from VAEs. By reaching the state of the art with faster inference and fewer parameters, our work shows that designing or pretraining in spaces adapted to robotics is a more effective path than simply scaling data and model size.
In the final version, we will (1) rewrite the abstract and introduction around these two points, with pretraining in an adapted space as complementary to our approach; (2) open our analysis with the feature-space comparison (Section 4.1, Figure 3B), currently one ablation among many, and expand the discussion of what makes this space suited to control; and (3) discuss the pretraining scale behind each baseline's backbone. We hope this addresses the reviewer's remaining concerns.
[1] Nilaksh, et al. "Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Models." Advances in Neural Information Processing Systems, Evaluations and Datasets Track, 2026.
We thank the reviewer for their feedback. We also invite them to take a look at our general response, which reports updated scores on our benchmarks and adds full-RoboTwin results to show scalability.
On the contribution. We agree that the main novelty of La-JEPA is not architectural. The current trend in world-action models is to scale video generation pretraining. This is natural: it is something WAMs can exploit better than VLAs, as world-modeling objectives have been shown to make policies scale better with data (DriveVLA-W0 [Li et al., 2025]). Fast-WAM and Faster-WAM follow this direction and keep a pretrained video generator as the policy backbone. Our work shows that more of the value currently lies in an adapted latent space and in the regularizing effect of the world loss itself than in knowledge transferred from video generation pretraining: La-JEPA has no generative backbone and trains a compact policy from scratch, yet matches or exceeds these much larger models with faster inference speeds. No other policy without a pretrained VLM or video backbone comes anywhere near competitive performance, let alone the state of the art among policies without embodied pretraining, which La-JEPA reaches on both LIBERO-Plus and RoboTwin.
We will rewrite the introduction to state our contribution more clearly: not a new architecture, but evidence, against the field's trend, that a compact policy trained from scratch in an adapted latent space matches or exceeds policies built on large generative backbones, with the unified ablation identifying which ingredients matter.
On V-JEPA pretraining and joint training. We use both as ingredients and do not claim them as our contributions. The question we answer is which of them matters, and how. The design study isolates the latent space and the future-prediction objective, and the unified ablations added in this rebuttal confirm it under a single recipe (see our answer to Reviewer QK7p): replacing V-JEPA features with VAE latents costs 13.7 points on clean RoboTwin and drops randomized success below 5%, and removing future prediction costs 2 to 5 points on every benchmark, even though futures are never predicted at inference. We also note that V-JEPA 2.1 is pretrained on about 163M images and videos (VisionMix163M: 142M images and about 21M videos), substantially less than the billions of images and videos behind the video generators that WAMs such as Fast-WAM build on: La-JEPA's results therefore cannot be explained by more pretraining data; they point to the representation itself.
On VLA-JEPA. VLA-JEPA uses JEPA-style latent prediction to pretrain a VLA backbone on video, then fine-tunes an action head. La-JEPA does the opposite: it has no pretrained policy backbone and trains the policy directly in the adapted V-JEPA latent space, with future prediction as a joint training objective rather than a pretraining stage. La-JEPA S also outperforms it on LIBERO (97.9 vs. 97.2) and LIBERO-Plus (80.0 vs. 78.0, official text, episode-weighted; VLA-JEPA reports 79.5 as a category average), our B and L models extending that advantage. We will add this comparison to the related work.