- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies predictive JEPA representations for world-action modeling. The authors pretrain a diffusion transformer from scratch in a frozen V-JEPA feature space and combine a VAE stream for absolute scene information with a delta-JEPA stream for changes relative to the current observation. Experiments on RoboTwin 2.0 show improved success under held-out domain randomization.
Critical strengths
Using predictive features as the state space of a generative world-action model is a meaningful direction. Controlled re-rendering experiments show that the selected JEPA encoder is less sensitive to texture changes, providing concrete support for this motivation. Evaluation covers all 50 RoboTwin tasks with randomized scenes held out of downstream training. The predictive-stream ablation improves OOD success by 6 percentage points, supporting the practical benefit of the added branch.
Critical weaknesses
- The benefit of the JEPA representation is not isolated. The VAE-only and JEPA-only models differ in backbone and pretraining, while removing the predictive stream also reduces capacity. The matched-step comparison does not establish whether the OOD gap would persist after both models are trained to a performance plateau. These limitations leave the claimed representation advantage difficult to distinguish from capacity and optimization effects.
- The contribution of delta-JEPA is not established. There is no comparison against absolute JEPA features within the same dual-stream architecture. The existing branch ablation therefore cannot determine whether temporal differencing improves robustness beyond simply adding a second representation, leaving a central design claim unsupported by a direct test.
Questions for authors
- Can the authors provide a VAE-only baseline under the same clean-only protocol? If possible, a clean-only adaptation of LingBot-VA would also help assess the gain over a strong VAE-based model.
- Does the VAE baseline catch up with further training?
- What changes when delta-JEPA is replaced with absolute JEPA features, keeping the dual-stream architecture, mask, and training setup fixed? This would isolate the contribution of temporal differencing.
We thank Reviewer xqnx for the positive assessment of our problem formulation, diagnostic controls, and 50-task evaluation, and for pinpointing the exact controlled ablations needed. All three requested experiments are now in §§4.4-4.5 (Tables 3 & 4, Figure 3).
1. Matched Clean-Only VAE-Only Baseline, LingBot-VA & Granularity Controls (W1 & Q1)
a) Same-backbone VAE-only baseline (
JEPA-DiT-ReconOnly,lambda_p = 0): On our exact 32-layerOmniJointAttentionbackbone, pretraining lineage, and 2,500demo_cleantrajectories, disabling only the predictive stream (lambda_p = 0) isolates representation from backbone and pretraining (Tables 3 & 4a):bs=256, 10k): Without JEPA (lambda_p=0)bs=256, 10k): Withdelta-JEPA(lambda_p=1)bs=512, 11k):JEPA-DiT-ReconOnly(lambda_p=0)bs=512, 11k, 100 trials/task): 4-streamJEPA-DiTJEPA-DiT-EEFAt step 11k,
ReconOnlymatchesJEPA-DiTon Clean (85.0%vs.85.2%), confirming full manipulation capacity on its own, while addingdelta-JEPAyields a pure +13.0 point OOD gain (46.0% -> 59.0%over 10,000 rollouts; a second 10,000-rollout run gives84.6% / 59.2%,<0.6%variance). Standalone from-scratch 3-streamJEPA-DiT-PredOnly(zero VAE tokens, matchedbs=512, lr=3e-5) also reaches 76.0% Clean / 35.2% OOD.b) Clean-only
LingBot-VAadaptation: PublishedLingBot-VA(92.2% / 91.0%) uses mixed training (50clean +500randomised demos/task in-distribution). Our Wan-5BLingBot-VAreproduction matches this under mixed training (90.0% / 90.4%), but adapting Wan-5B strictly ondemo_cleandrops to 68.8% Clean / 0.0%-14.0% OOD (§4.4), mirroringFast-WAM(77.8% -> 1.9%).c) Identical temporal granularity & capacity check: Under
delta-JEPA(§3.3, Eq. 2), V-JEPA 2.1 (tau=2) encodes boundary pairs[I_0,I_0], [I_16,I_16], [I_32,I_32], givingF=3slots (G=2future steps xN=16actions), matching the causal VAE'sF=3grid 1-to-1. Moreover, underrecon ⊥ jepaandtxt -> {txt}, adding thedelta-JEPAexpert adds zero parameters or attention keys to text/reconstruction, andAbsolute-JEPAbelow confirms extra stream capacity alone hurts OOD without differencing.2. Does Longer Training Close the OOD Gap? (Q2)
No (Table 4c, §4.5). Across all 50 tasks under clean-only training, extending past step 11k slightly raises Clean while degrading OOD by
-10.2points due to clean-appearance memorisation:ReconOnlysimilarly peaks at step 11k (85.0% / 46.0%) without catching up thereafter.3. Direct 4-Stream Comparison:
Delta-JEPAvs.Absolute-JEPA(W2 & Q3)Holding the exact 4-stream architecture, frozen encoders, and
recon ⊥ jepamask fixed, we compare predictingphi(g) - phi(g-1)(Delta-JEPA) vs. un-differencedphi(g)(Absolute-JEPA) on a 4-task bimanual suite (place_container_plate,place_empty_cup,stack_bowls_two,stack_bowls_three; Table 4b, §4.5):recon ⊥ jepaMask)phi(g), matched LoRA)phi(g), slot-tuned)phi(g)-phi(g-1), matched LoRA)phi(g)-phi(g-1), full fine-tune)Without differencing,
Absolute-JEPAreaches72.5%-77.5%Clean yet collapses to 2.5%-7.5% OOD because static scene redundancy lets the policy exploit clean appearance shortcuts. Subtractingphi(g-1)isolates pure state transitions, lifting matched-budget OOD by +32.5 points (>5x) and full fine-tuning to 87.0% OOD.