- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents World Tokens, a framework that uses future-video modeling during training to improve embodied policies. A World Adapter converts VLM features into shared tokens that condition both a video denoiser and an action expert. These tokens serve as the action expert’s sole visual-language context, while the video branch receives only an edge-map cue of the current scene. At deployment, the video branch is discarded. The method is evaluated on LIBERO, LIBERO-Plus, SIMPLER, and real-world manipulation.
Critical strengths
1.A clear architectural design addressing predictive supervision and control. The exclusive shared-token interface and restricted visual conditioning provide a concrete approach to encouraging the policy and video model to rely on a common representation. Removing the video branch at deployment also makes the approach practically appealing.
2.Broad evaluation with relevant architectural ablations. The paper reports improvements over its matched action-only baseline across simulation, robustness benchmarks, and real-world tasks. The ablations on video supervision, VLM bypass connections, adapter design, and RGB conditioning provide useful evidence that the proposed components matter within the evaluated configuration.
Critical weaknesses
1.The comparisons do not sufficiently establish the contribution beyond existing predictive representation learning. Prior work such as FLARE and VLA-JEPA already uses future latent prediction to improve policies without requiring online video generation. The distinguishing contribution here is therefore the specific shared interface and video-supervision design. However, the paper does not compare its video objective against a lightweight future-latent prediction objective under the same adapter, action architecture, and training setup. Although published VLA-JEPA results are included, these comparisons cannot isolate the benefit of the proposed supervision mechanism. Relative to the adapter-equipped variant without video supervision, the gain is 0.8 percentage points on LIBERO overall and 2.0 points on LIBERO-Long. These improvements are encouraging, but stronger controlled comparisons and training-cost measurements are needed to justify the additional video-model training.
2.The experiments do not adequately isolate the proposed temporal mechanism or establish its robustness. Requiring both branches to consume the same tokens does not ensure that they use the same information within those tokens. Moreover, restricting the video input to an edge map may encourage the tokens to encode missing appearance information, rather than specifically action-relevant dynamics. The attention visualizations and architectural ablations are consistent with the proposed explanation but do not distinguish it from improved static visual representation or auxiliary-task regularization. Controls based on current-frame reconstruction or static repeated-frame prediction are missing. The central mechanistic ablations are also concentrated on LIBERO; extending them to distribution shifts and real-world tasks, together with multiple independent training runs, would make the conclusions substantially stronger.
Questions for authors
- Under an identical policy architecture and training setup, how does future-video supervision compare with FLARE-style latent alignment or a JEPA-style future-state prediction objective?
- How much improvement remains when comparing future-video supervision against current-frame reconstruction or a repeated-current-frame video objective?
- Do the gains from video supervision and exclusive routing persist on LIBERO-Plus and real-world tasks, and across independent training seeds?
- What are the additional training compute and memory costs of the video branch, and how does the performance–cost trade-off compare with lightweight predictive objectives?
2 comments
Dear Reviewer btRi,
Please respond to the author soon and engage in the discussion.
Best,
AC
Response to Reviewer btRi
We thank the reviewer for the thoughtful analysis and for suggesting the exact controls needed to isolate temporal prediction and evaluate cost–benefit trade-offs. Below we address Q1–Q4 (and W1–W2), referencing our RoboTwin 2.0 setup in the General Response.
Q1. Future-Video Supervision vs. Lightweight Future-Latent Alignment (W1)
Under an identical policy architecture, 30k-step schedule, and batch size (128), we compared
World Tokensagainst a FLARE-style baseline on RoboTwin-Clean that aligns predicted tokens to future-frame VLM encodings:w/o WM(adapter retained, action-only)FLARE-style latent alignment yields only a marginal gain over
w/o WMand trailsWorld Tokensby 2.9–5.0 pts, showing that aligning to VLM feature targets does not capture the spatiotemporal dynamics provided by a pretrained video world model. (More broadly, neither FLARE nor VLA-JEPA leverages a pretrained video generation model; we believe pretrained video world models provide much stronger physical and spatiotemporal priors for robotic control—which is why modern WAMs build directly on video generation backbones.)Q2. Future-Video Supervision vs. Static Reconstruction (W2)
To rule out static appearance reconstruction or generic regularization, we replaced future-video prediction with a static current-frame reconstruction objective under the same cross-embodiment video continuation setup on RoboTwin-Clean:
Static reconstruction drops success by 4.0 pts—falling even below the 58.8% checkpoint without extra video—confirming that policy gains come from predicting future scene dynamics over time.
Q3. Persistence Beyond LIBERO and Across Training Seeds (W2)
We extended both the video-supervision (
w/o WM) and exclusive-routing ablations to LIBERO-Plus and RoboTwin 2.0:w/o WM)w/o WM)As shown in the General Response, video supervision also scales on RoboTwin 2.0 to visual OOD generalization (8.7% → 28.8%) and unseen-task transfer (18.0% → 28.3% / 24.0%). Due to tight rebuttal timelines and physical robot scheduling constraints, we will include the R1 Pro
w/o WMevaluation in the revised manuscript (whereWorld Tokensalready improves overQwen-DiTfrom 59.4% to 76.0%).Q4. Training Compute and Cost–Benefit Trade-off (W1)
The table below isolates the training cost of the 0.5B adapter and
Cosmos-Predict2.5-2Bvideo branch on RoboTwin 2.0 (30k steps, global batch size 128, per-GPU bs=16):Qwen3-VL-2B, action-only)w/o WM, adapter-only)World Tokens, full)All video-branch overhead is discarded at deployment (62 ms per chunk). Lightweight objectives cannot match its dynamics modeling: FLARE alignment reaches only 55.93% (Q1), and truncating the 28-block Cosmos DiT to 4 randomly initialized blocks causes future-prediction loss to plateau at 0.0942 vs. 0.0181 (~5.2× lower) for the pretrained model. Thus, the training-only cost buys clear in-domain, visual OOD, and unseen-task gains at zero inference overhead.