acceptodds
Under review as a conference paper at ICLR 2027

World Tokens: Enhancing Embodied Policies with Training-Time World Modeling

Abstract

Vision-language-action (VLA) models are efficient at inference but do not explicitly model how physical scenes evolve as a task unfolds, whereas world-action models (WAMs) capture this evolution with pretrained video world models at the cost of keeping future generation or a large video backbone in the control loop. We introduce World Tokens, an embodied policy architecture that uses world modeling only during training, through an exclusive predictive interface. A World Adapter transforms VLM features into a fixed set of world tokens, which condition a jointly fine-tuned future-video denoiser and serve as the action expert's sole visual-language context. The denoiser receives only a coarse layout cue of the current scene, so neither it nor the action expert can bypass the world tokens, and future-video supervision directly shapes the representation used for control. At deployment, the world-model branch is removed, leaving only the VLM, World Adapter, and action expert. We evaluate World Tokens on LIBERO, the zero-shot perturbations of LIBERO-Plus, SIMPLER, and real-world manipulation with a Galaxea R1 Pro robot. With a 2B backbone and no embodied action pretraining, it performs strongly across these benchmarks and consistently improves over a matched action-only baseline, while running at VLA-level latency.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, world models, robot manipulation, video diffusion, representation learning, embodied AI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_btRiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents World Tokens, a framework that uses future-video modeling during training to improve embodied policies. A World Adapter converts VLM features into shared tokens that condition both a video denoiser and an action expert. These tokens serve as the action expert’s sole visual-language context, while the video branch receives only an edge-map cue of the current scene. At deployment, the video branch is discarded. The method is evaluated on LIBERO, LIBERO-Plus, SIMPLER, and real-world manipulation.

Critical strengths

1.A clear architectural design addressing predictive supervision and control. The exclusive shared-token interface and restricted visual conditioning provide a concrete approach to encouraging the policy and video model to rely on a common representation. Removing the video branch at deployment also makes the approach practically appealing.

2.Broad evaluation with relevant architectural ablations. The paper reports improvements over its matched action-only baseline across simulation, robustness benchmarks, and real-world tasks. The ablations on video supervision, VLM bypass connections, adapter design, and RGB conditioning provide useful evidence that the proposed components matter within the evaluated configuration.

Critical weaknesses

1.The comparisons do not sufficiently establish the contribution beyond existing predictive representation learning. Prior work such as FLARE and VLA-JEPA already uses future latent prediction to improve policies without requiring online video generation. The distinguishing contribution here is therefore the specific shared interface and video-supervision design. However, the paper does not compare its video objective against a lightweight future-latent prediction objective under the same adapter, action architecture, and training setup. Although published VLA-JEPA results are included, these comparisons cannot isolate the benefit of the proposed supervision mechanism. Relative to the adapter-equipped variant without video supervision, the gain is 0.8 percentage points on LIBERO overall and 2.0 points on LIBERO-Long. These improvements are encouraging, but stronger controlled comparisons and training-cost measurements are needed to justify the additional video-model training.

2.The experiments do not adequately isolate the proposed temporal mechanism or establish its robustness. Requiring both branches to consume the same tokens does not ensure that they use the same information within those tokens. Moreover, restricting the video input to an edge map may encourage the tokens to encode missing appearance information, rather than specifically action-relevant dynamics. The attention visualizations and architectural ablations are consistent with the proposed explanation but do not distinguish it from improved static visual representation or auxiliary-task regularization. Controls based on current-frame reconstruction or static repeated-frame prediction are missing. The central mechanistic ablations are also concentrated on LIBERO; extending them to distribution shifts and real-world tasks, together with multiple independent training runs, would make the conclusions substantially stronger.

Questions for authors
  1. Under an identical policy architecture and training setup, how does future-video supervision compare with FLARE-style latent alignment or a JEPA-style future-state prediction objective?
  2. How much improvement remains when comparing future-video supervision against current-frame reconstruction or a repeated-current-frame video objective?
  3. Do the gains from video supervision and exclusive routing persist on LIBERO-Plus and real-world tasks, and across independent training seeds?
  4. What are the additional training compute and memory costs of the video branch, and how does the performance–cost trade-off compare with lightweight predictive objectives?
2 comments
AuthorsOpenReview ↗

Response to Reviewer btRi

We thank the reviewer for the thoughtful analysis and for suggesting the exact controls needed to isolate temporal prediction and evaluate cost–benefit trade-offs. Below we address Q1–Q4 (and W1–W2), referencing our RoboTwin 2.0 setup in the General Response.

Q1. Future-Video Supervision vs. Lightweight Future-Latent Alignment (W1)

Under an identical policy architecture, 30k-step schedule, and batch size (128), we compared World Tokens against a FLARE-style baseline on RoboTwin-Clean that aligns predicted tokens to future-frame VLM encodings:

Supervision Objective (RoboTwin-Clean, 50 tasks)SR
w/o WM (adapter retained, action-only)55.6%
FLARE-style alignment (future-frame VLM encoding)55.93%
World Tokens (Cosmos-2B video loss)58.8%
World Tokens + Cross-embodiment video60.9%

FLARE-style latent alignment yields only a marginal gain over w/o WM and trails World Tokens by 2.9–5.0 pts, showing that aligning to VLM feature targets does not capture the spatiotemporal dynamics provided by a pretrained video world model. (More broadly, neither FLARE nor VLA-JEPA leverages a pretrained video generation model; we believe pretrained video world models provide much stronger physical and spatiotemporal priors for robotic control—which is why modern WAMs build directly on video generation backbones.)

Q2. Future-Video Supervision vs. Static Reconstruction (W2)

To rule out static appearance reconstruction or generic regularization, we replaced future-video prediction with a static current-frame reconstruction objective under the same cross-embodiment video continuation setup on RoboTwin-Clean:

Video Objective (with cross-embodiment video)SR
Static current-frame reconstruction56.90%
World Tokens (future-video prediction)60.9% (+4.0)

Static reconstruction drops success by 4.0 pts—falling even below the 58.8% checkpoint without extra video—confirming that policy gains come from predicting future scene dynamics over time.

Q3. Persistence Beyond LIBERO and Across Training Seeds (W2)

We extended both the video-supervision (w/o WM) and exclusive-routing ablations to LIBERO-Plus and RoboTwin 2.0:

Benchmark / AblationControlWorld Tokens
LIBERO-Plus (OOD)75.1% (w/o WM)76.8% / 78.4% (+video)
RoboTwin-Clean (video sup.)55.6% (w/o WM)58.8% / 60.9% (+cross-emb. video)
RoboTwin-Clean (routing)57.6% (dual-routing)58.8% (exclusive routing)

As shown in the General Response, video supervision also scales on RoboTwin 2.0 to visual OOD generalization (8.7% → 28.8%) and unseen-task transfer (18.0% → 28.3% / 24.0%). Due to tight rebuttal timelines and physical robot scheduling constraints, we will include the R1 Pro w/o WM evaluation in the revised manuscript (where World Tokens already improves over Qwen-DiT from 59.4% to 76.0%).

Q4. Training Compute and Cost–Benefit Trade-off (W1)

The table below isolates the training cost of the 0.5B adapter and Cosmos-Predict2.5-2B video branch on RoboTwin 2.0 (30k steps, global batch size 128, per-GPU bs=16):

Configuration (RoboTwin 2.0, 30k steps)Training ComputePeak VRAM / GPURoboTwin-Clean SR
Baseline (Qwen3-VL-2B, action-only)70.7 GPU-h42.36 GiB57.1% (StarVLA)
+ World Adapter (w/o WM, adapter-only)106.4 GPU-h51.09 GiB (+8.73 GiB)55.6%
+ Cosmos-2B (World Tokens, full)174.4 GPU-h79.15 GiB (+36.79 GiB)58.8% / 60.9%

All video-branch overhead is discarded at deployment (62 ms per chunk). Lightweight objectives cannot match its dynamics modeling: FLARE alignment reaches only 55.93% (Q1), and truncating the 28-block Cosmos DiT to 4 randomly initialized blocks causes future-prediction loss to plateau at 0.0942 vs. 0.0181 (~5.2× lower) for the pretrained model. Thus, the training-only cost buys clear in-domain, visual OOD, and unseen-task gains at zero inference overhead.

Dear Reviewer btRi,

Please respond to the author soon and engage in the discussion.

Best,

AC

Reviewer_ZP8bOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

World Tokens uses future-video prediction during training-time to improve an action policy, and removes the video branch at deployment. A query-based World Adapter compresses VLM features into 256 tokens that provide the action expert's sole visual-language context and condition a jointly fine-tuned video denoiser. The video branch receives an edge map as its first-frame cue. The paper evaluates the resulting policy on LIBERO, zero-shot LIBERO-Plus, SIMPLER, and R1 Pro manipulation, and studies video supervision, routing, token extraction, and conditioning through ablations. The reported results support an effective policy with modest additional inference latency, although the proposed explanation of why the interface works is less firmly established.

Critical strengths
  1. A useful deployment design with measured inference cost. The policy uses visual-language context through a shared token interface and removes the entire video branch after training. Table 1 and Appendix B report 62 ms per action chunk versus 52 ms for Qwen-DiT, which supports the practical value of the design. Appendix B also reports a 1.5x advantage over Fast-WAM when denoising steps and compilation settings are matched. The contribution is a concrete way to apply predictive supervision while retaining a relatively efficient controller.
  2. Positive evidence across several evaluation settings, including a meaningful ablation. World Tokens improves over Qwen-DiT on LIBERO (98.2% vs. 96.9%), and the real robot (76.0% vs. 59.4%). Crucially, Table 2 retains the adapter in the w/o wm baseline: adding video supervision raises LIBERO-Long from 95.0% to 97.0%. This comparison provides evidence beyond simply increasing deployed model capacity, and the robustness and real-robot results make the method relevant beyond a saturated simulation benchmark.
Critical weaknesses
  1. The benefit of video supervision needs broader validation. Table 1 shows comparable performance between strong VLAs and WAMs. Although the LIBERO ablation supports L_vid, retained-Adapter, action-only controls on other benchmarks are needed to establish consistent gains attributable to video supervision.
  2. The paper specifies the Canny thresholds and smoothing parameter, but does not evaluate sensitivity to them. The RGB-versus-edge ablation is also confined to LIBERO, leaving the cue’s robustness in more complex scenes unclear.
  3. A direct video-model baseline is missing. The authors could combine the same video world model with the same action expert, without the VLM and World Adapter. Comparing success rate, latency, and memory usage under matched training conditions would clarify whether the proposed architecture offers a better performance–efficiency trade-off than directly using the video model for control.
  4. Limited evaluation on challenging benchmarks. Much of the paper’s controlled evidence comes from LIBERO, where several methods already achieve high success rates. Although the paper also evaluates on LIBERO-Plus, SIMPLER, and a real robot, the key ablations are confined to LIBERO. Evaluation on larger and more challenging benchmarks, such as RoboTwin or RoboDojo, would help establish whether the gains generalize.
Questions for authors
  1. Could you report the performance of the retained-Adapter, action-only baseline (w/o wm) on other benchmarks, e.g. LIBERO-Plus, and the real robot?
  2. Could you provide representative RGB–edge pairs from complex scenes and clarify how performance varies with the Canny thresholds and smoothing parameters?
  3. Could you compare the proposed method with a baseline combining the same video world model and action expert, without the VLM and World Adapter.
  4. Have you tried alternative first-frame cues beyond RGB and Canny edges (e.g., grayscale, Gaussian-blurred RGB, learned low-frequency encoder)? Does a learned cue outperform hand-crafted edges?
  5. What is the training-time compute and memory overhead introduced by the 0.5B adapter and the joint WM fine-tuning?
  6. Please provide more results on RoboTwin or RoboDojo?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ZP8b

We thank the reviewer for the positive assessment of our deployment design, latency analysis, and controlled ablations. Below we address Q1–Q6 (and W1–W4) in order.

Q1. w/o WM Results on Other Benchmarks (W1)

We evaluated the retained-Adapter, action-only baseline (w/o WM) on both LIBERO-Plus and RoboTwin 2.0:

Benchmarkw/o WMWorld Tokens+ Actionless Video
LIBERO-Plus (OOD robustness)75.1%76.8% (+1.7)78.4% (+3.3)
RoboTwin-Clean (50 bimanual tasks)55.6%58.8% (+3.2)60.9% (+5.3)

Removing L_vid consistently lowers success on both benchmarks, confirming that gains come from training-time world modeling rather than added adapter capacity. Due to tight rebuttal timelines and physical robot scheduling constraints, we were unable to run an additional w/o WM checkpoint on the R1 Pro robot during the rebuttal window, and therefore leave this for future work.

Q2. Canny Parameter Sensitivity and Complex Scenes (W2)

We ran a controlled ablation on LIBERO shifting the default soft, sparse edge map to an aggressively dense, hard binary edge map (~10× higher edge density, no Gaussian blur):

Canny ConfigurationLIBERO-Long SR
Default (60 / 140, σ=2.5\sigma=2.5)97.0%
Dense/Sharp (10 / 20, σ=0.0\sigma=0.0)96.8%

Both achieve nearly identical loss curves and LIBERO-Long success (97.0% vs. 96.8%), indicating that Canny serves merely as a coarse structural hint to relieve World Tokens from reconstructing exact spatial layouts, rather than a fragile hyperparameter. Representative RGB–edge pairs across LIBERO, SIMPLER, and RoboTwin 2.0 are provided in Appendix I (Figure S5).

Q3. Direct Video-Model + Action-Expert Baseline (W3)

We implemented a direct video-model baseline on RoboTwin-Clean by removing Qwen3-VL-2B and the World Adapter, connecting Cosmos-Predict2.5-2B directly to the same DiT-B action head via single-pass low-noise features:

Metric (H100 80GB, 3 views)Direct Video-Model (Cosmos-2B + DiT-B)World Tokens (Ours)
RoboTwin-Clean SR38.0%58.8% / 60.9%
Inference Latency / chunk322.0 ms61.85 ms (~5.2× faster)
Video Backbone at InferenceRetained (~13.9 GB)Discarded (0 GB)
Max Training Batch / GPUbs=4 (bs=16 OOMs)bs=16
  1. Accuracy & latency: Relying on the video DiT alone lacks the VLM's instruction-grounding priors (-20.8 pts) and retains the 2B DiT and VAE at deployment (5.2× slower).
  2. Training memory: Using Cosmos as the sole backbone forces all 3 camera views (including both wrist views) through the video model, limiting batch size (bs=4, bs=16 OOMs). In contrast, World Tokens fuses all 3 views in the VLM and only models the primary view with Cosmos.

Q4. Alternative First-Frame Cues

We have not yet tested grayscale, Gaussian-blurred RGB, or a learned low-frequency encoder. Since Canny only provides basic spatial structure and is insensitive to threshold and smoothing choices (97.0% vs. 96.8%), we kept it here and leave learned cues as future work.

Q5. Training Compute of the 0.5B Adapter and Joint WM Fine-Tuning

The table below isolates the training cost of the 0.5B World Adapter and joint Cosmos-Predict2.5-2B fine-tuning on RoboTwin 2.0 (30k steps, global batch size 128, per-GPU bs=16):

Configuration (RoboTwin 2.0, 30k steps)Training ComputePeak VRAM / GPU
Baseline (Qwen3-VL-2B, action-only)70.7 GPU-h42.36 GiB
+ World Adapter (w/o WM, adapter-only)106.4 GPU-h51.09 GiB (+8.73 GiB)
+ Cosmos-2B (World Tokens, full)174.4 GPU-h79.15 GiB (+36.79 GiB)

All video-branch overhead is strictly at training time; at deployment, the 0.5B adapter adds only 10 ms per chunk over the baseline (62 ms vs. 52 ms).

Q6. Additional Results on RoboTwin 2.0 (W4)

Please see the General Response and Appendix J (Tables S4–S6) for our full RoboTwin 2.0 evaluation (50 bimanual tasks, 3 seeds × 50 trajectories/task), which we plan to integrate into the main text in the next revision. In summary, World Tokens improves in-domain success (55.6% w/o WM → 58.8% → 60.9% with cross-embodiment video), visual OOD generalization (8.7% → 28.8%), and unseen-task transfer (18.0% → 28.3% with same-embodiment video and 24.0% with cross-embodiment video).

Thank you for the response and the additional RoboTwin results, which address my concerns to some extent.

However, much of the evidence still comes from relatively simple benchmarks, and the RoboTwin results do not show a clear advantage over strong existing methods. I therefore maintain my original rating of 3.

Thank you for your follow-up and for engaging with our rebuttal. To clarify our RoboTwin 2.0 setup, because full joint Clean + Randomized action training requires substantial compute, we ran controlled Clean-only ablations (30k steps, 2B backbone, no embodied pretraining) specifically to isolate the gains from video supervision and actionless/cross-embodiment video co-training (+3.2 to +20.1 pts over matched controls), rather than optimizing absolute leaderboard scores with large-scale pretraining. We appreciate your constructive feedback and will incorporate these clarifications into the next revision.

Reviewer_bFYSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes World Tokens, which uses a pretrained video world model to supervise VLA representations during training and removes the video branch during inference. The method introduces a World Adapter with exclusive token conditioning. Experiments on LIBERO, LIBERO-Plus, SIMPLER, and real robots show improvements over VLA baselines.

Critical strengths

The idea of removing the video world model during inference is practical and helps reduce inference latency.

The paper provides reasonably comprehensive experiments, including simulation and real-world evaluation. The improvements on LIBERO-Plus are encouraging.

Critical weaknesses

Limited novelty. Using future prediction or world modeling as auxiliary supervision for VLA is not new. The main contribution seems to be the exclusive token interface and the edge-map conditioning. I am not fully convinced these architectural choices provide enough novelty for ICLR, especially compared with existing predictive VLA methods.

Insufficient evidence for the benefit of world modeling. In Table 2, removing world-model supervision still achieves 97.4%, compared to 98.2% for the full method. Although the difference is statistically significant, the improvement is relatively small considering the additional cost of jointly fine-tuning a large video diffusion model. More importantly, the paper does not report the w/o wm baseline on LIBERO-Plus or real-world tasks, where the claimed generalization benefits should be more convincing.

Limited real-world evaluation. The real-world experiments only include four fruit pick-and-place tasks, which are relatively simple and similar to each other. I would expect more complex tasks involving contact-rich manipulation, multi-stage reasoning, or long-horizon execution to demonstrate the usefulness of world modeling.

Questions for authors

Can the authors provide w/o wm results on LIBERO-Plus and real-world tasks? I think this is important to support the main claim.

How much additional training compute is required? Can the authors compare against simpler future prediction objectives instead of a full video diffusion model?

What is the main advantage of the proposed world tokens compared with existing predictive VLA representations, beyond the exclusive conditioning design?

2 comments
AuthorsOpenReview ↗

Response to Reviewer bFYS

We thank the reviewer for recognizing the practical value of removing the video world model at deployment and the encouraging results across our evaluations. Below we address Q1–Q3 (and W1–W3) in detail, referencing our new RoboTwin 2.0 experiments in the General Response.

Q1. w/o WM on LIBERO-Plus and Complex Benchmarks (W2 & W3)

We agree that the original LIBERO benchmark is near saturation, leaving limited room to observe the full benefit of world-model supervision (97.4% vs. 98.2%). To evaluate whether L_vid provides larger gains under distribution shifts and on harder tasks, we compared w/o WM (retaining the 0.5B World Adapter while removing video loss) against World Tokens on both LIBERO-Plus and RoboTwin 2.0:

Benchmarkw/o WMWorld Tokens+ Actionless Video
LIBERO-Plus (OOD robustness)75.1%76.8% (+1.7)78.4% (+3.3)
RoboTwin-Clean (50 bimanual tasks)55.6%58.8% (+3.2)60.9% (+5.3)
  • Complex manipulation beyond pick-and-place: To address the concern regarding task complexity (W3), RoboTwin 2.0 evaluates 50 diverse 14-DoF bimanual tasks involving coordinated two-arm control, articulated objects, and contact-rich manipulation. On this non-saturated benchmark, adding world-model supervision yields a clear +3.2 pt gain over w/o WM (58.8% vs. 55.6%). Moreover, because our framework can co-train on unlabeled videos, incorporating actionless videos further boosts visual OOD generalization on RoboTwin-Randomized (8.7% → 28.8%) and zero-shot transfer to 5 unseen tasks (18.0% → 28.3% with same-embodiment video and 24.0% with cross-embodiment video).
  • Real-world evaluation: Due to tight rebuttal timelines and physical robot scheduling constraints, we were unable to run an additional w/o WM checkpoint on the R1 Pro robot during the rebuttal window, and therefore leave this for future work.

Q2. Training Compute and Simpler Future Prediction Objectives

To quantify the training cost, the table below compares the action-only baseline and World Tokens on RoboTwin 2.0 under the unified 30k-step, batch-size-128 protocol:

Configuration (RoboTwin 2.0, 30k steps)Training ComputePeak VRAM / GPURoboTwin-Clean SR
Baseline (Qwen3-VL-2B, action-only)70.7 GPU-h42.36 GiB57.1% (StarVLA)
World Tokens (+ Cosmos-2B, full)174.4 GPU-h79.15 GiB58.8% / 60.9%

We also tested whether simpler future prediction objectives can replace the pretrained video diffusion model under the exact same RoboTwin-Clean setup:

  • Predicting future-frame VLM features (latent alignment): Aligning predicted tokens directly to future-frame VLM encodings (without a video diffusion model) achieves only 55.93%, showing almost no gain over w/o WM (55.6%).
  • Lightweight non-pretrained video DiT: Truncating the 28-block Cosmos DiT to 4 randomly initialized blocks causes the future-prediction loss to plateau early at 0.0942, compared to 0.0181 (~5.2× lower) for the pretrained 28-block model. Without large-scale video pretraining, a lightweight module fails to capture complex manipulation dynamics.
  • Comparison with WAMs and autoregressive VLAs: Existing WAMs often rely on even larger video backbones (e.g., 5B parameters) that slow down both training and inference, while autoregressive predictive VLAs also incur high inference latency. Using a 2B pretrained world model only during training preserves VLA-level inference speed while delivering much stronger policy gains.

Q3. Main Advantage Over Existing Predictive VLAs (W1)

  1. Inheriting pretrained multi-frame dynamics priors: Most existing predictive VLA representations predict a single future frame or align latent features without jointly fine-tuning a pretrained video world model. By co-training with Cosmos-Predict2.5-2B, our World Tokens rapidly fit task-specific scene evolution and inherit rich physical and temporal priors from large-scale video pretraining.
  2. WAM-like gains from unlabeled videos at VLA speed: Existing predictive VLAs are typically confined to paired action-labeled demonstrations. Because our exclusive token interface forces the video denoiser and action expert to share the same bottleneck, World Tokens inherits a core strength of WAMs—benefiting directly from unlabeled (actionless) videos across visual domains, unseen tasks, and even other robot embodiments—while keeping deployment latency at 62 ms.

Dear Reviewer bFYS,

Please respond to the author soon and engage in the discussion.

Best,

AC

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive feedback. Across the reviews, two shared concerns stood out: (1) the actual role and necessity of video prediction, and (2) the limited persuasiveness of ablations on LIBERO.

The core motivation of World Tokens is to combine the strengths of VLAs and World Action Models (WAMs), leveraging WAMs' ability to benefit from video predictive modeling. In this rebuttal, we ran controlled video-modeling experiments on RoboTwin 2.0 (50 bimanual tasks; Clean and Randomized settings). Because full joint action training on Clean + Randomized data requires substantial compute and training iterations, we conduct our controlled ablations strictly in the Clean action setting (using Randomized scenes for zero-shot OOD evaluation and actionless video co-training). All models use the same Qwen3-VL-2B backbone and training-only Cosmos-Predict2.5-2B video denoiser, are trained for 30k steps with batch size 128, and are evaluated across 3 seeds with 50 trajectories per task per seed:

  • Overall performance: 57.1% (StarVLA baseline) → 58.8% (World Tokens) → 60.9% with cross-embodiment video.
  • Visual generalization: 7.7% (StarVLA baseline) → 8.7% (World Tokens) → 28.8% with randomized-scene video.
  • Task generalization: 16.3% (StarVLA baseline) → 18.0% (World Tokens) → 24.0% on unseen tasks with cross-embodiment video.

1. Actionless Video Improves Overall Performance

MethodSR
StarVLA baseline57.1%
World Tokens w/o WM55.6%
World Tokens58.8%
+ Same-embodiment video60.8%
+ Cross-embodiment video60.9%

Evaluated on all 50 RoboTwin-Clean tasks on the target embodiment (aloha-agilex). Co-training with additional actionless videos improves in-domain success over the StarVLA baseline (57.1%), whether using same-embodiment Randomized-setting videos (60.8%) or cross-embodiment Clean-setting videos from 4 other robot arms (arx-x5, franka, piper, ur5, excluding aloha-agilex; 60.9%). All World Tokens variants share the same training budget and schedule, so the gain is not due to longer training. Removing the world model drops performance to 55.6%, confirming that the gain comes from world modeling rather than extra parameters.


2. Randomized-Scene Video Improves Visual Generalization

MethodSR
StarVLA baseline7.7%
World Tokens8.7%
+ Randomized-scene video28.8%

All action demonstrations come from the Clean setting only. Adding actionless videos with randomized appearance and layout raises success on RoboTwin-Randomized from 7.7% (StarVLA baseline) and 8.7% (World Tokens) to 28.8%, without any action labels from the randomized domain.


3. Actionless Video Improves Task Generalization

MethodUnseen 5 TasksSeen 45 Tasks
StarVLA baseline16.3%58.2%
World Tokens18.0%61.3%
+ Same-embodiment video28.3%60.2%
+ Cross-embodiment video24.0%60.7%

We train on 45 Clean tasks and hold out all action labels for 5 OOD tasks (open_microwave, place_fan, place_object_scale, stamp_seal, turn_switch), for which only actionless videos are added. Zero-shot success on unseen tasks rises to 28.3% with same-embodiment video, and to 24.0% even when the videos come from the 4 other embodiments (arx-x5, franka, piper, ur5), while seen-task performance is preserved (outperforming the 58.2% StarVLA baseline).


Overall, training-time world modeling lets the policy improve from unlabeled, and even cross-embodiment video — a form of scalability that action-only VLAs fundamentally lack. We have included the full RoboTwin 2.0 results and Canny layout-cue visualizations in the revised Appendix (Appendices I and J; Figure S5, Tables S4–S6), and plan to integrate the RoboTwin 2.0 experiments into the main text in the next revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.