acceptodds
Under review as a conference paper at ICLR 2027

What Limits Cross-Task Generalization in World Action Models?

43 views

Abstract

Cross-task generalization remains difficult for world action models (WAMs), even when their predicted futures depict plausible manipulation. This gap can be attributed directly: we make this gap directly testable by decoupling a WAM into a world model and an inverse-dynamics model (IDM), then holding the IDM fixed while replacing the source of future images. We introduce *Align Once, Evaluate Everywhere* (AoE), a camera-space inverse-dynamics model pretrained on large-scale robot data, which provides this recovery interface for different future sources across tasks and environments. Through it, we find that actions recovered from recorded futures of unseen tasks execute almost as well as replayed demonstrations, whereas futures predicted by world models collapse across benchmarks. The failures are not generic: they concentrate on the gripper and interaction geometry while generation capacity is spent on task-irrelevant background. This diagnosis suggests that supervision should target exactly these regions. We therefore introduce the *vision-to-action* (VITA) loss, which passes generated futures through the frozen AoE and penalizes discrepancies in the recovered actions, redirecting generation toward action-relevant detail. Across two world models and three manipulation benchmarks, this simple change raises the execution success of generated futures from near zero to 47.0%. The same loss also transfers to WAMs and yields consistent gains in native control.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotics, World Action Model, Cross-task Generalization, World Model, Tabletop Manipulation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_u7dGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks why world action models (WAMs) fail to transfer. Its methodological move is to factor a WAM into a world model and an inverse-dynamics model (IDM), freeze the IDM, and then swap only the source of future frames, giving three conditions under one execution protocol: replay the recorded actions, recover actions from recorded frames, and recover actions from generated frames. The IDM, AoE, is a camera-space model trained on 9,637 h of robot, human and simulated video; it predicts a 12-D target per arm (start and end UV, relative depth, 6-D rotation, gripper) which is turned into an executable pose at run time using the robot state and camera calibration.

With that setup the authors report that recorded frames are largely recoverable (57.0% average on LIBERO against 72.3% for action replay) while generated futures are essentially dead: 0.3–0.4% at pass@1, 4.6–5.9% at pass@50. The blinded study in Appendix G is more interesting: 77% of Cosmos's futures are judged plausible and 67.5% as looking like the task was completed, yet 0.5% of them execute.

They then add VITA, a training-time loss that sends the differentiably decoded generated future through the frozen AoE under the same action noise and flow time as the recorded future, and penalises both the mismatch to the teacher's response on the recorded future and the mismatch to the ground-truth action-flow target, with a small gripper-ROI reconstruction term. That takes generated futures to 47.0% (WAN2.2) and 51.7% (Cosmos) on LIBERO and to about 30% on LIBERO-Plus. The same loss is applied to LingBot-VA and FastWAM, where the gains are smaller but still there, and the largest of them arrive out of distribution.

The appendix also has a scaling study on AoE's pretraining data, ablations on the camera-space representation and on the individual loss terms, the blinded human study, and the local repair and offset experiments.

Critical strengths

The measurement is the contribution here, not the loss. I came in sceptical of yet another "IDM as evaluator" setup, since the IDM is usually the thing you should be worried about, and this paper spends most of its appendix budget trying to kill exactly that worry. Reconstructed real frames still execute at 91.3%, so it is not the VAE or the image statistics. An independent geometric recovery pipeline also collapses on generated futures (3.0%), so it is not an AoE artefact. The blinded study separates "looks plausible" from "executes" cleanly. And the intervention experiments are the part I find most convincing: a 4-pixel offset of the interaction region in real video drops execution from 75.5% to 23.0% while 86% of the videos still look plausible, and repairing the interaction region of generated segments lifts success from 1.5% to 32.0% against 3.0% for background repair. That is a causal story about where the information is missing, not a correlation with a metric. As a side note, Table 17 is the best argument I have seen for why SSIM and LPIPS are the wrong selection criterion here: AoE action agreement gets AUROC 0.79 against 0.55 to 0.61 for the three image metrics.

VITA itself is a modest idea. It distils the teacher's response on generated frames toward its response on real frames, plus an action anchor, and I would not have expected that to be enough. The evidence that it transfers is better than I expected, in particular the evaluator swap in Appendix H: the same checkpoint goes from 0.5 to 29.9 with AoE, from 1.8 to 26.8 with J-IDM, and from 0.9 to 30.8 with DreamGen-IDM. The controls in Table 20 also matter, because "any extra auxiliary loss helps" is the obvious alternative explanation, and frozen DINO alignment plus a matched-budget native run do not reproduce the gain. The FastWAM result is the one that made me take the mechanism seriously: it never renders a future at inference, and its held-out cross-task success still doubles (11.2 to 23.6).

Critical weaknesses

My main worry is about what the headline number is measured against. Cosmos is adapted for 10,000 iterations, and the config in the supplement initialises from the 14B GR00T checkpoint with per-GPU batch 1, not the 4 the table says. WAN2.2's training recipe is never given anywhere: Appendix F.2 and Table 11 describe Cosmos only, and the supplement has no WAN2.2 world-model code at all, only the Wan2.2 VAE and DiT vendored inside FastWAM. Worse, there is not a single generated-video quality number in the paper. FVD is mentioned once, in Appendix G.2, as being "recorded separately at the model/checkpoint level", and no value appears. So a reader cannot fully separate "the world model produces plausible futures whose geometry is wrong" from "the world model was adapted too briefly to carry usable motion, and an action-supervised auxiliary loss rescues it". I think the human study and the VAE control make the second story unlikely for Cosmos, and I weight them a lot, but the title and abstract make a claim about WAMs in general from two models whose adaptation budgets are short or unknown. I would like either FVD/PSNR plus one longer or stronger baseline, or a narrower claim.

The second problem is that the paper is not careful about noise. There are no seeds, error bars or confidence intervals anywhere. RoboTwin has 25 tasks per fold, and several of the deltas the paper leans on sit right at the level where this matters: +0.5 for LingBot-VA on LIBERO, +1.1 for FastWAM in distribution. I know another round of full evaluation is expensive at this scale, but even two seeds on the LIBERO tables or a stated episode count per RoboTwin cell would let me tell a real effect from a lucky run. Table 6 also has two clearly negative cells (FastWAM noise −10.3, camera −2.2) which are printed and then never mentioned, under a sentence claiming "consistent gains". The pass@k accounting is lopsided in the same direction: baselines get pass@1, pass@50 and a full candidate sweep in Table 18, while the VITA rows in Tables 3 and 4 give one number with no k and no sweep. Table 16 labels the same checkpoints pass@1, which suggests this is fine, but then the main tables should say so, because otherwise "from near zero to 47.0%" is comparing a model change with an unstated sampling budget.

Smaller things, roughly in the order they bother me. Table 7 puts pretrain loss and validation RMSE next to each other for two representations in different units, which produces the odd reading that the 3D-pose model has a lower pretrain loss (0.026 vs 0.047) and a higher RMSE (0.146 vs 0.070); the downstream column is the only one that carries the argument, and the rest should be dropped or normalised. Table 3's caption is split by the table header, with "the checkmark indicates that the AoE is" above the header and the rest of the sentence below it. §3.1 says AoE works "with little data from a downstream environment", which contradicts the zero-shot protocol used everywhere else. The analysis in §3.3 and Appendix E is fine as algebra but does not do any work: Proposition 1 is completing the square, Propositions 2 and 3 are Taylor expansion and the chain rule, and the bound in Eq. (10) has unspecified local constants, an unmeasured teacher error, and a hypothetical reference feature whose decoded future is the recorded one. Its key assumption, Eq. (31), is basically the transfer property you would want to test. No experiment estimates any of the constants or checks the bound. I would either present this as intuition or pick one measurable quantity and report it. Finally, the strongest WAM result in the paper, FastWAM going from 11.2 to 23.6 on held-out tasks, only appears in Table 20, while the main text shows 97.6 to 98.7, where the effect is close to the ceiling.

Questions for authors
  1. What k and what reset policy are used for the VITA rows in Tables 3 and 4, and for the WAM tables? The baselines get a full pass@k sweep in Table 18; I would like the same curve for one VITA checkpoint. If 47.0 and 51.7 are pass@1, this question goes away, but please state it.

  2. What is WAN2.2's training configuration (initialisation, data, global batch, iterations, hardware), and why is it missing from the supplement? Relatedly, can you report FVD and PSNR/SSIM/DINO between generated and recorded futures for both world models, in distribution and cross-task, and add one longer-trained or otherwise stronger world model so that the "collapse" can be attributed to the content of the futures rather than to a short adaptation?

  3. 722 h of the 9,637 h corpus is human ego video (EgoDex), but the abstract calls it robot image–action pairs. How is a human-hand episode mapped to the 12-D per-arm target, specifically the gripper coordinate? The released code uses thumb–index tip distance in metres, while the robot datasets supply a binary command; given that the paper says the gripper is where the failure lives, could mixing those two conventions into one coordinate be part of the story? And does the corpus contain any episodes or assets from LIBERO, RoboTwin or RoboCasa scenes, and how was that checked?

  4. AoE is both the VITA teacher and the evaluator, and on recorded images it is the weakest of the three IDMs you report (47.0 against 55.5 for J-IDM and 54.8 for DreamGen-IDM); on RoboTwin-Hard with VITA futures it reports 40.4 against 66.7 for J-IDM. Can you run VITA with a teacher other than AoE, for example J-IDM trained on the source tasks, and see whether the gain survives? That would separate "better futures" from "futures that match this particular teacher".

  5. What temporal interval separates the frame pairs AoE consumes at execution, and is it the same for recorded and generated video? I could not work out what ties a per-pair displacement to one controller step, and if the generated video has a different effective motion scale than the recorded one, that alone could produce a near-zero success rate. Also, what is the wall-clock and memory cost of the VITA step (1.14 B frozen teacher on 8-frame chunks every second optimiser step, differentiable VAE decode and VJP, teacher offload) relative to native training?

  6. The LIBERO-Plus "Noise" column is 0.0% for recorded-image replay with AoE and 0.1% with VITA. Is that a property of the futures or of AoE's sensitivity to sensor noise? Should that column be read as part of the cross-environment comparison, or as a failure mode of the evaluator?

4 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer u7dG (part 1 / 3)

We sincerely thank Reviewer for the exceptionally thorough reading, for highlighting the strength of our causal intervention and evaluator-swap experiments, and for the constructive questions. We have addressed all six questions and every methodological and presentation concern in Appendix J (J.1–J.6, Tables 26–36), Section 4.4 (Table 7), Section 5, and the updated Supplementary Material (code/).

1. Pass@kk Protocol, Reset Policy & Full Candidate Sweep for VITA (Q1 & W2)

  • All VITA numbers in Tables 3 and 4 (including 47.0% for WAN2.2 and 51.7% for Cosmos) are strictly pass@1 (one sampled future per initial state). Native WAM tables (Tables 5 and 6) report single closed-loop rollouts under each model's standard replanning schedule (Appendix J.2, lines 1961–1965).
  • Full pass@k sweep for VITA (Appendix J.2, Table 30 & code/evaluation/pass_at_k.py): Using the exact same shared reset states across candidates (2,0002,000 initial states on LIBERO), WAN2.2 + VITA + AoE scales from 46.95% (pass@1) to 65.40% (pass@5), 69.45% (pass@10), 72.90% (pass@25), and 74.55% (pass@50), compared with 0.30% -1.20% -1.90% -3.30% -4.55% for WAN2.2 + AoE without VITA.

2. WAN2.2 Recipe, Code Supplement, Video Metrics (FVD/PSNR/SSIM/DINO) & Stronger/Longer Baselines (Q2 & W1)

  • Code Supplement & WAN2.2 Configuration (Appendix J.1, Table 28 & Appendix J.2): Our initial code supplement extracted minimal dependencies and set a per-GPU batch of 1 in the example script to run on smaller GPUs, whereas the reported Cosmos runs used a per-GPU batch of 4 on 8-node H800-141GB GPUs (Table 12 & Table 28). We apologize for omitting the WAN2.2 configuration table earlier (WAN2.2 required less VRAM and was run on a separate server cluster). We have now added the full training recipe in Table 28 (Wan2.2-TI2V-5B, 32 H800 GPUs, dataloader batch 2, effective global batch 32, resolution 480×832480 \times 832, 81 frames, LR 2×10−52 \times 10^{-5}, 10,000 steps) and included the complete code for WAN2.2 (code/aoe_guided_wan22/), Cosmos (code/aoe_guided_cosmos/), and Cosmos3-Nano-Policy-DROID (code/aoe_guided_cosmos3_droid/) in the revised supplement. Full runnable code and pretrained checkpoints will be open-sourced upon acceptance.
  • Video Quality Metrics & 3×3\times Longer Native Training (Appendix J.2, Table 29 & code/evaluation/video_quality.py): We evaluated FVD, PSNR, SSIM, and DINO cosine similarity alongside Pass@1 execution across 2,000 clips per condition on LIBERO in-distribution (ID) and cross-task (XT) splits, including a 3×3\times longer native Cosmos run (30k steps):
SplitCheckpointFVD ↓\downarrowPSNR ↑\uparrowSSIM ↑\uparrowDINO ↑\uparrowPass@1 (%) ↑\uparrow
in-distributionCosmos, native (10k)11823.4.804.86428.5
in-distributionCosmos, longer native (30k)10124.0.819.87736.0
in-distributionCosmos + VITA (10k)9424.4.833.88468.0
in-distributionWAN2.2, native (10k)12822.8.791.85226.0
in-distributionWAN2.2 + VITA (10k)10223.8.822.87463.0
cross-taskCosmos, native (10k)15721.2.751.8210.4
cross-taskCosmos, longer native (30k)14021.8.769.8365.5
cross-taskCosmos + VITA (10k)12622.2.788.84851.7
cross-taskWAN2.2, native (10k)16520.8.738.8130.3
cross-taskWAN2.2 + VITA (10k)13221.8.777.84147.0

Tripling native training (30k steps) improves XT FVD (157→140157 \rightarrow 140), yet cross-task execution only reaches 5.5% (vs. 51.7% for Cosmos + VITA at 10k steps).

  • Stronger Pretrained Backbone (Cosmos3-Nano-Policy-DROID): We also evaluated Cosmos3-Nano-Policy-DROID on LIBERO and RoboTwin (Appendix J.1, Table 27) and on a physical robot (Section 4.4, Table 7). Without VITA, it still collapses zero-shot on unseen tasks (0.5% on LIBERO, 1.8% / 1.2% on RoboTwin, 4.0% on unseen real-world tasks); with VITA, it reaches 55.6% on LIBERO, 61.8% / 40.2% on RoboTwin Easy/Hard, and 14.7% on unseen real-world tasks.
AuthorsOpenReview ↗

Rebuttal for Reviewer u7dG (part 2 / 3)

3. Multi-Seed Evaluation, Episode Accounting & Why Noise and Camera Degrade (W2 & Q6)

  • Seeds & Episode Accounting (Appendix J.3, Table 33 & Appendix J.4, Table 32): Table 33 details the exact RoboTwin accounting (25 held-out tasks/fold ×\times 2 configs ×\times 50 trials = 2,500 trials per fold per condition). Table 32 reports an independent second training seed (Seed 1) and paired bootstrap 95% confidence intervals on LIBERO: LingBot-VA ID (98.5% -99.0% Seed 0; 98.6% -98.9% Seed 1), FastWAM ID (97.6% -98.7% Seed 0, 95% CI [+0.4, +1.8]; 97.7% -98.4% Seed 1), and FastWAM Held-out tasks (11.2% -23.6% Seed 0, 95% CI [+10.1, +14.7]; 11.6% -22.9% Seed 1).
  • Why Noise (-10.3 on FastWAM; 0.0%/0.1% on WM+AoE) and Camera (-2.2 on FastWAM) occur (Section 5 & Appendix J.4): We apologize for not explaining these columns earlier and have added a dedicated discussion in Appendix J.4 (lines 2030–2039) and Section 5:
  • Noise (0.0% / 0.1% & -10.3): LIBERO-Plus Noise applies heavy image blurring/corruption that obscures the robot gripper. Vision-based IDMs cannot recover actions when the visual input lacks identifiable gripper geometry—even on ground-truth recorded videos under Noise (Table 24), AoE gets 0.0%, J-IDM 0.9%, and DreamGen-IDM 3.6%. Inspecting the single successful rollout (0.1% in Table 4) confirmed it was an incidental random completion. Consequently, Noise reflects a visibility boundary of visual IDMs; enforcing auxiliary supervision on blurred frames where the teacher cannot localize the gripper injects noisy gradients, causing the -10.3 drop on Noise for FastWAM.
  • Camera (-2.2): LIBERO-Plus Camera perturbs camera extrinsics. Because AoE predicts in camera space and relies on calibrated extrinsics, uncalibrated extrinsic shifts move the arm to out-of-distribution image coordinates during auxiliary supervision.

4. EgoDex Hand Mapping, Grasp Masking Ablation, Asset Overlap & Ethics Note (Q3)

  • Human Hand Mapping & Grasp Masking Ablation (Appendix J.6, Table 36 & code/canonical_action/data/egodex2aligned.py): Each wrist is transformed into camera coordinates via (Tc,tw)−1Th,tw(T_{c,t}^w)^{-1}T_{h,t}^w, and the 12th coordinate measures thumb–index tip pinch aperture de=∥pthumb tip,ew−pindex tip,ew∥2d_e = \|p_{\text{thumb tip},e}^w - p_{\text{index tip},e}^w\|_2 in meters. To test whether mixing continuous human pinch distance with binary robot gripper commands hurts gripper learning, we pretrained an AoE variant with all human grasp targets masked out (Table 36): held-out robot grasp open/close balanced accuracy (88.6% vs. 89.0%) and LIBERO execution on recorded (56.8% vs. 57.0%) and generated futures (0.4% vs. 0.3%) are virtually identical.
  • Corpus Overlap, Raters & License (Appendix J.6, lines 2093–2114): All held-out LIBERO evaluation tasks and held-out RoboTwin fold tasks are excluded prior to AoE pretraining. As stated in Table 10 and Appendix J.6, the corpus includes 122.1h of RoboCasa, so our cross-task claims concern unseen task identities and demonstrations rather than asset-level novelty. We also clarified that the blinded study used 6 raters (first author + 5 external contributors, 3 raters/video) and cited the EgoDex CC BY-NC-ND license.

5. Training VITA with an Alternative Teacher (Source-Trained J-IDM, Q4)

In Appendix J.3 (Table 31), we trained WAN2.2 on the same source-task split using a source-trained J-IDM as the VITA teacher and evaluated all checkpoints across AoE, J-IDM, and the completely independent DreamGen-IDM evaluator:

Training SupervisorAoE Evaluator (Zero-shot, 7 cats)J-IDM Evaluator (In-dist., 6 cats)DreamGen-IDM Evaluator (In-dist., 6 cats)
None (Native WAN2.2)0.5%1.8%0.9%
AoE (Zero-shot cross-task teacher)29.9%26.8%30.8%
Source-trained J-IDM (In-domain teacher)27.0%28.1%27.8%

Under the neutral DreamGen-IDM evaluator (which supervised neither model), Source-trained J-IDM supervision boosts execution from 0.9% to 27.8%, confirming that action-grounded supervision transfers across teachers. Crucially, Source-trained J-IDM is trained in-domain on the target environment's source tasks (as 3D-pose IDMs do not transfer across environments), whereas our AoE teacher is strictly zero-shot cross-task yet achieves even stronger transfer (30.8% under DreamGen-IDM).

AuthorsOpenReview ↗

Rebuttal for Reviewer u7dG (part 3 / 3)

6. Temporal Interval, Motion Scale & Training Overhead (Q5)

  • Replay Timing & Motion Scale (Appendix J.5, Table 35): Recorded and generated clips share the exact same temporal grid: f=20 Hzf=20\text{ Hz} effective frame rate, s=5s=5 frames separation (250 ms250\text{ ms} per pair), and h=20 Hzh=20\text{ Hz} controller rate (hs/f=5hs/f = 5 control ticks per pair). Median recovered translation per pair on the matched time grid shows that native generated futures exhibit attenuated motion amplitude (1.4 cm on LIBERO, 1.7 cm on RoboTwin vs. 2.1 cm and 2.5 cm for recorded videos), while VITA restores the displacement to 1.9 cm and 2.2 cm.
  • Wall-Clock & Memory Cost (Appendix J.5, Table 34 & code/evaluation/training_cost.py): On 32 H800 GPUs, adding VITA changes step time and peak GPU memory from 25s -32s (1.28×1.28\times) and 62 -72 GiB on Cosmos, 18s -21s (1.17×1.17\times) and 55 -62 GiB on LingBot-VA, and 2.4s -4.1s (1.71×1.71\times) and 49 -60 GiB on FastWAM, with zero inference-time cost for WAMs.

7. Minor Clarifications (Table 8 [formerly Table 7], §3.1, §3.3 & Formatting)

  • Table 8 Pretrain Loss vs. Valid RMSE: Beyond differing coordinate units (Appendix J.6), our pretraining corpus is assembled incrementally across datasets; when environment-specific data is limited, the 3D-pose IDM overfits to training coordinate frames (lower training loss 0.026 vs. 0.047 but higher validation RMSE 0.146 vs. 0.070 and 0.0% cross-task transfer), whereas scaling diverse data in camera space resolves this overfitting.
  • We fixed the split caption in Table 3, clarified the zero-shot vs. 10-shot protocol distinction in §3.1 and Appendix J.6, and framed §3.3/Appendix E explicitly as a local sensitivity analysis corresponding to the empirical offset/repair interventions in Figure 11.

As the discussion period is drawing to a close, we kindly invite you to share any remaining questions or concerns about AoE so that we can address them before the deadline. We look forward to your valuable feedback. : )

Best wishes,

Authors of Submission #13777

Reviewer_asNWOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper finds that robot world models fail across tasks not mainly because the generated videos look bad overall, but because the model spends its compute on irrelevant details like backgrounds, while failing to accurately render the gripper, objects, and contact positions. Based on this observation, the authors propose AoE, which converts future videos into executable actions. Meanwhile, using a frozen AoE as a supervisor, VITA directs the world model to focus on action-relevant details.

Critical strengths

1.This paper addresses an important practical problem in robot manipulation: the gap between visually plausible future predictions and physically executable robot actions. The proposed VITA loss significantly improves the execution success rate of generated future videos from near zero to over 50% on LIBERO, while also improving the native control performance of world-action models, especially under distribution shift. These results clearly demonstrate the practical value of the work and the effectiveness of the method.

2.The paper conducts extensive experiments across multiple benchmarks, world models, and world-action models. The authors systematically perform ablation studies, cross-task and cross-environment evaluations, and further analyze the relationship between visual plausibility and action executability. These experiments provide strong support for the proposed method and offer deeper insight into the limitations of existing world models.

Critical weaknesses

1.The experiments are mainly conducted on simulation benchmarks such as LIBERO and RoboTwin. While the results show significant improvements in task success rates in simulation, performance on real robots remains unclear. Adding real-robot experiments would help further validate the practical value of the method.

2.VITA encourages generated futures to match the action-flow predictions of a frozen inverse dynamics model. However, agreement in action representations does not necessarily guarantee physically consistent object states or contact dynamics. Further analysis would help establish when action-based alignment is a reliable proxy for executable and physically plausible futures.

Questions for authors

My question is the same as the issue raised in the weaknesses.

4 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer asNW (part 1 / 2)

We thank Reviewer for the positive assessment and for highlighting the practical value and systematic ablations of our work. In the revised manuscript and supplementary material, we have directly addressed both of your points by adding real-robot cross-task evaluations (including deformable and fluid manipulation tasks in Section 4.4, Figure 4, Table 7, and rollout videos in Supplementary Material: ICLR2027_VITA_real_world_exp_demo_upload_partial_1/) and quantitative analyses of action agreement vs. object-state transitions and foundational world-model priors (Appendix J.1, Tables 26–27, and Section 5 Limitations).

1. Real-World Robot Validation (Weakness 1)

We added real-world manipulation experiments in Section 4.4 (Figure 4, Table 7) with detailed scoring criteria in Appendix J.1. We fine-tuned both LingBot-VA (native action head) and Cosmos3-Nano-Policy-DROID (decoupled WM + frozen AoE) on 300 demonstrations of in-domain Tabletop Tidying, and evaluated them on Tabletop Tidying and three unseen out-of-domain tasks—Tissue Extraction (deformable object), Water Pouring (fluid transfer), and Bottle Sorting (multi-object spatial arrangement)—over 25 trials per task per model (100 trials per model in total).

Method[In-Domain] Tabletop Tidying[Cross-Task] Tissue Extraction[Cross-Task] Water Pouring[Cross-Task] Bottle SortingCross-Task Overall (Unseen Avg.)
Cosmos3-Nano-Policy-DROID77.6 / 60.028.0 / 4.015.0 / 0.026.2 / 8.023.1 / 4.0
Cosmos3-Nano-Policy-DROID + VITA (Ours)83.2 / 72.0 (+5.6)47.4 / 12.0 (+19.4)26.5 / 8.0 (+11.5)52.4 / 24.0 (+26.2)42.1 / 14.7 (+19.0 / +10.7%)
LingBot-VA85.6 / 76.07.0 / 0.036.5 / 12.020.7 / 8.021.4 / 6.7
LingBot-VA + VITA (Ours)91.2 / 84.0 (+5.6)14.5 / 0.0 (+7.5)45.5 / 20.0 (+9.0)47.6 / 24.0 (+26.9)35.9 / 14.7 (+14.5 / +8.0%)

Note: Entries report Mean Completion Score / Binary Success Rate (%). Cross-Task Overall averages over the three unseen tasks (75 trials per model). Real-world rollout videos are included in the supplementary material under ICLR2027_VITA_real_world_exp_demo_upload_partial_1/. Across the three held-out real-world tasks, VITA raises cross-task execution success from 4.0% to 14.7% for Cosmos3-Nano-Policy-DROID (mean cross-task score from 23.1 to 42.1, +19.0) and from 6.7% to 14.7% for LingBot-VA (mean cross-task score from 21.4 to 35.9, +14.5), confirming that the simulation gains transfer to physical hardware.

AuthorsOpenReview ↗

Rebuttal for Reviewer asNW (part 2 / 2)

2. When is Action Alignment a Reliable Proxy for Object States & Contact Dynamics? (Weakness 2)

We completely agree with your insight: matching action representations constrains the relative interaction geometry between the gripper and the target object, while post-contact object states (e.g., deformable sheet separation, fluid flow, and grasp stability) also rely on the foundational physical perception of the underlying world model. We substantiate this relationship through three complementary analyses in the revised paper:

  • Paired Analysis of Action Agreement vs. Object-State Transitions (Appendix J.1, Table 26): To directly test how action agreement relates to object-state consistency, we partitioned 200 held-out Cosmos + VITA simulation rollouts by the median AoE action-agreement score and paired human judgments of visible object-state transitions (+/-) with actual simulator execution outcomes (+/-):
AoE Agreement StratumTrials(+, +) Valid Object Transition & Exec. Success(+, -) Valid Object Transition, Exec. Fail(-, +) Invalid Transition, Exec. Success(-, -) Both Fail
High Agreement11069 (62.7%)23 (20.9%)5 (4.5%)13 (11.8%)
Low Agreement9024 (26.7%)46 (51.1%)6 (6.7%)14 (15.6%)
  1. Why action alignment is necessary: In the Low Agreement group, even when the world model visually depicts an apparent object transition (24 + 46 = 77.8%), 46 of 70 trials fail physical execution ((+, -)) because imprecise pre-contact and contact geometry causes missed grasps or slips. High AoE agreement raises successful execution (+, +) from 26.7% to 62.7% and achieves a 0.79 within-task AUROC for predicting execution success (Appendix G.2, Table 18), far outperforming pixel/semantic metrics (SSIM: 0.55, DINO: 0.61) because AoE jointly attends to both the end-effector and the manipulated object (Appendix A, Figure 6; Appendix G.2, Figure 11).
  2. Where action alignment hits its boundary: Even in the High Agreement stratum, 20.9% (23/110) of trials fall into (+, -) and 11.8% (13/110) into (-, -), where valid kinematic approach is insufficient to overcome post-contact instability or inaccurate object-state physics.
  • Evidence from Real-World Deformable (Tissue Extraction) and Fluid (Water Pouring) Tasks (Section 4.4, Table 7): Our real-world tasks specifically probe non-rigid and physics-sensitive dynamics. On Tissue Extraction (which requires pinching a flexible tissue sheet, pulling it, and fully separating it from the dispenser; see 2026_11_07_15_29_tissue _extraction_score_85_success_0.mp4), VITA doubles LingBot-VA's completion score from 7.0 to 14.5 (+7.5) by guiding the gripper to accurately pinch and pull the exposed sheet, yet its full-extraction success remains 0.0% due to the base model's limited representation of deformable separation. In contrast, when pairing VITA with Cosmos3-Nano-Policy-DROID, which carries stronger real-world physical priors from DROID pretraining, the completion score jumps from 28.0 to 47.4 (+19.4) and full extraction succeeds in 12.0% of trials (and Water Pouring rises by +8.0% in both models).
  • Controlled Backbone Comparison in Simulation (Appendix J.1, Table 27): Holding the frozen AoE teacher and VITA loss fixed while upgrading the world model from standard Cosmos to Cosmos3-Nano-Policy-DROID further improves zero-shot cross-task execution from 51.7% to 55.6% on LIBERO (with long-horizon multi-step LIBERO-Long rising from 23.2% to 28.0%) and from 57.7% / 36.4% to 61.8% / 40.2% on RoboTwin Easy/Hard. We have also explicitly documented this boundary in Section 5 Limitations.

Thank you for the detailed responses and additional experiments. The newly added real-world evaluations, together with the quantitative analyses of action consistency and object-state transitions, substantially strengthen the experimental evidence. In particular, these results provide valuable evidence for VITA's effectiveness in real-world robotic tasks and further clarify the complementary roles of action alignment and the physical priors of the underlying world model. I appreciate the authors' efforts in addressing my concerns. However, I think that the overall motivation and central insights of the paper could still be further strengthened. I will maintain my original rating.

Thank you for your prompt response. We are very glad that our clarification fully addresses your concerns, and we sincerely appreciate the valuable questions you raised regarding AoE. : )

Best wishes,

Authors of Submission #13777

Reviewer_nZoTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how world model and world-action model performance is affected by the quality of their imagination. The results indicate that the effect is significant and for this, the paper introduces an AoE IDM model. The AoE model is then integrated into WM, where it improves read-out of actions from generated videos and into WAMs where it improves performance especially in out-of-distribution tasks. The method improves model performance in simulated benchmarks, but its real-world validation on a real robot is missing.

Critical strengths

The main critical strength is a novel loss that integrates a pre-trained external IDM model into WM/WAM training and clearly improves the performance of the multiple WM/WAM models on LIBERO and RoboTwin2.0 benchmarks. The method is thoroughly ablated and studied, giving understanding of how each of the components affects the final performance.

Critical weaknesses

The paper has two critical weaknesses.

First is a missing real-world evaluation of the WAM models on the real robot. Without this it is impossible to understand how the real world environments affect the method, whether its gains persist in the real-world or they are artifacts of simplified LIBERO and RoboTwin environments.

Second, the paper lacks baselines such as DreamZero or Cosmos3-DROID, that feature WAM with a strong pre-training on large dataset, coupled with the standardized robotic platform (DROID). Without this, it is hard to understand whether the questions asked in this paper are generic to most WAMs, or, are caused by the LIBERO/RoboTwin environments.

Questions for authors
  1. Was real-world robotic evaluation run with this method? Were gains observed in the simulated environments also observed in real-world environments?
  2. Is it possible that imagination problems are caused by the small scale of the training data, and with larger datasets (DROID, ABC) the issues will disappear?
6 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer nZoT (part 1 / 3)

We thank Reviewer for the constructive feedback and for recognizing the novelty of integrating our pretrained AoE inverse-dynamics model via the VITA loss as well as our thorough ablations. Following your suggestions, we have added real-world robot evaluations (Section 4.4, Figure 4, Table 7, and Appendix J.1) and evaluated the exact DROID-pretrained baseline (Cosmos3-Nano-Policy-DROID) the reviewer suggests across both real-world and simulated cross-task benchmarks (Table 7 and Appendix J.1, Table 27). We have also included the real-world rollout videos (ICLR2027_VITA_real_world_exp_demo_upload_partial_1/) and the complete training/evaluation code for Cosmos3-Nano-Policy-DROID (code/aoe_guided_cosmos3_droid/) in the updated Supplementary Material.

1. Real-World Robotic Evaluation (Weakness 1 & Question 1)

We added a dedicated real-world evaluation in Section 4.4 (Figure 4, Table 7) and detailed the scoring protocol in Appendix J.1. We fine-tuned both LingBot-VA (using its native action head) and Cosmos3-Nano-Policy-DROID (recovering actions from generated future videos via the frozen AoE) on 300 demonstrations of in-domain Tabletop Tidying, and evaluated them on the seen task (Tabletop Tidying) as well as three unseen out-of-domain manipulation skills: Tissue Extraction, Water Pouring, and Bottle Sorting (25 trials per task per model; 100 trials per model in total).

Method[In-Domain] Tabletop Tidying[Cross-Task] Tissue Extraction[Cross-Task] Water Pouring[Cross-Task] Bottle SortingCross-Task Overall (Unseen Avg.)
Cosmos3-Nano-Policy-DROID + AoE77.6 / 60.028.0 / 4.015.0 / 0.026.2 / 8.023.1 / 4.0
Cosmos3-Nano-Policy-DROID + VITA (Ours)83.2 / 72.0 (+5.6)47.4 / 12.0 (+19.4)26.5 / 8.0 (+11.5)52.4 / 24.0 (+26.2)42.1 / 14.7 (+19.0)
LingBot-VA85.6 / 76.07.0 / 0.036.5 / 12.020.7 / 8.021.4 / 6.7
LingBot-VA + VITA (Ours)91.2 / 84.0 (+5.6)14.5 / 0.0 (+7.5)45.5 / 20.0 (+9.0)47.6 / 24.0 (+26.9)35.9 / 14.7 (+14.5)

Note: Table entries report Mean Completion Score / Binary Success Rate (%). Cross-Task Overall averages exclusively over the three unseen out-of-domain tasks (75 trials per model).

  • Consistent Real-World Gains: Just as in simulation, VITA consistently improves real-world execution across both the decoupled WM+AoE path and the native WAM action head. On the in-domain task, VITA improves completion scores by +5.6 points for both models (raising success rates from 60.0% to 72.0% and 76.0% to 84.0%). On the three unseen out-of-domain tasks (75 trials), VITA more than triples the cross-task success rate of Cosmos3-Nano-Policy-DROID from 4.0% to 14.7% (boosting average cross-task score from 23.1 to 42.1, +19.0) and raises LingBot-VA from 6.7% to 14.7% (boosting average cross-task score from 21.4 to 35.9, +14.5).
  • Real-World Rollout Videos: Representative rollout videos across all four tasks are provided in the supplementary material under ICLR2027_VITA_real_world_exp_demo_upload_partial_1/.
AuthorsOpenReview ↗

Rebuttal for Reviewer nZoT (part 2 / 3)

2. Evaluation on DROID-Pretrained Models & The Role of Data Scale (Weakness 2 & Question 2)

Evaluating Cosmos3-Nano-Policy-DROID across benchmarks: To investigate whether strong pretraining on large-scale embodied datasets alleviates the imagination-to-execution gap, we evaluated Cosmos3-Nano-Policy-DROID on both our real-world setup (Table 7 above) and the cross-task LIBERO and RoboTwin 2.0 benchmarks (Appendix J.1, Table 27; implementation provided in code/aoe_guided_cosmos3_droid/). In simulation, we evaluate both its native IDM and the frozen AoE to examine whether our findings hold across action-decoding pathways.

Image SourceAction DecoderLIBERO SpatialLIBERO ObjectLIBERO GoalLIBERO LongLIBERO Avg. (%)RoboTwin Easy (%)RoboTwin Hard (%)
Dataset image (Replay)AoE71.674.053.029.257.065.360.7
Cosmos (Zero-shot)AoE0.40.60.40.00.41.61.1
Cosmos3-Nano-Policy-DROID (Zero-shot)AoE0.40.80.60.20.51.81.2
Cosmos3-Nano-Policy-DROID (Zero-shot)Native IDM0.81.20.20.00.62.10.9
Cosmos + VITA (Zero-shot)AoE63.679.240.623.251.7 (+51.3)57.7 (+56.1)36.4 (+35.3)
Cosmos3-Nano-Policy-DROID + VITA (Zero-shot)Native IDM41.683.438.09.643.2 (+42.6)58.9 (+56.8)43.3 (+42.4)
Cosmos3-Nano-Policy-DROID + VITA (Zero-shot)AoE67.082.245.228.055.6 (+55.1)61.8 (+60.0)40.2 (+39.0)

“Zero-shot” denotes evaluation on unseen tasks after downstream fine-tuning on the training tasks. Parenthesized values indicate percentage-point improvements over the corresponding baseline using the same model initialization and action decoder.

  • Strong embodied pretraining alone does not eliminate the gap under our downstream adaptation setting. Despite its large-scale DROID pretraining, Cosmos3-Nano-Policy-DROID achieves only 0.5% average cross-task success on LIBERO with AoE and 0.6% with its native IDM after standard downstream fine-tuning. Performance is similarly low on RoboTwin: 1.8% / 1.2% with AoE and 2.1% / 0.9% with the native IDM on Easy/Hard. The low performance therefore occurs with both action decoders, rather than only with the external AoE. In the real-world evaluation, the AoE-based pathway likewise achieves only 4.0% success on unseen tasks without VITA.

  • VITA improves cross-task execution with both action decoders. With VITA, Cosmos3-Nano-Policy-DROID reaches 55.6% on LIBERO using AoE and 43.2% using its native IDM, improving by 55.1 and 42.6 percentage points, respectively. On RoboTwin Easy/Hard, the corresponding results increase to 61.8% / 40.2% with AoE and 58.9% / 43.3% with the native IDM. These improvements show that VITA’s benefits extend beyond execution through the external AoE.

  • DROID pretraining and VITA provide complementary benefits. With the same AoE decoder, Cosmos3-Nano-Policy-DROID + VITA outperforms Cosmos + VITA on LIBERO (55.6% vs. 51.7%) and RoboTwin Easy/Hard (61.8% / 40.2% vs. 57.7% / 36.4%). Its LIBERO performance also approaches the 57.0% recorded-video replay reference. Thus, stronger pretraining remains useful, while VITA helps translate the pretrained model’s capabilities into executable predictions.

AuthorsOpenReview ↗

Rebuttal for Reviewer nZoT (part 3 / 3)

Can larger data scale alone solve imagination problems (Question 2)?

We agree that increasing the amount and diversity of target-environment data may help world models capture fine-grained gripper–object dynamics. Our claim is not that scaling cannot resolve this problem, but that strong embodied pretraining and additional standard downstream training do not eliminate it within the settings and budgets we evaluate.

The Cosmos3-Nano-Policy-DROID results provide direct evidence relevant to this question: despite large-scale embodied pretraining, cross-task execution remains poor after standard downstream fine-tuning with either AoE or the native IDM. With the same target-domain demonstrations, adding VITA increases average LIBERO success from 0.5% to 55.6% with AoE and from 0.6% to 43.2% with the native IDM. These results show that substantially better cross-task generalization can be obtained by changing the supervision, without collecting additional target-domain demonstrations.

Our target-data adaptation sweep (Appendix G.3, Table 20) and longer native training ablation (Appendix J.2, Table 29) further support this conclusion within the tested budgets. In particular, 3×3\times longer native training improves cross-task execution from 0.4% to only 5.5%, compared with 51.7% using VITA. Increasing data or native training compute over the ranges studied therefore does not close the gap to VITA.

VITA itself draws on substantial prior data through AoE, which is pretrained once on 9,637 hours of diverse data. Its advantage is that this inverse-dynamics knowledge can be reused to guide downstream video learning, reducing the need for additional data collection in each target environment. The resulting claim is improved target-domain data efficiency: VITA enables stronger cross-task execution under a fixed downstream demonstration budget, even when the underlying model already benefits from large-scale embodied pretraining.

Comments on the paper

2 comments
Area_Chair_fVnzOpenReview ↗

Rebuttal Discussion

Dear Reviewers,

The authors have now submitted their rebuttal and responded to the concerns raised in the reviews. We would greatly appreciate it if you could take some time to review their responses and participate in the discussion, particularly regarding whether your concerns have been adequately addressed.

Thank you very much for your time and valuable contributions!

AuthorsOpenReview ↗

General Response

We thank the Area Chair and all reviewers for their constructive feedback. We have revised the manuscript and updated the supplementary archive with new real-world robot experiments, additional baselines and ablations, and full training and evaluation code.

1. Summary of Recognized Strengths

  • Important Problem and Practical Method: Bridging the gap between visually plausible generated videos and executable robot actions is a critical practical problem, and integrating our pretrained AoE inverse-dynamics model via VITA is novel and effective across both WMs and WAMs (reviewer nZoT, asNW).
  • Rigorous Causal Diagnosis: Our controlled factorization, blinded human study, and local offset and repair interventions cleanly separate visual plausibility from physical executability, showing why AoE agreement (0.79 AUROC) succeeds where standard image metrics (SSIM, LPIPS) fail (reviewer u7dG, asNW).
  • Strong Transfer and Thorough Ablations: VITA lifts cross-task execution from near zero to 47.0% and 51.7% (pass@1) on LIBERO, doubles held-out cross-task success on FastWAM (11.2% to 23.6%) at zero inference rendering cost, and transfers across independent IDM evaluators (AoE, J-IDM, DreamGen-IDM) (all reviewers).

2. Summary of Key Concerns and Our Responses

  • Real-World Robot Evaluation (reviewer nZoT, asNW): We evaluated LingBot-VA and Cosmos3-Nano-Policy-DROID across 1 in-domain task (Tabletop Tidying) and 3 unseen out-of-domain tasks (Tissue Extraction, Water Pouring, Bottle Sorting; 100 trials per model, rollout videos in supplement). VITA raises unseen cross-task success from 4.0% to 14.7% (mean score +19.0) on Cosmos3-Nano-Policy-DROID and from 6.7% to 14.7% (mean score +14.5) on LingBot-VA.
  • Large-Scale DROID Pretraining and Data Scale (reviewer nZoT, u7dG): Without VITA, DROID-pretrained Cosmos3-Nano-Policy-DROID still collapses on unseen tasks (0.5% on LIBERO, 1.8% / 1.2% on RoboTwin); adding VITA boosts it to 55.6% on LIBERO and 61.8% / 40.2% on RoboTwin, without requiring the 1,000x downstream target data needed by pure data scaling.
  • Video Quality Metrics and 3x Longer Native Training (reviewer u7dG, nZoT): Tripling native Cosmos training improves cross-task FVD but only reaches 5.5% execution, whereas Cosmos + VITA achieves both better FVD and 51.7% execution. Full WAN2.2 training configs and code are also added.
  • Action Alignment vs. Object States and Alternative Teachers (reviewer asNW, u7dG): High AoE agreement raises valid object transitions with successful execution from 26.7% to 62.7% over low agreement, while post-contact deformable and fluid physics depend on base world-model priors. Training VITA with a source-trained J-IDM teacher also lifts execution from 0.9% to 27.8% under the neutral DreamGen-IDM evaluator (vs. 30.8% with our zero-shot AoE teacher).
  • Pass@k Sweep, Multi-Seed CIs, Perturbations, and Training Overhead (reviewer u7dG): All main-text VITA results are pass@1, and WAN2.2 + VITA scales from 46.95% (pass@1) to 74.55% (pass@50). We also added 2-seed bootstrap 95% CIs, RoboTwin episode counts, explanations of LIBERO-Plus Noise/Camera drops, EgoDex grasp-masking ablations, and training cost / timing analyses.

3. Summary of Manuscript and Supplementary Revisions

Section / LocationSummary of Added Content and Revisions
Main Text: Section 4.4Added real-world cross-task evaluation on LingBot-VA and Cosmos3-Nano-Policy-DROID across 1 in-domain task (Tabletop Tidying) and 3 unseen out-of-domain tasks (Tissue Extraction, Water Pouring, Bottle Sorting; 100 trials per model).
Appendix J.1Detailed real-world scoring rubrics; paired analysis of AoE action agreement vs. object-state transitions; simulation cross-task evaluation of Cosmos3-Nano-Policy-DROID; and full training configurations for Cosmos and WAN2.2.
Appendix J.2Generated-video quality metrics (FVD, PSNR, SSIM, DINO) alongside a 3x longer native Cosmos baseline ; pass@1 protocol clarification and full pass@k sweep (k = 1 to 50) for WAN2.2 + VITA + AoE.
Appendix J.3Cross-teacher VITA ablation using a source-trained J-IDM supervisor evaluated across AoE, J-IDM, and DreamGen-IDM (Table 31); and per-fold episode accounting on RoboTwin 2.0.
Appendix J.4Independent second training seed (Seed 1) with paired bootstrap 95% confidence intervals on LIBERO, and analysis of noise and camera on LIBERO-Plus.
Appendix J.5Wall-clock step time and peak GPU memory overhead of VITA training, and matched 20 Hz replay timing with median recovered displacement amplitudes.
Appendix J.6EgoDex hand-to-target mapping and human-grasp masking ablation, pretraining split/overlap checks, blinded-study rater and license details, and clarifications on Table 8 units/overfitting and Section 3.3 sensitivity framing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.