- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks why world action models (WAMs) fail to transfer. Its methodological move is to factor a WAM into a world model and an inverse-dynamics model (IDM), freeze the IDM, and then swap only the source of future frames, giving three conditions under one execution protocol: replay the recorded actions, recover actions from recorded frames, and recover actions from generated frames. The IDM, AoE, is a camera-space model trained on 9,637 h of robot, human and simulated video; it predicts a 12-D target per arm (start and end UV, relative depth, 6-D rotation, gripper) which is turned into an executable pose at run time using the robot state and camera calibration.
With that setup the authors report that recorded frames are largely recoverable (57.0% average on LIBERO against 72.3% for action replay) while generated futures are essentially dead: 0.3–0.4% at pass@1, 4.6–5.9% at pass@50. The blinded study in Appendix G is more interesting: 77% of Cosmos's futures are judged plausible and 67.5% as looking like the task was completed, yet 0.5% of them execute.
They then add VITA, a training-time loss that sends the differentiably decoded generated future through the frozen AoE under the same action noise and flow time as the recorded future, and penalises both the mismatch to the teacher's response on the recorded future and the mismatch to the ground-truth action-flow target, with a small gripper-ROI reconstruction term. That takes generated futures to 47.0% (WAN2.2) and 51.7% (Cosmos) on LIBERO and to about 30% on LIBERO-Plus. The same loss is applied to LingBot-VA and FastWAM, where the gains are smaller but still there, and the largest of them arrive out of distribution.
The appendix also has a scaling study on AoE's pretraining data, ablations on the camera-space representation and on the individual loss terms, the blinded human study, and the local repair and offset experiments.
Critical strengths
The measurement is the contribution here, not the loss. I came in sceptical of yet another "IDM as evaluator" setup, since the IDM is usually the thing you should be worried about, and this paper spends most of its appendix budget trying to kill exactly that worry. Reconstructed real frames still execute at 91.3%, so it is not the VAE or the image statistics. An independent geometric recovery pipeline also collapses on generated futures (3.0%), so it is not an AoE artefact. The blinded study separates "looks plausible" from "executes" cleanly. And the intervention experiments are the part I find most convincing: a 4-pixel offset of the interaction region in real video drops execution from 75.5% to 23.0% while 86% of the videos still look plausible, and repairing the interaction region of generated segments lifts success from 1.5% to 32.0% against 3.0% for background repair. That is a causal story about where the information is missing, not a correlation with a metric. As a side note, Table 17 is the best argument I have seen for why SSIM and LPIPS are the wrong selection criterion here: AoE action agreement gets AUROC 0.79 against 0.55 to 0.61 for the three image metrics.
VITA itself is a modest idea. It distils the teacher's response on generated frames toward its response on real frames, plus an action anchor, and I would not have expected that to be enough. The evidence that it transfers is better than I expected, in particular the evaluator swap in Appendix H: the same checkpoint goes from 0.5 to 29.9 with AoE, from 1.8 to 26.8 with J-IDM, and from 0.9 to 30.8 with DreamGen-IDM. The controls in Table 20 also matter, because "any extra auxiliary loss helps" is the obvious alternative explanation, and frozen DINO alignment plus a matched-budget native run do not reproduce the gain. The FastWAM result is the one that made me take the mechanism seriously: it never renders a future at inference, and its held-out cross-task success still doubles (11.2 to 23.6).
Critical weaknesses
My main worry is about what the headline number is measured against. Cosmos is adapted for 10,000 iterations, and the config in the supplement initialises from the 14B GR00T checkpoint with per-GPU batch 1, not the 4 the table says. WAN2.2's training recipe is never given anywhere: Appendix F.2 and Table 11 describe Cosmos only, and the supplement has no WAN2.2 world-model code at all, only the Wan2.2 VAE and DiT vendored inside FastWAM. Worse, there is not a single generated-video quality number in the paper. FVD is mentioned once, in Appendix G.2, as being "recorded separately at the model/checkpoint level", and no value appears. So a reader cannot fully separate "the world model produces plausible futures whose geometry is wrong" from "the world model was adapted too briefly to carry usable motion, and an action-supervised auxiliary loss rescues it". I think the human study and the VAE control make the second story unlikely for Cosmos, and I weight them a lot, but the title and abstract make a claim about WAMs in general from two models whose adaptation budgets are short or unknown. I would like either FVD/PSNR plus one longer or stronger baseline, or a narrower claim.
The second problem is that the paper is not careful about noise. There are no seeds, error bars or confidence intervals anywhere. RoboTwin has 25 tasks per fold, and several of the deltas the paper leans on sit right at the level where this matters: +0.5 for LingBot-VA on LIBERO, +1.1 for FastWAM in distribution. I know another round of full evaluation is expensive at this scale, but even two seeds on the LIBERO tables or a stated episode count per RoboTwin cell would let me tell a real effect from a lucky run. Table 6 also has two clearly negative cells (FastWAM noise −10.3, camera −2.2) which are printed and then never mentioned, under a sentence claiming "consistent gains". The pass@k accounting is lopsided in the same direction: baselines get pass@1, pass@50 and a full candidate sweep in Table 18, while the VITA rows in Tables 3 and 4 give one number with no k and no sweep. Table 16 labels the same checkpoints pass@1, which suggests this is fine, but then the main tables should say so, because otherwise "from near zero to 47.0%" is comparing a model change with an unstated sampling budget.
Smaller things, roughly in the order they bother me. Table 7 puts pretrain loss and validation RMSE next to each other for two representations in different units, which produces the odd reading that the 3D-pose model has a lower pretrain loss (0.026 vs 0.047) and a higher RMSE (0.146 vs 0.070); the downstream column is the only one that carries the argument, and the rest should be dropped or normalised. Table 3's caption is split by the table header, with "the checkmark indicates that the AoE is" above the header and the rest of the sentence below it. §3.1 says AoE works "with little data from a downstream environment", which contradicts the zero-shot protocol used everywhere else. The analysis in §3.3 and Appendix E is fine as algebra but does not do any work: Proposition 1 is completing the square, Propositions 2 and 3 are Taylor expansion and the chain rule, and the bound in Eq. (10) has unspecified local constants, an unmeasured teacher error, and a hypothetical reference feature whose decoded future is the recorded one. Its key assumption, Eq. (31), is basically the transfer property you would want to test. No experiment estimates any of the constants or checks the bound. I would either present this as intuition or pick one measurable quantity and report it. Finally, the strongest WAM result in the paper, FastWAM going from 11.2 to 23.6 on held-out tasks, only appears in Table 20, while the main text shows 97.6 to 98.7, where the effect is close to the ceiling.
Questions for authors
-
What k and what reset policy are used for the VITA rows in Tables 3 and 4, and for the WAM tables? The baselines get a full pass@k sweep in Table 18; I would like the same curve for one VITA checkpoint. If 47.0 and 51.7 are pass@1, this question goes away, but please state it.
-
What is WAN2.2's training configuration (initialisation, data, global batch, iterations, hardware), and why is it missing from the supplement? Relatedly, can you report FVD and PSNR/SSIM/DINO between generated and recorded futures for both world models, in distribution and cross-task, and add one longer-trained or otherwise stronger world model so that the "collapse" can be attributed to the content of the futures rather than to a short adaptation?
-
722 h of the 9,637 h corpus is human ego video (EgoDex), but the abstract calls it robot image–action pairs. How is a human-hand episode mapped to the 12-D per-arm target, specifically the gripper coordinate? The released code uses thumb–index tip distance in metres, while the robot datasets supply a binary command; given that the paper says the gripper is where the failure lives, could mixing those two conventions into one coordinate be part of the story? And does the corpus contain any episodes or assets from LIBERO, RoboTwin or RoboCasa scenes, and how was that checked?
-
AoE is both the VITA teacher and the evaluator, and on recorded images it is the weakest of the three IDMs you report (47.0 against 55.5 for J-IDM and 54.8 for DreamGen-IDM); on RoboTwin-Hard with VITA futures it reports 40.4 against 66.7 for J-IDM. Can you run VITA with a teacher other than AoE, for example J-IDM trained on the source tasks, and see whether the gain survives? That would separate "better futures" from "futures that match this particular teacher".
-
What temporal interval separates the frame pairs AoE consumes at execution, and is it the same for recorded and generated video? I could not work out what ties a per-pair displacement to one controller step, and if the generated video has a different effective motion scale than the recorded one, that alone could produce a near-zero success rate. Also, what is the wall-clock and memory cost of the VITA step (1.14 B frozen teacher on 8-frame chunks every second optimiser step, differentiable VAE decode and VJP, teacher offload) relative to native training?
-
The LIBERO-Plus "Noise" column is 0.0% for recorded-image replay with AoE and 0.1% with VITA. Is that a property of the futures or of AoE's sensitivity to sensor noise? Should that column be read as part of the cross-environment comparison, or as a failure mode of the evaluator?
4 comments
Rebuttal for Reviewer u7dG (part 2 / 3)
3. Multi-Seed Evaluation, Episode Accounting & Why Noise and Camera Degrade (W2 & Q6)
- Seeds & Episode Accounting (Appendix J.3, Table 33 & Appendix J.4, Table 32): Table 33 details the exact RoboTwin accounting (25 held-out tasks/fold 2 configs 50 trials = 2,500 trials per fold per condition). Table 32 reports an independent second training seed (Seed 1) and paired bootstrap 95% confidence intervals on LIBERO: LingBot-VA ID (98.5% -99.0% Seed 0; 98.6% -98.9% Seed 1), FastWAM ID (97.6% -98.7% Seed 0, 95% CI [+0.4, +1.8]; 97.7% -98.4% Seed 1), and FastWAM Held-out tasks (11.2% -23.6% Seed 0, 95% CI [+10.1, +14.7]; 11.6% -22.9% Seed 1).
- Why Noise (-10.3 on FastWAM; 0.0%/0.1% on WM+AoE) and Camera (-2.2 on FastWAM) occur (Section 5 & Appendix J.4): We apologize for not explaining these columns earlier and have added a dedicated discussion in Appendix J.4 (lines 2030–2039) and Section 5:
- Noise (0.0% / 0.1% & -10.3): LIBERO-Plus Noise applies heavy image blurring/corruption that obscures the robot gripper. Vision-based IDMs cannot recover actions when the visual input lacks identifiable gripper geometry—even on ground-truth recorded videos under Noise (Table 24), AoE gets 0.0%, J-IDM 0.9%, and DreamGen-IDM 3.6%. Inspecting the single successful rollout (0.1% in Table 4) confirmed it was an incidental random completion. Consequently, Noise reflects a visibility boundary of visual IDMs; enforcing auxiliary supervision on blurred frames where the teacher cannot localize the gripper injects noisy gradients, causing the -10.3 drop on Noise for FastWAM.
- Camera (-2.2): LIBERO-Plus Camera perturbs camera extrinsics. Because AoE predicts in camera space and relies on calibrated extrinsics, uncalibrated extrinsic shifts move the arm to out-of-distribution image coordinates during auxiliary supervision.
4. EgoDex Hand Mapping, Grasp Masking Ablation, Asset Overlap & Ethics Note (Q3)
- Human Hand Mapping & Grasp Masking Ablation (Appendix J.6, Table 36 & code/canonical_action/data/egodex2aligned.py): Each wrist is transformed into camera coordinates via , and the 12th coordinate measures thumb–index tip pinch aperture in meters. To test whether mixing continuous human pinch distance with binary robot gripper commands hurts gripper learning, we pretrained an AoE variant with all human grasp targets masked out (Table 36): held-out robot grasp open/close balanced accuracy (88.6% vs. 89.0%) and LIBERO execution on recorded (56.8% vs. 57.0%) and generated futures (0.4% vs. 0.3%) are virtually identical.
- Corpus Overlap, Raters & License (Appendix J.6, lines 2093–2114): All held-out LIBERO evaluation tasks and held-out RoboTwin fold tasks are excluded prior to AoE pretraining. As stated in Table 10 and Appendix J.6, the corpus includes 122.1h of RoboCasa, so our cross-task claims concern unseen task identities and demonstrations rather than asset-level novelty. We also clarified that the blinded study used 6 raters (first author + 5 external contributors, 3 raters/video) and cited the EgoDex CC BY-NC-ND license.
5. Training VITA with an Alternative Teacher (Source-Trained J-IDM, Q4)
In Appendix J.3 (Table 31), we trained WAN2.2 on the same source-task split using a source-trained J-IDM as the VITA teacher and evaluated all checkpoints across AoE, J-IDM, and the completely independent DreamGen-IDM evaluator:
| Training Supervisor | AoE Evaluator (Zero-shot, 7 cats) | J-IDM Evaluator (In-dist., 6 cats) | DreamGen-IDM Evaluator (In-dist., 6 cats) |
|---|---|---|---|
| None (Native WAN2.2) | 0.5% | 1.8% | 0.9% |
| AoE (Zero-shot cross-task teacher) | 29.9% | 26.8% | 30.8% |
| Source-trained J-IDM (In-domain teacher) | 27.0% | 28.1% | 27.8% |
Under the neutral DreamGen-IDM evaluator (which supervised neither model), Source-trained J-IDM supervision boosts execution from 0.9% to 27.8%, confirming that action-grounded supervision transfers across teachers. Crucially, Source-trained J-IDM is trained in-domain on the target environment's source tasks (as 3D-pose IDMs do not transfer across environments), whereas our AoE teacher is strictly zero-shot cross-task yet achieves even stronger transfer (30.8% under DreamGen-IDM).
Rebuttal for Reviewer u7dG (part 3 / 3)
6. Temporal Interval, Motion Scale & Training Overhead (Q5)
- Replay Timing & Motion Scale (Appendix J.5, Table 35): Recorded and generated clips share the exact same temporal grid: effective frame rate, frames separation ( per pair), and controller rate ( control ticks per pair). Median recovered translation per pair on the matched time grid shows that native generated futures exhibit attenuated motion amplitude (1.4 cm on LIBERO, 1.7 cm on RoboTwin vs. 2.1 cm and 2.5 cm for recorded videos), while VITA restores the displacement to 1.9 cm and 2.2 cm.
- Wall-Clock & Memory Cost (Appendix J.5, Table 34 & code/evaluation/training_cost.py): On 32 H800 GPUs, adding VITA changes step time and peak GPU memory from 25s -32s () and 62 -72 GiB on Cosmos, 18s -21s () and 55 -62 GiB on LingBot-VA, and 2.4s -4.1s () and 49 -60 GiB on FastWAM, with zero inference-time cost for WAMs.
7. Minor Clarifications (Table 8 [formerly Table 7], §3.1, §3.3 & Formatting)
- Table 8 Pretrain Loss vs. Valid RMSE: Beyond differing coordinate units (Appendix J.6), our pretraining corpus is assembled incrementally across datasets; when environment-specific data is limited, the 3D-pose IDM overfits to training coordinate frames (lower training loss 0.026 vs. 0.047 but higher validation RMSE 0.146 vs. 0.070 and 0.0% cross-task transfer), whereas scaling diverse data in camera space resolves this overfitting.
- We fixed the split caption in Table 3, clarified the zero-shot vs. 10-shot protocol distinction in §3.1 and Appendix J.6, and framed §3.3/Appendix E explicitly as a local sensitivity analysis corresponding to the empirical offset/repair interventions in Figure 11.
As the discussion period is drawing to a close, we kindly invite you to share any remaining questions or concerns about AoE so that we can address them before the deadline. We look forward to your valuable feedback. : )
Best wishes,
Authors of Submission #13777
Rebuttal for Reviewer u7dG (part 1 / 3)
We sincerely thank Reviewer for the exceptionally thorough reading, for highlighting the strength of our causal intervention and evaluator-swap experiments, and for the constructive questions. We have addressed all six questions and every methodological and presentation concern in Appendix J (J.1–J.6, Tables 26–36), Section 4.4 (Table 7), Section 5, and the updated Supplementary Material (code/).
1. Pass@k Protocol, Reset Policy & Full Candidate Sweep for VITA (Q1 & W2)
2. WAN2.2 Recipe, Code Supplement, Video Metrics (FVD/PSNR/SSIM/DINO) & Stronger/Longer Baselines (Q2 & W1)
Tripling native training (30k steps) improves XT FVD (157→140), yet cross-task execution only reaches 5.5% (vs. 51.7% for Cosmos + VITA at 10k steps).