- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies two design choices in world-action models: the pretrained features used as visual inputs and prediction targets, and the objective used to predict those features. A common mixture-of-transformers policy is evaluated with several visual encoders and two different modeling objectives: frame-wise autoregressive regression or diffusion. V-JEPA 2.1 features perform best among the tested diffusion configurations, while autoregression improves action success with both Wan VAE and V-JEPA features. The resulting recipe reaches 85.9% average RoboTwin 2.0 success without additional robot pretraining and 91.5% after 3,300 hours of robot pretraining. Offline probes and gradient-alignment measurements provide possible explanations for these differences.
Critical strengths
-
A useful empirical question with substantial observed effects. Table 2 provides a two-by-two comparison: autoregression raises Wan VAE success from 38.0% to 73.9% and V-JEPA success from 74.2% to 85.9%. The improvement is therefore not confined to changing the encoder alone.
-
Broad evaluation within the selected benchmark. The study covers all 50 RoboTwin tasks and both clean and randomized conditions, with 100 episodes per task and condition. The additional robot-pretraining experiment shows that the combined recipe's advantage persists beyond task-only training.
-
The representation diagnostics go beyond a leaderboard comparison. The paper connects control performance to semantic, geometric, and action probes, and measures both intermediate-representation quality and visual/action gradient alignment. Appendix E documents episode-level splits and bootstrap intervals for the linear-probe analysis.
-
Detailed implementation is described. The appendices specify model dimensions, observation windows, feature normalization, attention masks, optimizer settings, and action execution. The text appropriately describes gradient alignment as a possible explanation and acknowledges that the noise presets also change loss normalization.
Critical weaknesses
-
The contribution of future-feature supervision is not isolated. Using the same encoder for observations and prediction targets is a reasonable design, and the comparison supports the proposed representation recipe. However, without an action-only control for each encoder, it remains unclear how much of the advantage comes from stronger input features versus the additional predictive supervision. Keeping the architecture and encoder fixed while removing the visual prediction loss would clarify whether learning future dynamics provides a consistent control benefit.
-
The autoregression-versus-diffusion comparison changes several factors together. Clean teacher-forced L1 regression and noise-weighted velocity MSE differ in loss, conditioning, masking, and effective supervision strength. Fixed weights of one do not equalize their gradient contributions. The final-checkpoint cosine measurements are informative but do not establish causality. Matched direct-regression losses, visual-loss-weight sweeps, and measurements during training would strengthen the explanation.
-
Full-data RoboTwin evaluation does not establish robustness to domain shift. Appendix C.1 uses the full 27,500-episode pool, with 50 Clean and 500 Randomized demonstrations per task, before reserving 1% for validation. Thus, evaluation on Randomized conditions after training on randomized data is not a clean-to-randomized (C2R) generalization test. The results do not rule out reliance on benchmark-specific visual or task regularities, although they do not by themselves demonstrate overfitting. A C2R setting, trained only on Clean demonstrations and evaluated on Randomized conditions without adaptation, is needed to test whether the encoder and objective rankings survive a domain shift. Low-data comparisons and multiple training seeds would further strengthen this conclusion.
-
The scaling comparison supports the tested recipe, not a general efficiency or scaling law. Only two robot-data budgets are tested, and both controlled vision experts start from random initialization. This differs from WAMs that inherit pretrained video-transformer weights. Table 4 also includes heterogeneous external systems and explicitly guessed pretraining durations. Those rows should remain context rather than evidence of superior data efficiency; compute, convergence curves, and a pretrained-video-backbone control would be informative.
Questions for authors
- How much does future-feature prediction improve over action-only training with the same current encoder and model capacity?
- Do the V-JEPA and autoregression advantages persist in a clean-to-randomized (C2R) setting, with training restricted to Clean demonstrations and no adaptation to Randomized conditions?
- Does autoregression remain better after tuning visual-loss weights and matching the direct-prediction loss? How does gradient alignment evolve during training?
- Is the ordering of the main encoder/objective configurations stable across reduced demonstration budgets and multiple training seeds?
3 comments
3. Clean-to-randomized generalization and full-data protocol. (W3, Q2)
- Full-Data Training Protocol: In our main experiments (Tables 1–4), WAM training uses the full RoboTwin 2.0 training corpus containing both Clean (50 episodes/task) and Randomized (500 episodes/task) demonstrations across all 50 tasks (Appendix C.1). We will state this explicitly in Section 2.2 so readers do not need to look up the appendix.
- Dedicated Clean-to-Randomized (C2R) Evaluation: To directly evaluate zero-shot transfer from Clean training to Randomized evaluation within the rebuttal period, we trained both the diffusion baseline and our frame-wise autoregression model from scratch (with the same frozen V-JEPA 2.1-G encoder) exclusively on 2,500 Clean episodes (~1/11th of the full training data) for 1,400 optimizer updates and evaluated them across all 50 Randomized tasks (20 episodes per task, 1,000 total episodes):
| Clean-to-Randomized (C2R) Method | Randomized Success (%) | Total Episodes Succeeded |
|---|---|---|
| Fast-WAM (External Reproduction on RoboTwin Leaderboard) [1, 2] | 1.9% | — |
| Diffusion visual prediction (Our Baseline, 1/11th Data, 1,400 updates) | 2.5% | 25 / 1,000 |
| Frame-wise autoregression (Ours, 1/11th Data, 1,400 updates) | 2.7% | 27 / 1,000 |
- Interpretation & Limitations under Severe Domain Shift: In our matched 1,400-update control, frame-wise autoregression succeeds on 27/1,000 episodes (2.7%) versus 25/1,000 episodes (2.5%) for our diffusion baseline (for context, an external reproduction of Fast-WAM on the official RoboTwin leaderboard reports 1.9% under a separate protocol [1, 2]). Because this difference is only 2 episodes out of 1,000 and absolute C2R performance remains very low across all models, we do not claim a reliable zero-shot C2R generalization advantage from this experiment. Instead, these results indicate that training solely on Clean demonstrations under a current-only inference interface () is insufficient to bridge strong visual domain randomization. Relatedly, Zhao et al. [3] show that preserving inference-time future conditioning can substantially improve out-of-distribution robustness under domain shift; exploring how our frame-wise AR representation objective interacts with inference-time future conditioning is a promising direction for future work. We will report these C2R numbers and this candid limitation in the revised manuscript.
4. Reduced demonstration budgets and multiple training seeds. (W3, Q4)
- Data-Efficiency Evidence Across Budgets: Our experiments demonstrate consistent gains across three distinct data regimes:
- Low-data Clean-only regime (2,500 episodes, ~1/11th data, trained from scratch): AR outperforms diffusion by +20.9 pp (49.3% vs. 28.4%) and action-only by +15.9 pp (49.3% vs. 33.4%).
- Full RoboTwin task regime (27,500 episodes): AR outperforms diffusion by +11.7 pp on V-JEPA 2.1-G (85.9% vs. 74.2%) and +35.9 pp on Wan VAE (73.9% vs. 38.0%).
- Large-scale robot pretraining regime (3,300 hours InternData + RoboTwin): Our combined recipe (V-JEPA 2.1 + frame-wise AR) maintains a +16.4 pp lead over the Wan VAE + diffusion baseline (91.5% vs. 75.1%).
- Multiple Training Seeds: Due to the high computational cost of training 28-layer MoT models and evaluating 10,000 closed-loop rollouts per checkpoint (50 tasks 200 episodes), we did not have enough time during the rebuttal period to run additional random seeds. Currently, all main runs use reproducible seed 42 (Table 5), while Figure 2(a) reports 95% bootstrap confidence intervals across held-out episodes—which reflect episode-level variation rather than training-seed variance. We will add multi-seed training runs in the revised manuscript to quantify seed-to-seed variance directly.
5. Initialization and robot-data pretraining. (W4) Training task-specific latent dynamics and control modules from random initialization on top of frozen perceptual encoders is an established practice in world-model and policy learning—for example, UWM [4] trains its video-action diffusion transformer from scratch over a frozen pretrained visual VAE, and DINO-WM [5] learns latent transition dynamics from scratch on frozen pretrained patch representations. Initializing the vision expert from scratch in our study ensures a strictly fair comparison between AR and diffusion: a pretrained video-diffusion checkpoint (e.g., Wan2.2-5B) cannot be dropped into a frame-wise AR V-JEPA feature predictor without mismatching both the latent space and the causal attention structure. Crucially, when we provide both recipes with the exact same 3,300-hour InternData robot pretraining prior to RoboTwin fine-tuning (Table 4), our combined recipe (V-JEPA 2.1 + frame-wise AR) still achieves 91.5% vs. 75.1% for the Wan VAE + diffusion baseline (+16.4 pp), confirming that its advantage persists when scaling up with pretrained dynamics weights. We will clarify this initialization rationale and cite these precedents in Section 2.1 and Section 5.
References:
- [1] Yuan et al., Fast-WAM: Do World Action Models Need Test-time Future Imagination?, arXiv:2603.16666, 2026.
- [2] RoboTwin Official Leaderboard (Clean-to-Randomized Evaluation),
https://robotwin-platform.github.io/leaderboard. - [3] Zhao et al., Faster-WAM: Efficient Inference-Time Future Conditioning for Robust World Action Models, arXiv:2608.04404, 2026.
- [4] Zhu et al., Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets, arXiv:2504.02792, 2025.
- [5] Zhou et al., DINO-WM: World Models on Pre-trained Visual Features Enable Zero-shot Planning, Proceedings of ICML, 2025.
We sincerely thank Reviewer kqAw for appreciating our two-by-two recipe comparison, 50-task closed-loop evaluation, and detailed diagnostics. We address each of your points below.
1. Future-feature prediction vs. action-only training (λv=0) with the same encoder. (W1, Q1) Due to limited rebuttal time, we conducted a controlled three-way comparison trained from scratch on a matched 2,500-episode Clean training split (roughly 1/11th of the main paper's 27,500-episode training data) with the same frozen V-JEPA 2.1-G encoder. Because these models are trained from scratch on 1/11th of the data, their absolute success rates are lower than the full-data models in Tables 1–2, while the comparison within this group is strictly controlled:
Frame-wise autoregression outperforms action-only training by +15.9 percentage points, demonstrating that predicting clean future visual features provides a substantial inductive bias beyond simply passing strong frozen features to the action expert. Conversely, noisy diffusion prediction underperforms action-only training by 5.0 pp, suggesting that denoising noisy future latents competes for capacity within the shared vision expert without aiding action generation. (These matched supplementary controls will be reported in their own subsection, clearly separated from the full-data Tables 1–2.)
2. Matching loss weights and gradient alignment dynamics. (W2, Q3)