- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents OpenWAM, an open research stack for World-Action Models (WAMs), policies that inherit world knowledge from video-generation priors and learn control from embodied data. It has three parts. OpenWAM-Infra factorizes the WAM design space into composable modules (visual encoder, stream backbones, visibility mask) behind one trainer, one policy server, and one evaluation protocol, assembling six architectures from configuration. OpenWAM-Study runs controlled experiments on RoboTwin2.0 and distills three findings: world knowledge transfers best through a capable generative backbone and a compact, information-rich latent space; world-action synergy requires world-to-action information flow in training and synchronized denoising at inference; and embodied pretraining mainly improves out-of-domain generalization, with egocentric video and robot trajectories contributing different strengths and one-stage co-training the best integration. OpenWAM-alpha composes the recipe into a 5B world stream plus 1B action stream model pretrained on 518.5M frames (6,369 hours) of egocentric and robot data, evaluated on eight simulation benchmarks and three real-robot embodiments, including a dexterous hand whose 30-D action space never appears in pretraining. It sits near the top on most benchmarks, and the paper includes an unusually honest analysis of its one clear failure (LIBERO-Plus).
Critical strengths
- Open source contribution. Six architectures, five video backbones, four visual encoders, four attention masks, one trainer, one policy server, eight benchmark adapters, released with pretrained weights and data recipes. StarVLA[1] did this for VLAs; nothing equivalent existed for WAMs, and the design decisions (self-contained checkpoints, benchmarks as thin WebSocket clients, the 80-D unified action space) are the right ones. Independent of any finding, this lowers the cost of doing careful WAM research for everyone, and OpenWAM-alpha is immediately useful as a strong open baseline across three real embodiments.
- The controlled studies ask the right questions, and several answers are crisp. The mask ablation isolates the direction of information flow with a clean five-point split (isolated and video-sees-action at 87.4-87.6 versus action-sees-video and mutual at 92.2-92.4), which is a real result. The denoising sweep is a thorough null result: sixteen asynchronous configurations, none beating synchronized denoising, which usefully kills the plan-then-imagine design direction at test time. The LIBERO-Plus analysis (E.5) is model self-criticism: it traces the deficit to single-arm data coverage and pixel-latent sensitivity, corroborated across three baselines with different prediction targets.
References
[1] Community, StarVLA. "StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developing." arXiv preprint arXiv:2604.05014 (2026).
Critical weaknesses
- There is no variance anywhere, and the recipe-driving decisions are sub-point. The entire Study rests on one benchmark and one run per configuration, success rate only. The margins that decide the recipe are 92.36 versus 92.60 (dual versus tri-system), 92.39 versus 92.15 (action-sees-video versus mutual), 93.0 versus 92.3 (sync versus the best async schedule), and the claimed mask reversal after pretraining (Figure 7) is +0.70, -0.24, +0.72, and +0.16 points. Without seeds or any significance testing, Findings 2 and 3 and the reversal claim could be noise, and the stakes are higher because the findings are confirmatory of results the paper itself cites (the test-time schedule conclusion, the representation-latent results, the co-training recipe): their value depends on being measured better than the originals, and they are not.
- The cross-model rankings are not auditable. Figure 9 and Appendix H compare against roughly 25 models, but the paper never states which numbers were re-run under the OpenWAM-Infra protocol (same preprocessing, episode counts, and evaluation conditions) and which are quoted from papers or leaderboards. The comparison is also confounded by the variable the paper itself calls decisive: pretraining data coverage, with baselines ranging from 6.4k hours (here) to over 100k hours. Finally, "top-tier performance across eight simulation benchmarks" overstates the record: on LIBERO-Plus the model scores 69.2 against 89.0 for the best baseline, 12th of 13 in its own table. The honest claim, consistently strong with one explained weak spot, is a good claim, and the abstract should say that.
Questions for authors
- Can you add multi-seed runs (3 or more) for the Study's decision-driving comparisons and report mean and std: the architecture table (Table 1), the mask tables (Table 2 and Figure 7), and the denoising sweep (Figure 5)? Most of these margins are under one point, so Findings 2 and 3 and the mask-reversal claim stand or fall on whether they survive seed noise. This is the single change that would most affect my confidence in the paper.
- For every baseline number in Figure 9 and Appendix H, please state whether it was re-run under your protocol or quoted, and where quoted, how episode counts, observation preprocessing, and evaluation conditions differ from yours. As it stands, the cross-model rankings, and therefore the "top-tier" claim, cannot be audited.
5 comments
Rebuttal for Reviewer VVgp – Question1 (Independent training runs)
(b) Independent training runs: seeds 42 and 0.
All original experiments were trained with seed=42. We retrained the configurations in Tables 1 and 2 and the pretrained information-flow study with seed=0, using exactly the same data and training configurations as the original runs, and independently evaluated the resulting models. Below we report the new scores and mean ± population std across the original and new training runs (n=2, ddof=0). Success rates are in %.
Architecture ablation on RoboTwin2.0-Full (Table 1).
| Architecture | Seed=0 Clean | Seed=0 Randomized | Seed=0 Avg. | Clean mean ± std | Randomized mean ± std | Avg. mean ± std |
|---|---|---|---|---|---|---|
| Single-system Vanilla | 85.48 | 86.10 | 85.79 | 85.340 ± 0.140 | 85.950 ± 0.150 | 85.645 ± 0.145 |
| Single-system MoE | 86.38 | 83.16 | 84.77 | 86.300 ± 0.080 | 83.100 ± 0.060 | 84.700 ± 0.070 |
| Dual-system Joint Self-Attention | 92.48 | 92.64 | 92.56 | 92.410 ± 0.070 | 92.510 ± 0.130 | 92.460 ± 0.100 |
| Dual-system Joint Cross-Attention | 87.97 | 88.79 | 88.38 | 87.915 ± 0.055 | 88.715 ± 0.075 | 88.315 ± 0.065 |
| Dual-system Detached Cross-Attention | 92.34 | 91.94 | 92.14 | 92.200 ± 0.140 | 91.790 ± 0.150 | 91.995 ± 0.145 |
| Dual-system IDM | 87.46 | 87.90 | 87.68 | 87.610 ± 0.150 | 88.020 ± 0.120 | 87.815 ± 0.135 |
| Tri-system Joint Self-Attention | 93.10 | 92.56 | 92.83 | 92.970 ± 0.130 | 92.460 ± 0.100 | 92.715 ± 0.115 |
Attention-mask ablation on RoboTwin2.0-Full (Table 2).
| Attention mask | Seed=0 Clean | Seed=0 Randomized | Seed=0 Avg. | Clean mean ± std | Randomized mean ± std | Avg. mean ± std |
|---|---|---|---|---|---|---|
| Isolated | 88.21 | 86.93 | 87.57 | 88.145 ± 0.065 | 86.835 ± 0.095 | 87.490 ± 0.080 |
| Video Sees Action | 88.09 | 87.61 | 87.85 | 88.005 ± 0.085 | 87.475 ± 0.135 | 87.740 ± 0.110 |
| Action Sees Video | 93.28 | 92.08 | 92.68 | 93.130 ± 0.150 | 91.940 ± 0.140 | 92.535 ± 0.145 |
| Mutual | 92.64 | 92.08 | 92.36 | 92.570 ± 0.070 | 91.940 ± 0.140 | 92.255 ± 0.105 |
Information flow before and after pretraining (Figure 7). ASV denotes Action Sees Video; Δ = Mutual − ASV is measured in percentage points.
| Setting | Seed=0 ASV | Seed=0 Mutual | Seed=0 Δ | ASV mean ± std | Mutual mean ± std | Δ mean ± std |
|---|---|---|---|---|---|---|
| From scratch / RoboTwin-Full (Avg.) | 92.68 | 92.36 | −0.32 | 92.535 ± 0.145 | 92.255 ± 0.105 | −0.280 ± 0.040 |
| Pretrained / RoboTwin-Full (Avg.) | 92.45 | 92.76 | +0.31 | 92.395 ± 0.055 | 92.630 ± 0.130 | +0.235 ± 0.075 |
| Pretrained / RoboTwin-C2R (ID) | 87.23 | 87.79 | +0.56 | 87.105 ± 0.125 | 87.735 ± 0.055 | +0.630 ± 0.070 |
| Pretrained / RoboTwin-C2R (OOD) | 25.56 | 26.22 | +0.66 | 25.630 ± 0.070 | 26.320 ± 0.100 | +0.690 ± 0.030 |
Across these two independent training runs, the average-score rankings in Tables 1 and 2 are unchanged, and the preference for Mutual after pretraining is reproduced. The new scores differ from the originals by at most 0.30 percentage points, providing additional evidence for the stability of these qualitative conclusions.
Rebuttal for Reviewer VVgp – Question2
Q2. Baseline-score sources and evaluation protocols.
We sincerely thank you for raising this concern. OpenWAM-α scores are our evaluations; the external baseline scores in Figure 9 and Appendix H are quoted from published papers or official leaderboards.
Our benchmark clients are built on the official benchmark repositories, environments, or evaluation services. They reuse benchmark-owned tasks and success conditions, with local adapters handling policy communication and observation/action conversion:
| Benchmark | Evaluation implementation | Baseline-score source |
|---|---|---|
| LIBERO | Official LIBERO environments, task files, and initial states | Published comparison tables in the three papers cited below |
| LIBERO-Plus | Official LIBERO-Plus environments and task variants | Published comparison tables in the three papers cited below |
| VLABench | Official VLABench Evaluator and frozen evaluation configurations | Published comparison tables in the three papers cited below |
| RoboTwin2.0-Full | Official RoboTwin evaluation script, tasks, and success criteria | Published comparison tables in the three papers cited below |
| RoboTwin2.0-Clean2Random | Official RoboTwin evaluation; clean-only downstream training | Official RoboDojo leaderboard, including its RoboTwin panel |
| RoboDojo | External XPolicyLab benchmark evaluation infrastructure | Official RoboDojo leaderboard |
| EBench | Official Generalist/Test-Mini split and EBench/GenManip online evaluation procedure | Official EBench evaluation website |
| RoboCasa365 | Official RoboCasa environments, task horizons, and success signals | Official RoboCasa leaderboard |
| RoboCasa-GR1 | Official GR1 tabletop environments and success signals | Original benchmark paper and subsequent papers evaluating the same benchmark |
The published comparison tables used for LIBERO, LIBERO-Plus, VLABench, and RoboTwin2.0-Full are from Qwen-RobotManip, Xiaomi-Robotics-1, and ABot-M0.5.
We screened quoted results for compatibility with each benchmark's task definitions, downstream training-data regime, and evaluation setting, excluding known incompatible entries. Observation preprocessing and pretraining data remain policy-specific: each policy retains its native camera processing, normalization, action representation, and pretrained checkpoint.
We sincerely thank you for your careful and thorough review. We will incorporate these clarifications and additional evaluation details into the final version of the paper.
Rebuttal for Reviewer VVgp – Question1 (Fixed-checkpoint evaluations runs)
Q1. Stability across evaluation and training seeds.
We sincerely thank you for highlighting this concern. We separately report variability across evaluation seeds with fixed checkpoints and across independent training runs.
(a) Fixed-checkpoint evaluations: seeds 42, 0, and 10.
Keeping each configuration's original trained checkpoint fixed, we ran two additional evaluations with seed=0 and seed=10, supplementing the original seed=42 evaluation for Tables 1 and 2 and Figures 5 and 7. RoboTwin evaluation comprises 10,000 test episodes. Below we report mean ± sample std over these three evaluation seeds (n=3, ddof=1). Success rates are in %, with std and differences in percentage points.
Architecture ablation (Table 1).
Attention-mask ablation (Table 2).
Denoising schedules (Figure 5). Synchronized denoising achieves 92.840 ± 0.260%, compared with the following asynchronous schedules:
Information flow before and after pretraining (Figure 7). ASV denotes Action Sees Video; Δ = Mutual − ASV, with its mean and std computed from the paired per-seed differences.
The observed success-rate standard deviations are at most 0.361 percentage points. Across all three evaluation seeds, the average-score rankings in Tables 1 and 2 remain unchanged, synchronized denoising outperforms every tested asynchronous schedule, and the preference for Mutual after pretraining is reproduced. These results support the consistency of the reported qualitative conclusions under evaluation-seed variation.