- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces WorldArena 2.0, extending embodied world-model evaluation along three dimensions: visuotactile modeling, world models as reinforcement learning environments, and evaluation across simulated and physical platforms. The benchmark combines perceptual metrics with downstream task performance, including policy optimization within learned environments, synthetic-data generation, and action planning. Its experiments highlight discrepancies between prediction quality and functional utility.
Critical strengths
-
The evaluation targets practically relevant capabilities beyond visual quality. Incorporating tactile sensing, policy improvement through learned environments, and physical execution makes the benchmark relevant to embodied applications. These extensions address meaningful limitations of evaluations centered primarily on video generation.
-
The experiments provide useful diagnostic observations. The discrepancies between tactile reconstruction quality and manipulation success, together with uneven functional performance across platforms, illustrate why perceptual metrics alone are insufficient. Evaluating seven world models under three reward designs also provides useful evidence about reward sensitivity.
Critical weaknesses
-
The world-model-based RL evaluation omits important reference methods. Although the paper evaluates multiple world models within a shared policy-optimization framework, it does not experimentally compare against WMPO [1] or VLA-MBPO [2]. WMPO is cited but not evaluated. Both methods directly address VLA policy improvement through learned world models, making them particularly relevant to the benchmark’s functionality extension. The standardized framework is useful for component comparisons, but it does not establish how the evaluated pipelines compare with existing systems designed specifically for this purpose. A complementary system-level comparison under comparable data and computational budgets would substantially strengthen this part of the benchmark.
-
The protocol does not sufficiently disentangle world-model quality from rollout configuration and computational budget. Appendix C.6 states that action chunk sizes, step limits, and batch sizes vary across models, while the main comparison fixes optimization iterations. Consequently, equal iteration counts do not imply equal simulated experience, recursive prediction depth, or computational expenditure. Appendix C.7 reports per-iteration times ranging from 7.8 to 28.8 minutes. Preserving native temporal interfaces is reasonable, but the resulting rankings reflect the complete configured training pipelines, with limited evidence about which differences arise from the world models themselves. Comparisons under matched compute and matched interaction budgets, together with rollout-horizon sensitivity analyses, are needed to make these rankings more interpretable.
-
The quantitative coverage and statistical evidence are insufficient to establish reliable benchmark rankings. The main RL evaluation covers only two RoboTwin tasks, uses a single training initialization seed, and disables domain randomization during checkpoint evaluation. Moreover, Appendix C.3 reports the highest success rate observed over repeated evaluations on the same held-out configurations. This measures peak validation performance and can introduce selection optimism; it does not establish performance on an independent final test set. Physical comparisons are also based on only 10–20 trials per condition. The reported Wilson intervals and explicit limitations are helpful, but they do not capture training variability or checkpoint-selection effects. Broader task coverage, multiple training seeds, and independent evaluation of selected checkpoints are needed to support stable comparative conclusions.
[1] Fangqi Zhu, Zhengyang Yan, Zicong Hong, Quanxin Shou, Xiao Ma, and Song Guo. WMPO: World Model-based Policy Optimization for Vision-Language-Action Models. arXiv:2511.09515, 2025.
[2] Zhilong Zhang, Haoxiang Ren, Yihao Sun, Yifei Sheng, Haonan Wang, Haoxin Lin, Zhichao Wu, Pierre-Luc Bacon, and Yang Yu. Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models. arXiv:2603.20607, 2026.
Questions for authors
-
Can the authors include WMPO and VLA-MBPO as reference systems in the learned-environment RL evaluation, with clearly specified adaptation procedures and comparable resource budgets?
-
Do the relative rankings persist when comparing equal wall-clock budgets or equal amounts of generated interaction data?
-
How do the results change when checkpoints are selected on validation configurations and evaluated separately on unseen configurations across multiple training seeds?
6 comments
Response (2/4): Matched budgets and rollout-horizon sensitivity
W2. Rollout configurations, interaction volume, computational budgets, and horizon sensitivity
We agree that a fixed iteration count does not imply equal interaction or computational budgets. We therefore added comparisons along three dimensions: interaction volume, computation time, and recursive prediction depth.
- Matched interaction budget (≤38,400 generated low-level actions). We recorded cumulative generated actions, world-model queries, and optimization updates for each checkpoint. The peak success rates within this budget are:
| Optimization method | Ctrl-World | Wan | OpenSora |
|---|---|---|---|
| WMPO adaptation | 65.14% | 61.43% | 61.43% |
| VLA-MBPO adaptation | 64.94% | 63.77% | 64.94% |
Under both methods, backend rankings match those obtained from the peaks over the full training runs.
- Matched computational budget (wall-clock time / GPU-hours).
- VLA-MBPO: All three backends use eight GPUs and the same timing convention. Within a 43–48 GPU-hour window, Ctrl-World achieves 64.94% (43.7 GPU-hours) = OpenSora 64.94% (47.6 GPU-hours) > Wan 63.77% (43.2 GPU-hours).
- WMPO: Time per training step is 1.93 h for Ctrl-World, 0.50 h for OpenSora, and 0.32 h for Wan, differing by approximately sixfold. Within a wall-clock window of approximately 3.45 h, Ctrl-World achieves 64.75% > Wan 61.43% = OpenSora 61.43%.
The relative ordering of the three backends remains unchanged when interaction volume and computation time are matched separately. Although Ctrl-World is approximately six times slower per step, it still leads within the same time budget. The ranking is therefore not caused by budget differences.
- Recursive-depth (horizon) sensitivity. Action chunking bounds the number of recursive world-model generations by ⌈H/K⌉ (Appendix C.6). On Adjust Bottle with the proxy reward, we set the training horizon H of Ctrl-World and Wan to 0.5× (Short), 1× (Normal), and 2× (Long) their paper defaults. We matched the number of optimization updates and total interaction budget as closely as possible, keeping all other settings unchanged:
| Model | Configuration | Training H | Chunk | Recursive depth | Cumulative trajectories | Action chunks | Optimization updates | Peak success |
|---|---|---|---|---|---|---|---|---|
| Ctrl-World | Short | 100 | 5 | 20 | 128 | 2,560 | 8 | 69.63% |
| Ctrl-World | Normal (paper) | 200 | 5 | 40 | 64 | 2,560 | 8 | 70.70% |
| Ctrl-World | Long | 400 | 5 | 80 | 32 | 2,560 | 8 | 59.47% |
| Wan(WoVR) | Short | 120 | 8 | 15 | 128 | 1,920 | 8 | 64.94% |
| Wan(WoVR) | Normal (paper) | 240 | 8 | 30 | 64 | 1,920 | 8 | 67.19% |
| Wan(WoVR) | Long | 480 | 8 | 60 | 32 | 1,920 | 8 | 55.18% |
Each checkpoint is evaluated over 1,024 episodes. The three horizon settings yield three findings:
- The default horizon is optimal for both backends. Both attain their highest peaks under Normal (1×): 70.70% for Ctrl-World and 68.65% for Wan. The horizon choices in the main table do not favor a particular model.
- The relative ordering is preserved across all three horizons and is not driven by interaction volume. Ctrl-World's peaks exceed Wan's by 4.7, 2.1, and 4.3 percentage points under Short, Normal, and Long, respectively. Under Normal, Wan generates more actions than Ctrl-World because of their different native chunk lengths (15,360 versus 12,800), yet Ctrl-World still leads.
- Excessive recursive depth exposes differences in long-horizon stability. Doubling recursive depth reduces the peaks of both backends by 11–13 percentage points. With continued training under Long, Wan's success decreases from 55.18% to 35.84%, below SFT's 55.08%, while Ctrl-World decreases from 59.47% to 55.96%, a much smaller decline. This agrees with the divergence under long-trajectory WMPO in W1 and demonstrates the need to test world models' long-horizon stability through interactive RL.
Table 2 reports each world model's utility under its native temporal interface and a fixed iteration budget. The updated manuscript now uses this description and includes the three sets of comparisons above.
Response (3/4): Training variability and independent evaluation
W3. Training randomness, checkpoint selection, and evaluation on independent configurations
- Multiple seeds and checkpoint-selection effects. We added training seeds 2345 and 3456 for Ctrl-World and WoVR on Adjust Bottle, each with 20 checkpoints × 1,024 evaluation episodes. Together with the original seed 1234, this gives three seeds. Per-seed results are provided in our response to Reviewer prVG's W1.
- Multiple seeds. The three-seed mean peaks are 69.53% for Ctrl-World and 67.38% for WoVR under the proxy reward, and 65.43% and 61.50%, respectively, under the similarity reward. The ordering is consistent and holds for every seed (6/6), with both models well above SFT (55.08%).
- Selection effects. Each checkpoint is evaluated over 1,024 episodes, giving a standard error of approximately 1.5 percentage points per estimate. The estimated upper bound on the optimism from selecting the peak among 20 checkpoints is approximately 3 percentage points. Without selection, the last checkpoints (step 20) under the proxy reward achieve 66.31%/63.18% for Ctrl-World and 60.06%/66.50% for WoVR, remaining 5–11 percentage points above SFT. In the updated manuscript, we have renamed the metric peak validation performance and report peak and last-checkpoint results in separate tables.
- Independent evaluation on unseen configurations and under paired appearance randomization. To eliminate peak-selection bias and address the absence of domain randomization during evaluation, we evaluated the peak checkpoint selected on validation configurations for each seed on two sets of test configurations that had never been used for training or peak selection, with 1,024 episodes per set:
- Test set A (unseen initial configurations): Newly sampled initial layouts outside the validation configurations.
- Test set B (paired appearance robustness test): Physical initial states are paired scene by scene with the validation configurations. Only the background-wall and tabletop textures are independently sampled from RoboTwin's unseen texture library, and the colors and intensities of one directional light and two point lights are randomized. Appearance is determined by the scene ID, so all policies encounter exactly the same scenes, and changes in success rates arise solely from visual appearance.
| Model | Unseen initial configurations (per seed) | Mean | Appearance randomization (per seed) | Mean |
|---|---|---|---|---|
| SFT | 49.12 | 49.12 | 30.18 | 30.18 |
| Ctrl-World | 62.21 / 60.45 / 59.67 | 60.77 | 39.94 / 44.63 / 44.04 | 42.87 |
| WoVR | 52.64 / 53.22 / 53.81 | 53.22 | 40.72 / 45.21 / 42.68 | 42.87 |
Success rates are percentages; the seed order is 1234 / 2345 / 3456. The results yield three findings:
- Policy gains persist on independent configurations. All 12 results across two models, three seeds, and two test sets exceed SFT. On unseen initial configurations, Ctrl-World's mean exceeds SFT by 11.7 percentage points and WoVR's by 4.1 percentage points (pooling three seeds, one-sided Fisher's exact test: p = 0.013).
- Gains are larger under visual distribution shift. With unseen textures and lighting, SFT drops to 30.18%, while Ctrl-World and WoVR both achieve a mean of 42.87%, each exceeding SFT by 12.7 percentage points (p < 10⁻¹²). Because physical states are paired scene by scene, this gap shows that training within world models does not cause policies to overfit the appearance of validation configurations and instead improves robustness to visual shifts.
- Peak-selection bias agrees with the estimate. The unseen initial configurations are themselves harder: even SFT, which involves no selection, drops from 55.08% to 49.12%. After accounting for this 6.0-percentage-point difficulty difference, the additional decrease in Ctrl-World's mean peak is approximately 2.8 percentage points (69.53% → 60.77%), consistent with the estimated upper bound of approximately 3 percentage points.
The leading-model ordering is clearer on unseen initial configurations: all three Ctrl-World seeds (minimum 59.67%) outperform all three WoVR seeds (maximum 53.81%). Under appearance randomization, the two models tie. Accordingly, we characterize Table 2 as peak validation performance and use the independent tests in this section as the basis for generalization conclusions.
- Real-robot validation. The tactile ablation uses identical training and evaluation settings, with n = 20 trials per task and condition. FastWAM improves from 3/20 to 9/20 on Chip and from 4/20 to 6/20 on Wiping. In this rebuttal, we additionally performed tactile ablations on Wiping and Tofu using three visuotactile models. Removing tactile input reduces success in all six comparisons, with a pooled one-sided Fisher's exact test giving p = 0.007 (see our response to Reviewer prVG's W3).
Response (1/4): System-level reference methods
W1. System-level comparisons with WMPO and VLA-MBPO
We thank the reviewer for this suggestion. We adapted the core optimization mechanisms of WMPO and VLA-MBPO to Ctrl-World, Wan, and OpenSora on Adjust Bottle, covering six configurations and 47 valid checkpoints.
All configurations share the same fine-tuned π0.5 initialization, single head-camera view, 14-dimensional absolute joint actions, frozen world models, and proxy reward. Native action chunk sizes are retained (Ctrl-World, 5; Wan and OpenSora, 8). Each checkpoint is evaluated over 1,024 episodes in the ground-truth RoboTwin simulator.
The results support three observations.
The updated manuscript cites WMPO [1] and VLA-MBPO [2] and identifies these implementations as adaptations under a unified interface.
[1] Fangqi Zhu, Zhengyang Yan, Zicong Hong, Quanxin Shou, Xiao Ma, and Song Guo. WMPO: World Model-based Policy Optimization for Vision-Language-Action Models. arXiv:2511.09515, 2025.
[2] Zhilong Zhang, Haoxiang Ren, Yihao Sun, Yifei Sheng, Haonan Wang, Haoxin Lin, Zhichao Wu, Pierre-Luc Bacon, and Yang Yu. Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models. arXiv:2603.20607, 2026.