- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents RoboWorld100, a dataset and benchmark for training and evaluating action-conditioned (video) world models. Unlike previous datasets and benchmarks, which focus on narrow task-specific behaviors (e.g., pick-and-place), it collects diverse, task-relevant human-play episodes spanning 104 task–platform settings, ten interaction categories, and nine collection platforms (both real and simulated). The paper also addresses the limitations of evaluation protocols that rely mainly on visual quality. It proposes two additional metrics: action consistency, measured with frozen platform-specific inverse dynamics models (IDMs), and task-progress consistency, which compares progress curves of generated and real videos estimated by Robometer. The paper further evaluates long-horizon rollout stability. Finally, it tests the world models as learned simulators, both for estimating policy success rates and for in-model reinforcement learning with a Robometer-based reward, with the resulting policies evaluated on a real robot.
Critical strengths
- The paper is generally well written and easy to follow.
- The benchmark focuses on evaluating action-conditioned world models in more diverse scenarios that can occur in the real world, across multiple embodiments and camera configurations.
- The additional evaluation protocols based on the IDM and Robometer are realistic. They help keep evaluation from focusing too much on visual fidelity rather than physical interactions.
- The paper clearly states its limitations.
Critical weaknesses
- The evaluation is mainly limited to Cosmos (Nano and Edge) and GigaWorld. I would like to see whether more baselines, such as DreamDojo, can be evaluated. This is particularly important here: since the paper introduces a new benchmark, future researchers should be able to compare against a wide range of (robotics) world models.
- The policy evaluation and RL experiments are too small in scale. They use only four tasks with ten trials each for success-rate estimation, and three tasks on a single platform (Tianji) for RL. Since this is a benchmark paper, I think more tasks, trials, and platforms should be included.
- It is great that the benchmark includes scenarios involving deformable objects. However, there is no specific protocol for evaluating whether a model improves in these settings, since the current metrics do not directly assess the correctness of contact or deformation (as the authors also acknowledge).
Questions for authors
- Could the authors provide more baselines?
5 comments
W1. Additional world-model baselines (2/2)
Table R1b. Platform-level Long rollout results, with 5 trajectories per task.
| Platform | Model | MAE ↓ | MSE ↓ | PSNR ↑ (dB) | SSIM ↑ | Frame FID ↓ | FVD ↓ | Position ↓ (mm) | Rotation ↓ (°) | Gripper ↓ (normalized) | Progress ↑ (0–100) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| ALOHA | Cosmos Nano | 0.0687 | 0.0180 | 17.48 | 0.6997 | 29.47 | 20.03 | 20.10 | 4.89 | 0.0382 | 88.24 |
| Cosmos Edge | 0.0637 | 0.0162 | 17.97 | 0.7113 | 32.88 | 17.10 | 19.62 | 4.91 | 0.0248 | 88.18 | |
| GigaWorld | 0.0765 | 0.0199 | 17.03 | 0.6845 | 40.27 | 28.05 | 23.82 | 5.94 | 0.0203 | 85.20 | |
| DreamDojo-2B | 0.0669 | 0.0176 | 17.56 | 0.7035 | 27.01 | 14.86 | 20.22 | 5.16 | 0.0249 | 91.13 | |
| Tianji | Cosmos Nano | 0.0514 | 0.0133 | 18.92 | 0.7401 | 51.46 | 23.51 | 13.67 | 3.33 | 0.0289 | 88.92 |
| Cosmos Edge | 0.0477 | 0.0119 | 19.56 | 0.7498 | 52.03 | 21.83 | 13.50 | 3.41 | 0.0285 | 86.19 | |
| GigaWorld | 0.0545 | 0.0130 | 18.94 | 0.7442 | 63.05 | 24.69 | 16.66 | 4.22 | 0.0325 | 86.45 | |
| DreamDojo-2B | 0.0472 | 0.0114 | 19.65 | 0.7572 | 55.15 | 18.12 | 14.94 | 3.93 | 0.0350 | 88.45 | |
| Flexiv | Cosmos Nano | 0.0939 | 0.0291 | 15.41 | 0.6172 | 39.87 | 17.37 | 19.75 | 2.97 | 0.0630 | 69.91 |
| Cosmos Edge | 0.0865 | 0.0263 | 15.91 | 0.6376 | 38.60 | 18.16 | 18.77 | 2.94 | 0.0646 | 71.94 | |
| GigaWorld | 0.0829 | 0.0208 | 16.92 | 0.6600 | 35.92 | 20.03 | 18.09 | 2.87 | 0.0575 | 73.25 | |
| DreamDojo-2B | 0.0855 | 0.0256 | 16.00 | 0.6334 | 38.81 | 16.02 | 17.78 | 2.78 | 0.0578 | 72.10 | |
| Astribot | Cosmos Nano | 0.0902 | 0.0280 | 15.63 | 0.5807 | 33.14 | 21.73 | 19.84 | 3.55 | 0.0216 | 86.16 |
| Cosmos Edge | 0.0785 | 0.0229 | 16.57 | 0.6032 | 34.27 | 19.61 | 15.24 | 2.17 | 0.0243 | 86.71 | |
| GigaWorld | 0.0891 | 0.0249 | 16.06 | 0.5941 | 48.27 | 26.79 | 16.35 | 2.29 | 0.0219 | 86.07 | |
| DreamDojo-2B | 0.0730 | 0.0206 | 16.95 | 0.6180 | 34.32 | 14.38 | 15.52 | 2.36 | 0.0369 | 90.60 | |
| Finger | Cosmos Nano | 0.1151 | 0.0360 | 14.54 | 0.4314 | 50.76 | 31.33 | 12.94 | 3.39 | 0.0417 | 89.76 |
| Cosmos Edge | 0.1247 | 0.0397 | 14.12 | 0.4084 | 66.84 | 47.46 | 13.27 | 3.52 | 0.0407 | 84.69 | |
| GigaWorld | 0.1380 | 0.0452 | 13.51 | 0.3975 | 80.76 | 56.08 | 18.76 | 4.23 | 0.0493 | 83.06 | |
| DreamDojo-2B | 0.1225 | 0.0387 | 14.23 | 0.4200 | 63.52 | 35.13 | 14.05 | 3.73 | 0.0481 | 83.79 | |
| Gripper | Cosmos Nano | 0.1002 | 0.0295 | 15.60 | 0.4545 | 25.73 | 17.57 | 15.86 | 1.76 | 0.0262 | 87.60 |
| Cosmos Edge | 0.1059 | 0.0319 | 15.19 | 0.4460 | 31.05 | 22.74 | 17.12 | 1.89 | 0.0227 | 88.69 | |
| GigaWorld | 0.1270 | 0.0409 | 14.11 | 0.3895 | 41.58 | 29.54 | 23.62 | 2.87 | 0.0278 | 88.55 | |
| DreamDojo-2B | 0.1090 | 0.0332 | 14.98 | 0.4399 | 32.10 | 20.33 | 16.94 | 2.00 | 0.0227 | 89.28 | |
| UMI Single | Cosmos Nano | 0.0568 | 0.0122 | 19.17 | 0.5923 | 19.64 | 9.60 | 12.44 | 1.75 | 0.0285 | 85.82 |
| Cosmos Edge | 0.0572 | 0.0121 | 19.19 | 0.6008 | 20.11 | 10.44 | 12.39 | 1.73 | 0.0260 | 84.77 | |
| GigaWorld | 0.0772 | 0.0181 | 17.52 | 0.5208 | 36.92 | 15.54 | 17.35 | 2.41 | 0.0440 | 83.07 | |
| DreamDojo-2B | 0.0549 | 0.0115 | 19.47 | 0.6158 | 23.84 | 11.16 | 13.43 | 1.93 | 0.0306 | 84.37 | |
| ARX simulator | Cosmos Nano | 0.0506 | 0.0170 | 17.90 | 0.7768 | 39.74 | 29.04 | 20.96 | 7.07 | 0.0516 | 84.44 |
| Cosmos Edge | 0.0444 | 0.0141 | 18.71 | 0.7917 | 36.81 | 25.48 | 20.99 | 7.47 | 0.0413 | 85.77 | |
| GigaWorld | 0.0549 | 0.0175 | 17.73 | 0.7524 | 44.81 | 31.27 | 23.34 | 7.94 | 0.0372 | 91.62 | |
| DreamDojo-2B | 0.0448 | 0.0145 | 18.58 | 0.7891 | 34.71 | 26.60 | 21.81 | 7.79 | 0.0508 | 85.24 | |
| Ego | Cosmos Nano | 0.0604 | 0.0149 | 18.44 | 0.6905 | 41.45 | 20.64 | — | — | — | 88.58 |
| Cosmos Edge | 0.0686 | 0.0172 | 17.83 | 0.6717 | 57.14 | 37.39 | — | — | — | 87.12 | |
| GigaWorld | — | — | — | — | — | — | — | — | — | — | |
| DreamDojo-2B | 0.0628 | 0.0157 | 18.26 | 0.6876 | 42.30 | 21.17 | — | — | — | 83.99 |
On Astribot, DreamDojo-2B has the lowest FVD and highest progress consistency on both splits. In long rollouts, Cosmos Edge still has lower position and rotation errors. DreamDojo-2B also leads Cosmos Nano in FVD and progress on ALOHA, while Cosmos Nano leads on Finger, illustrating platform- and metric-dependent rankings.
† The official GigaWorld interface lacks an Ego adapter. Camera and hand motion components for Ego are reported separately in Table R1c. Long-rollout evaluation uses continuous rollout trajectories.
Table R1c. Ego motion consistency (auxiliary metrics).
| Split | Model | Camera pos. ↓ (mm) | Camera rot. ↓ (°) | Left wrist pos. ↓ (mm) | Left wrist rot. ↓ (°) | Left fingertip ↓ (mm) | Right wrist pos. ↓ (mm) | Right wrist rot. ↓ (°) | Right fingertip ↓ (mm) |
|---|---|---|---|---|---|---|---|---|---|
| Evaluation | Cosmos Nano | 30.92 | 2.00 | 40.96 | 8.21 | 10.42 | 43.34 | 7.96 | 9.20 |
| Cosmos Edge | 34.53 | 2.17 | 46.80 | 9.27 | 12.54 | 48.72 | 9.23 | 10.85 | |
| GigaWorld | — | — | — | — | — | — | — | — | |
| DreamDojo-2B | 34.01 | 2.25 | 47.19 | 9.90 | 13.06 | 49.09 | 9.89 | 10.95 | |
| Long rollout | Cosmos Nano | 33.66 | 2.20 | 40.87 | 6.37 | 11.36 | 43.04 | 7.79 | 9.48 |
| Cosmos Edge | 36.81 | 2.33 | 46.33 | 7.46 | 13.24 | 48.06 | 8.54 | 10.63 | |
| GigaWorld | — | — | — | — | — | — | — | — | |
| DreamDojo-2B | 37.16 | 2.39 | 46.96 | 7.66 | 15.46 | 48.92 | 8.97 | 11.58 |
W2
We thank the reviewer for the suggestion regarding the scale of the downstream experiments. The core objective of RoboWorld100 is to systematically evaluate action-conditioned world models. Our main evaluation covers ten interaction categories and nine collection platforms, comparing models in terms of video quality, action consistency, and task progress. Real-robot policy evaluation and in-model RL further examine the utility of these models for policy evaluation and optimization, so we conduct focused application experiments on Tianji.
On the evaluated tasks, the Play simulator reduces the mean absolute gap between simulated and real-robot success rates from 40 to 15 percentage points, and in-model RL increases mean real-robot success from 26.7% to 63.3%. These results provide empirical evidence for the downstream utility of world models. Expanding real-robot validation requires additional platform adaptation, robot deployment, and repeated trials, incurring substantial resource and time costs. We therefore focus this part of the study on the evaluated tasks. We agree that broader real-robot experiments would further assess the generalizability of these gains, and we will clarify the purpose and scope of the application experiments in the revised manuscript.
W3
We thank the reviewer for highlighting the limitations of contact and deformation evaluation. We agree with this concern and have explicitly acknowledged it in the limitations discussion of the paper. The current protocol measures consistency between generated results and reference executions in terms of video quality, action consistency, and task progress, but still lacks direct validation of the physical correctness of contact and deformation. The deformable-object scenarios in RoboWorld100 provide data and test settings for studying these issues. Developing dedicated evaluations of contact relationships, object interpenetration, and deformation continuity is an important direction for further improving the benchmark. We identify this as future work and limit our current evaluation conclusions to the consistency measured by the above metrics.
W1. Additional world-model baselines (1/2)
We thank the reviewer for suggesting additional world-model baselines. We initialized DreamDojo-2B from its official pretrained weights and post-trained it on RoboWorld100 data. We have completed the full Evaluation and Long rollout assessments across all nine platforms, using the same held-out trajectories and evaluators as the original baselines. Post-training, video generation, and full-metric evaluation required substantial computational resources and time.
The following platform-level results are aggregated with equal weights across tasks within each platform. FVD uses R3D-18 features, bold indicates the best mean values, and — indicates missing or inapplicable results. Action-consistency scores for Gripper use the original benchmark samples.
Table R1a. Platform-level Evaluation results, with 20 trajectories per task.
Table R1a shows that the additional baseline performs differently across evaluation dimensions. On Tianji, for example, DreamDojo-2B achieves better FVD than Cosmos Nano (9.77 vs. 13.01), but has higher position error (11.36 vs. 10.05 mm) and lower progress consistency (91.46 vs. 92.04). This further indicates that visual quality alone is insufficient to assess how well a model reproduces motion and task progression.