acceptodds
Under review as a conference paper at ICLR 2027

RoboWorld100: Learning and Evaluating Robot World Models from Diverse Play

Abstract

Action‑conditioned world models promise learned simulators for robot policy evaluation and reinforcement learning, but must capture physical interactions beyond the execution patterns of individual tasks. Task‑relevant play broadens interaction coverage beyond prescribed solutions, yet existing play datasets remain limited in task, robot platform, and observation diversity. We introduce RoboWorld100, a framework and benchmark integrating human‑play data collection with world‑model training and evaluation. It covers ten categories of manipulation and physical interaction and over 100 task‑platform settings across real and simulated environments, spanning diverse robots, collection schemes, and camera configurations. Beyond visual quality, our evaluation measures action consistency and task‑progress consistency to assess whether predictions faithfully reflect supplied actions and execution outcomes. Closed‑loop policy evaluation and in‑model reinforcement learning further examine whether learned dynamics support policy optimization and whether improvements transfer to real robots.

Primary area:
datasets and benchmarks
Keywords:
robot world models, human play, robotic manipulation, action-conditioned video prediction, embodied AI, world model evaluation, model-based reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CoFbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RoboWorld100, a dataset and benchmark for training and evaluating action-conditioned (video) world models. Unlike previous datasets and benchmarks, which focus on narrow task-specific behaviors (e.g., pick-and-place), it collects diverse, task-relevant human-play episodes spanning 104 task–platform settings, ten interaction categories, and nine collection platforms (both real and simulated). The paper also addresses the limitations of evaluation protocols that rely mainly on visual quality. It proposes two additional metrics: action consistency, measured with frozen platform-specific inverse dynamics models (IDMs), and task-progress consistency, which compares progress curves of generated and real videos estimated by Robometer. The paper further evaluates long-horizon rollout stability. Finally, it tests the world models as learned simulators, both for estimating policy success rates and for in-model reinforcement learning with a Robometer-based reward, with the resulting policies evaluated on a real robot.

Critical strengths
  • The paper is generally well written and easy to follow.
  • The benchmark focuses on evaluating action-conditioned world models in more diverse scenarios that can occur in the real world, across multiple embodiments and camera configurations.
  • The additional evaluation protocols based on the IDM and Robometer are realistic. They help keep evaluation from focusing too much on visual fidelity rather than physical interactions.
  • The paper clearly states its limitations.
Critical weaknesses
  • The evaluation is mainly limited to Cosmos (Nano and Edge) and GigaWorld. I would like to see whether more baselines, such as DreamDojo, can be evaluated. This is particularly important here: since the paper introduces a new benchmark, future researchers should be able to compare against a wide range of (robotics) world models.
  • The policy evaluation and RL experiments are too small in scale. They use only four tasks with ten trials each for success-rate estimation, and three tasks on a single platform (Tianji) for RL. Since this is a benchmark paper, I think more tasks, trials, and platforms should be included.
  • It is great that the benchmark includes scenarios involving deformable objects. However, there is no specific protocol for evaluating whether a model improves in these settings, since the current metrics do not directly assess the correctness of contact or deformation (as the authors also acknowledge).
Questions for authors
  • Could the authors provide more baselines?
5 comments

W1. Additional world-model baselines (1/2)

We thank the reviewer for suggesting additional world-model baselines. We initialized DreamDojo-2B from its official pretrained weights and post-trained it on RoboWorld100 data. We have completed the full Evaluation and Long rollout assessments across all nine platforms, using the same held-out trajectories and evaluators as the original baselines. Post-training, video generation, and full-metric evaluation required substantial computational resources and time.

The following platform-level results are aggregated with equal weights across tasks within each platform. FVD uses R3D-18 features, bold indicates the best mean values, and — indicates missing or inapplicable results. Action-consistency scores for Gripper use the original benchmark samples.

Table R1a. Platform-level Evaluation results, with 20 trajectories per task.

PlatformModelMAE ↓MSE ↓PSNR ↑ (dB)SSIM ↑Frame FID ↓FVD ↓Position ↓ (mm)Rotation ↓ (°)Gripper ↓ (normalized)Progress ↑ (0–100)
ALOHACosmos Nano0.06140.016018.080.725020.7812.0418.144.920.039290.89
Cosmos Edge0.05740.014318.590.736520.7410.8718.495.040.022390.38
GigaWorld0.07130.018417.420.704430.4522.7624.286.150.020784.75
DreamDojo-2B0.06460.017317.720.715723.1111.3819.845.420.022890.24
TianjiCosmos Nano0.03160.007121.660.829828.9713.0110.052.640.025392.04
Cosmos Edge0.02850.005922.470.841630.7911.4910.142.720.023591.06
GigaWorld0.03760.008021.040.815342.0916.0613.013.490.024689.06
DreamDojo-2B0.03070.006422.070.838736.919.7711.363.110.027991.46
FlexivCosmos Nano0.06640.018417.500.688219.568.0616.972.720.034081.46
Cosmos Edge0.06150.016517.980.703520.888.4816.662.670.033080.53
GigaWorld0.06420.014818.360.711822.8510.9917.452.780.039578.57
DreamDojo-2B0.06690.018317.490.686324.457.5516.862.740.033683.45
AstribotCosmos Nano0.07320.021416.770.622819.008.3617.352.840.012590.89
Cosmos Edge0.05810.015318.280.662818.848.0814.671.910.012491.04
GigaWorld0.07610.020216.980.630237.2715.6316.272.070.013489.36
DreamDojo-2B0.06060.016417.970.656024.517.6614.932.100.024192.44
FingerCosmos Nano0.07520.019617.200.550024.1512.436.751.710.018492.52
Cosmos Edge0.07930.020716.990.538832.2919.357.191.780.018892.22
GigaWorld0.08660.023416.430.523544.0520.5711.532.600.029188.11
DreamDojo-2B0.08060.021816.780.534631.0412.547.631.930.021992.37
GripperCosmos Nano0.06480.015618.360.576215.087.959.481.100.012694.27
Cosmos Edge0.06870.016918.010.565619.8110.9510.351.210.011893.55
GigaWorld0.09500.026815.900.485231.2118.9016.402.050.017091.83
DreamDojo-2B0.07200.018317.630.553322.5810.5310.981.350.012793.67
UMI SingleCosmos Nano0.03440.005622.660.74739.255.078.151.160.018290.44
Cosmos Edge0.03270.005123.190.76679.565.888.411.160.018589.92
GigaWorld0.05090.009820.130.639020.008.5813.531.840.030687.61
DreamDojo-2B0.03910.006921.750.712615.717.259.671.260.021890.53
ARX simulatorCosmos Nano0.03620.010819.870.810917.7210.6714.704.650.018794.03
Cosmos Edge0.03150.008620.800.826918.399.2714.864.810.018894.40
GigaWorld0.04520.012919.010.776031.7519.4618.165.600.021793.62
DreamDojo-2B0.03460.010220.040.815218.1310.3215.685.240.023093.67
EgoCosmos Nano0.04300.010020.130.753320.7210.54———90.39
Cosmos Edge0.04480.010120.070.752830.7521.75———89.67
GigaWorld——————————
DreamDojo-2B0.04520.010719.910.741323.0210.58———89.37

Table R1a shows that the additional baseline performs differently across evaluation dimensions. On Tianji, for example, DreamDojo-2B achieves better FVD than Cosmos Nano (9.77 vs. 13.01), but has higher position error (11.36 vs. 10.05 mm) and lower progress consistency (91.46 vs. 92.04). This further indicates that visual quality alone is insufficient to assess how well a model reproduces motion and task progression.

W1. Additional world-model baselines (2/2)

Table R1b. Platform-level Long rollout results, with 5 trajectories per task.

PlatformModelMAE ↓MSE ↓PSNR ↑ (dB)SSIM ↑Frame FID ↓FVD ↓Position ↓ (mm)Rotation ↓ (°)Gripper ↓ (normalized)Progress ↑ (0–100)
ALOHACosmos Nano0.06870.018017.480.699729.4720.0320.104.890.038288.24
Cosmos Edge0.06370.016217.970.711332.8817.1019.624.910.024888.18
GigaWorld0.07650.019917.030.684540.2728.0523.825.940.020385.20
DreamDojo-2B0.06690.017617.560.703527.0114.8620.225.160.024991.13
TianjiCosmos Nano0.05140.013318.920.740151.4623.5113.673.330.028988.92
Cosmos Edge0.04770.011919.560.749852.0321.8313.503.410.028586.19
GigaWorld0.05450.013018.940.744263.0524.6916.664.220.032586.45
DreamDojo-2B0.04720.011419.650.757255.1518.1214.943.930.035088.45
FlexivCosmos Nano0.09390.029115.410.617239.8717.3719.752.970.063069.91
Cosmos Edge0.08650.026315.910.637638.6018.1618.772.940.064671.94
GigaWorld0.08290.020816.920.660035.9220.0318.092.870.057573.25
DreamDojo-2B0.08550.025616.000.633438.8116.0217.782.780.057872.10
AstribotCosmos Nano0.09020.028015.630.580733.1421.7319.843.550.021686.16
Cosmos Edge0.07850.022916.570.603234.2719.6115.242.170.024386.71
GigaWorld0.08910.024916.060.594148.2726.7916.352.290.021986.07
DreamDojo-2B0.07300.020616.950.618034.3214.3815.522.360.036990.60
FingerCosmos Nano0.11510.036014.540.431450.7631.3312.943.390.041789.76
Cosmos Edge0.12470.039714.120.408466.8447.4613.273.520.040784.69
GigaWorld0.13800.045213.510.397580.7656.0818.764.230.049383.06
DreamDojo-2B0.12250.038714.230.420063.5235.1314.053.730.048183.79
GripperCosmos Nano0.10020.029515.600.454525.7317.5715.861.760.026287.60
Cosmos Edge0.10590.031915.190.446031.0522.7417.121.890.022788.69
GigaWorld0.12700.040914.110.389541.5829.5423.622.870.027888.55
DreamDojo-2B0.10900.033214.980.439932.1020.3316.942.000.022789.28
UMI SingleCosmos Nano0.05680.012219.170.592319.649.6012.441.750.028585.82
Cosmos Edge0.05720.012119.190.600820.1110.4412.391.730.026084.77
GigaWorld0.07720.018117.520.520836.9215.5417.352.410.044083.07
DreamDojo-2B0.05490.011519.470.615823.8411.1613.431.930.030684.37
ARX simulatorCosmos Nano0.05060.017017.900.776839.7429.0420.967.070.051684.44
Cosmos Edge0.04440.014118.710.791736.8125.4820.997.470.041385.77
GigaWorld0.05490.017517.730.752444.8131.2723.347.940.037291.62
DreamDojo-2B0.04480.014518.580.789134.7126.6021.817.790.050885.24
EgoCosmos Nano0.06040.014918.440.690541.4520.64———88.58
Cosmos Edge0.06860.017217.830.671757.1437.39———87.12
GigaWorld——————————
DreamDojo-2B0.06280.015718.260.687642.3021.17———83.99

On Astribot, DreamDojo-2B has the lowest FVD and highest progress consistency on both splits. In long rollouts, Cosmos Edge still has lower position and rotation errors. DreamDojo-2B also leads Cosmos Nano in FVD and progress on ALOHA, while Cosmos Nano leads on Finger, illustrating platform- and metric-dependent rankings.

† The official GigaWorld interface lacks an Ego adapter. Camera and hand motion components for Ego are reported separately in Table R1c. Long-rollout evaluation uses continuous rollout trajectories.

Table R1c. Ego motion consistency (auxiliary metrics).

SplitModelCamera pos. ↓ (mm)Camera rot. ↓ (°)Left wrist pos. ↓ (mm)Left wrist rot. ↓ (°)Left fingertip ↓ (mm)Right wrist pos. ↓ (mm)Right wrist rot. ↓ (°)Right fingertip ↓ (mm)
EvaluationCosmos Nano30.922.0040.968.2110.4243.347.969.20
Cosmos Edge34.532.1746.809.2712.5448.729.2310.85
GigaWorld————————
DreamDojo-2B34.012.2547.199.9013.0649.099.8910.95
Long rolloutCosmos Nano33.662.2040.876.3711.3643.047.799.48
Cosmos Edge36.812.3346.337.4613.2448.068.5410.63
GigaWorld————————
DreamDojo-2B37.162.3946.967.6615.4648.928.9711.58

W2

We thank the reviewer for the suggestion regarding the scale of the downstream experiments. The core objective of RoboWorld100 is to systematically evaluate action-conditioned world models. Our main evaluation covers ten interaction categories and nine collection platforms, comparing models in terms of video quality, action consistency, and task progress. Real-robot policy evaluation and in-model RL further examine the utility of these models for policy evaluation and optimization, so we conduct focused application experiments on Tianji.

On the evaluated tasks, the Play simulator reduces the mean absolute gap between simulated and real-robot success rates from 40 to 15 percentage points, and in-model RL increases mean real-robot success from 26.7% to 63.3%. These results provide empirical evidence for the downstream utility of world models. Expanding real-robot validation requires additional platform adaptation, robot deployment, and repeated trials, incurring substantial resource and time costs. We therefore focus this part of the study on the evaluated tasks. We agree that broader real-robot experiments would further assess the generalizability of these gains, and we will clarify the purpose and scope of the application experiments in the revised manuscript.

W3

We thank the reviewer for highlighting the limitations of contact and deformation evaluation. We agree with this concern and have explicitly acknowledged it in the limitations discussion of the paper. The current protocol measures consistency between generated results and reference executions in terms of video quality, action consistency, and task progress, but still lacks direct validation of the physical correctness of contact and deformation. The deformable-object scenarios in RoboWorld100 provide data and test settings for studying these issues. Developing dedicated evaluations of contact relationships, object interpenetration, and deformation continuity is an important direction for further improving the benchmark. We identify this as future work and limit our current evaluation conclusions to the consistency measured by the above metrics.

Reviewer_Q8GMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

RoboWorld100 introduces a task-relevant human-play dataset and evaluation framework for action-conditioned robot world models, covering 104 task–platform settings, ten interaction categories, and nine collection platforms. The benchmark evaluates visual fidelity, action consistency through platform-specific inverse dynamics models, and task-progress consistency using Robometer, alongside continuous long-horizon rollouts. Experiments compare three pretrained world models, investigate the benefits of play over task-centered data, and demonstrate applications to policy evaluation and in-model reinforcement learning with real-robot transfer.

Critical strengths

Broad and useful interaction coverage. The dataset includes precision manipulation, deformable objects, occlusion, tool use, and dynamic interactions across robot, handheld, and egocentric collection setups. This breadth provides a potentially valuable resource for studying world-model limitations.

Evaluation beyond visual similarity. Separately measuring visual fidelity, motion agreement, and task progress exposes model-ranking differences that image metrics alone would miss. Continuous rollouts without resets further reveal degradation over time.

Encouraging downstream evidence. Play-trained simulators reduce the reported mean policy-success estimation error from 40 to 15 percentage points. In-model RL also improves average real-robot success from 26.7% to 63.3% across three tasks.

Careful aspects of the protocol. Source-episode splits, training-only normalization, final-checkpoint evaluation, and paired trajectory bootstrap intervals improve experimental transparency.

Critical weaknesses

The proposed evaluators lack independent validation. IDM and progress scores compare two model predictions, but the paper does not establish their accuracy against recorded motion or human judgments on this benchmark. The play-data comparison is not matched for training exposure. Appendix C.3 explicitly states that episode lengths differ, so matching episodes and epochs does not match frame exposure or optimizer updates. The results therefore do not isolate the benefit of broader interaction coverage. (Table 4; Appendix C.3.) Action semantics limit the interpretation of simulator fidelity. Some platforms condition on measured future states or reconstructed motion rather than executable commands. These settings evaluate prediction given realized motion, leaving command-to-motion errors outside the model. Their relationship to closed-loop robot simulation needs clearer qualification. (Table 9; Appendix B.)

Questions for authors

How accurate are the IDM and progress evaluators against independent ground truth? Do play-data gains persist under matched frame and update budgets? How does RL transfer compare when using a task-centered simulator? How are future-state conditions obtained during closed-loop deployment?

5 comments

Q1. Independent validation of the evaluators

We thank the reviewer for requesting independent validation. We added IDM evaluation against source motion labels and human validation of progress-consistency rankings.

IDM. As described in the paper, action consistency compares motion inferred from generated and reference videos by the same frozen IDM. We validated the IDMs on 1,480 original recordings across nine platforms, disjoint from their training and checkpoint-selection sets, with checkpoints and hyperparameters fixed. Ego uses reconstructed motion labels.

Table R2. Motion-recovery errors (mean [95% CI]). We average within recordings and tasks, with equal task weights per platform. CIs use 20,000 recording-level bootstrap resamples within fixed tasks. UMI Single and ARX use held-out Long rollout. Ego components share 120 recordings, counted once.

Platform / componentRecordingsPosition (mm) ↓Rotation (°) ↓Gripper (% of range) ↓
ALOHA28020.22 [19.74, 20.72]5.98 [5.84, 6.12]1.33 [1.28, 1.38]
Astribot36017.78 [17.44, 18.13]2.24 [2.17, 2.31]1.41 [1.36, 1.45]
Tianji40010.81 [10.56, 11.05]2.84 [2.76, 2.91]1.87 [1.79, 1.96]
Flexiv8017.41 [16.79, 18.04]3.00 [2.86, 3.15]4.03 [3.71, 4.36]
Gripper12019.24 [18.65, 19.83]2.19 [2.15, 2.24]2.96 [2.83, 3.10]
Finger8011.08 [10.61, 11.57]2.81 [2.71, 2.92]3.67 [3.36, 4.01]
UMI Single2013.75 [12.52, 15.16]2.08 [1.94, 2.22]1.78 [1.72, 1.84]
ARX (sim.)2020.98 [19.00, 22.91]7.99 [7.17, 8.87]2.69 [2.45, 2.96]
Ego camera12018.60 [17.37, 19.87]1.93 [1.84, 2.02]—
Ego wrists12033.15 [32.30, 34.00]8.61 [8.37, 8.87]—
Ego fingertips1209.67 [9.42, 9.94]——

Table R2b. Position error (mm) against a zero-motion baseline and correlation between IDM-estimated and source-label motion distances.

PlatformIDM position errorZero-motion position errorDistance Spearman ρ
ALOHA20.2246.520.910
Astribot17.7836.580.855
Tianji10.8123.450.882
Flexiv17.4130.990.869
Gripper19.2453.350.932
Finger11.0827.610.942
UMI Single13.7534.580.871
ARX (sim.)20.9836.250.762

We compare motion distances between up to 128 predetermined pairs of non-overlapping windows per recording. Spearman correlations are computed within recordings and averaged across recordings and tasks. Lower errors than zero motion and correlations of 0.762–0.942 support motion recovery and sensitivity to motion differences on these recordings.

Progress consistency. We sampled two trajectories per task from 26 tasks across nine platforms and ten interaction categories, independently of automatic scores. Three annotators independently judged anonymized predictions, choosing which better matched the reference's task progression, including the timing of progress, stalls, failures, and recovery. Of 52 comparisons, 37 had a decisive majority preference and 15 were majority ties. The metric agreed with 35/37 decisive preferences (94.6%), supporting relative ranking on these cases.

Table R3a. Agreement with human progress-consistency judgments.

Rating sourceA/B preferencesTiesAgreements / A/B preferencesAgreement
Rater 1371533/3789.2%
Rater 244840/4490.9%
Rater 345740/4588.9%
Majority vote371535/3794.6%

The 95% CI is [86.8%, 100%], using 10,000 task-cluster bootstrap resamples that retain trajectories and votes within each task and condition on the three annotators.

Table R3b. Inter-rater agreement across all 52 comparisons (A/B/tie).

Rater pairIdentical judgmentsExact agreement
1 and 239/5275.0%
1 and 338/5273.1%
2 and 329/5255.8%

All three annotators agreed on 27/52 pairs. Fleiss' κ is 0.496.

Table R3c. Cases where the human majority and metric ranking disagree.

CaseTaskRater 1Rater 2Rater 3Human majorityMetric preferenceScore gap (0–100)
3Gripper — Block a rolling ball with a plate.BBABA0.19
8Flexiv — Pour tea leaves into a teapot and close the lid.BABBA1.28

Q2. Play-data gains under matched training budgets

We thank the reviewer for raising the issue of training-budget fairness. Given the available compute and rebuttal timeline, we selected two Tianji tasks, wiping a whiteboard and pouring balls, from the original six-task comparison. The tasks were selected before obtaining the new results and cover opposite directions of the original episode-length imbalance, with longer Play episodes for wiping and longer Task-centered episodes for pouring.

We trained four Cosmos Edge models, one per task and data source, using a single training seed. Within each task, we matched retained frames, cumulative frame exposure, and completed optimizer updates.

Matched budgets and shared settings

Each data source contributes 100 trajectories per task. Trajectories are paired by length and cropped into contiguous segments to match the segment-length distribution and number of training windows. Both conditions share initialization, trainable modules, and the learning-rate schedule. Normalization statistics are computed exclusively from the matched training data. The batch size is 8, each window contains 33 frames and 32 transitions, and each window is used five times.

Budget per data source (matched within each task)Wipe whiteboardPour balls
Training trajectories100100
Retained frames17,16017,080
Training windows13,96013,880
Optimizer updates8,7258,675
Cumulative frame exposure2,303,4002,290,200
Total training tokens145,090,875144,384,830

Both conditions use the same 20 Evaluation and five Long rollout trajectories per task and the same frozen evaluators.

Complete Evaluation results

MetricWipe / Task-centeredWipe / PlayPour / Task-centeredPour / PlayMean / Task-centeredMean / Play
PSNR (dB) ↑20.7622.6416.4318.1618.5920.40
SSIM ↑0.81890.85930.68520.73180.75210.7956
FVD (R3D-18) ↓23.0314.1438.9417.8830.9916.01
Position (mm) ↓14.4612.2516.6012.2415.5312.24
Rotation (°) ↓3.012.864.282.983.642.92
Gripper (norm.) ↓0.03950.01850.10130.02340.07040.0209
Progress (0–100) ↑90.3490.8980.3688.5385.3589.71
MAE ↓0.03910.02810.07270.05320.05590.0407
MSE ↓0.00940.00600.02320.01570.01630.0108
Frame FID ↓34.8324.0773.3849.6454.1036.85

The two-task Evaluation averages improve across visual, action, and progress metrics under matched data and training budgets. Mean progress consistency increases by 4.36 points, with a paired trajectory-bootstrap 95% CI of [1.94, 6.84].

We first average within each task and then give the two tasks equal weight. Frame FID uses frame sets and FVD uses clip sets within each task and split. FVD uses R3D-18 features, MAE and MSE use pixels in [0,1], and gripper errors are normalized by the IDM training range.

Supplementary Long rollout results

MetricWipe / Task-centeredWipe / PlayPour / Task-centeredPour / PlayMean / Task-centeredMean / Play
PSNR (dB) ↑16.3418.7314.7316.1415.5417.44
SSIM ↑0.72480.76540.64470.64660.68470.7060
FVD (R3D-18) ↓52.2629.6857.4636.1054.8632.89
Position (mm) ↓22.1918.4817.4113.6719.8016.07
Rotation (°) ↓4.383.934.813.354.593.64
Gripper (norm.) ↓0.14790.02960.14380.04910.14580.0393
Progress (0–100) ↑83.8784.4178.0176.1480.9480.27
MAE ↓0.07270.05120.09390.07510.08330.0632
MSE ↓0.02340.01350.03370.02450.02850.0190
Frame FID ↓64.7150.7198.3876.0881.5563.39

Long-rollout visual and action metrics also improve, but progress consistency does not show a clear gain: the two-task mean difference (Play minus Task-centered) is −0.67 points, with a paired 95% CI of [−5.71, 3.92].

The progress-gain intervals use 10,000 paired trajectory-bootstrap resamples within each fixed task. They reflect test-trajectory uncertainty conditional on the fixed tasks and trained checkpoints, rather than variability across training seeds.

Q3. RL transfer from a Task-centered simulator

We thank the reviewer for suggesting this comparison. We have now completed the same DSRL procedure in Task-centered and Play-trained simulators on three Tianji tasks. Within each task, the two conditions share the same base policy, DSRL settings, simulator-training budget, and interaction budget. We evaluate each resulting policy on the real robot using 20 trials per task. The results are shown below.

Real-robot success before and after DSRL in the two simulators.

TaskBase policyTask-centered + DSRLPlay + DSRL
Pour3/20 (15%)8/20 (40%)12/20 (60%)
Sweep2/20 (10%)2/20 (10%)6/20 (30%)
Dice11/20 (55%)10/20 (50%)20/20 (100%)
Mean success rate26.7%33.3%63.3%

Relative to the base policy, DSRL in the Task-centered simulator improves mean real-robot success by 6.7 percentage points, while DSRL in the Play-trained simulator improves it by 36.7 points. Play achieves higher observed real-robot success on all three tasks, exceeding Task-centered by 30 percentage points on average. Under these matched settings, the results support stronger real-robot transfer from policy optimization in the simulator trained on human-collected, task-relevant play.

Q4. Conditioning and state updates in closed-loop deployment

We thank the reviewer for asking how conditioning is obtained during closed-loop deployment. The reported Tianji experiments condition the world model on policy-generated absolute joint-position and gripper targets, using the same action representation and normalization as in training. No future measured states are used. We initialize the loop with a real image and proprioceptive state. At each iteration, the final generated observation corresponding to the current action segment is fed back to the policy. The policy’s next proprioceptive input is set to the final joint-position and gripper targets in that action segment, under a target-tracking approximation that neglects controller latency and tracking error. Object changes and task progression are predicted by the world model and fed back through the generated images.

Reviewer_DKscOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RoboWorld100, a human-play dataset of 104 task–platform settings across nine platforms (arms, handheld grippers, UMI, egocentric hands, simulation) and ten interaction categories. It fine-tunes Cosmos Nano, Cosmos Edge and GigaWorld per platform and scores them on visual metrics, IDM-based action consistency, and Robometer-based progress consistency, including a three-stage long-horizon analysis. It claims that play beats task-centered data at equal episode counts (Table 4), that a play-trained simulator estimates policy success better (40 to 15 pp gap, Table 5), and that DSRL inside the world model raises real-robot success from 27% to 63% on three Tianji tasks (Table 6).

Critical strengths
  1. The dataset is much broader than prior play datasets. It spans nine platforms with native action interfaces (arms, handheld grippers, UMI, egocentric hands, simulation), 104 task–platform settings and retained failures. PlayWorld, the closest prior work, uses one robot. A cross-embodiment play dataset for adapting world models does not yet exist, and this would fill that gap if released.
  2. The open-loop evaluation protocol is careful. Fixed episode-level splits, paired generated/reference rollouts under matched actions, a separate long-horizon split, and paired bootstrap intervals (App. D.1) make the world-model comparisons in Tables 2–3 reproducible.
Critical weaknesses

In short: the dataset is the contribution. The downstream claims that play data helps repeat PlayWorld's experiments (Yin et al., 2026) with weaker controls, so they add little beyond what PlayWorld showed. The learned evaluators behind the new metrics are never validated.

  1. The play-data experiments (Tables 4–6) follow PlayWorld's template but drop the controls that made PlayWorld's results convincing. PlayWorld already showed, on one robot, that play-trained video world models predict contact better, estimate policy success better, and support in-model RL (DSRL) that transfers to hardware. RoboWorld100 runs the same three experiments:

    PlayWorldRoboWorld100
    Data matchingEqual hours (6h vs. 6h)Equal episode count (100); frames and updates unmatched (App. C.3); task set not stated
    Test data533 held-out clips from 20+ policies' rollouts, incl. failures20 Evaluation trajectories from the play-collected split, so in-distribution for the play model only
    Policy evaluation18 policies × 20 real trials; correlation with real success1 policy per task × 4 tasks × 10 trials; absolute gap only. App. F.1 says ranking "requires multiple checkpoints" and describes those controls in the future tense
    In-model RL baselineSame RL inside the demo-trained world modelBase SFT policy only; no RL inside the task-centered model

    Each change weakens the attribution to play data. With 10 trials, Table 5's binomial intervals are about ±30 points, wider than the 40→15 gap. Without RL in the task-centered model, Table 6 shows that in-model RL helps, not that play data helps.

    The protocol also falls below the standard of other work the paper cites. WorldEval (Li et al., 2025b) validates its evaluator with 4 policies and 40 rollouts per task, reporting Pearson r and rank violations (MMRV). RoboWorld (Jeon et al., 2026) and GigaWorld-1 report correlation or ranking across policies. DiWA (Chandra et al., CoRL 2025) already showed that a world model trained on play data supports offline RL that transfers to a real robot. Needed: Adopt PlayWorld's controls: a test set of held-out policy rollouts independent of both data sources, matched frames or hours, several checkpoints per task with rank correlation, and DSRL inside the task-centered world model. Discuss what the cross-embodiment results show that PlayWorld did not.

  2. Neither the IDM nor Robometer is validated, yet the paper's metrics depend on both. The paper argues that action and progress consistency reveal differences visual metrics miss (Sec. 4.1). It never reports IDM error on real held-out video against ground-truth actions, so a Nano–Edge gap of 11.07 vs. 11.43 mm cannot be compared with the evaluator's own noise. Progress scores sit at 89–92 for every model (Table 12), and Robometer also supplies the RL reward. KineBench (Liu et al., 2026b), which the paper cites, argues that learned IDMs are brittle outside their training data and make world-model errors indistinguishable from extractor errors. That is the failure mode the paper's action-consistency metric is exposed to, and the paper does not address it. Needed: Report per-platform IDM error on real test video and Robometer agreement with human success labels on real and generated clips. Show whether these metrics rank simulators in the same order as closed-loop success.

Questions for authors
  1. Beyond embodiment breadth, what does RoboWorld100 show about play data that PlayWorld did not? How does Table 4 change on a test set of held-out policy rollouts rather than play-collected episodes?
  2. Can Table 5 be repeated with several checkpoints per task and more trials, reporting rank correlation with real success as PlayWorld does?
  3. What does the same DSRL procedure achieve inside the task-centered world model (Table 6)?
  4. What is each platform's IDM error on real held-out video? This sets the noise floor for Tables 2–3.
7 comments

W1.

We thank the reviewer for the suggestions on the controls in the Play-data comparison. To address the mismatch in training exposure, we have added a comparison with matched data quantities and training budgets. We address the independent policy test set, policy success-rate ranking, and the DSRL comparison in a Task-centered simulator in Q1–Q3, respectively.

Given the available compute and time, we selected two Tianji tasks in advance from the original six-task comparison, wiping a whiteboard and pouring balls. These cover the cases where the original Play trajectories were longer and where the original Task-centered trajectories were longer, respectively. We trained four Cosmos Edge models using a single training seed, matching retained frames, cumulative frame exposure, and optimizer updates within each task. Both conditions share initialization, trainable modules, and the learning-rate schedule.

Table R1. Training settings for the matched-budget comparison. Values give the budget for each data source.

Training budget per data source (matched within each task)Wipe whiteboardPour balls
Training trajectories100100
Retained frames17,16017,080
Training windows13,96013,880
Optimizer updates8,7258,675
Cumulative frame exposure2,303,4002,290,200
Total training tokens145,090,875144,384,830

Each data source contributes 100 trajectories per task. Trajectories are paired by length and cropped into contiguous segments to match the segment-length distribution and the number of training windows. The batch size is 8, each window contains 33 frames and 32 transitions, and each window is used five times. Normalization statistics are computed separately using only the corresponding matched training data. Both conditions use the same 20 Evaluation and 5 Long rollout trajectories and the same frozen evaluators.

Table R2. Evaluation results after training with matched budgets.

MetricWipe whiteboard / Task-centeredWipe whiteboard / PlayPour balls / Task-centeredPour balls / PlayTwo-task mean / Task-centeredTwo-task mean / Play
PSNR (dB) ↑20.7622.6416.4318.1618.5920.40
SSIM ↑0.81890.85930.68520.73180.75210.7956
FVD (R3D-18) ↓23.0314.1438.9417.8830.9916.01
Position (mm) ↓14.4612.2516.6012.2415.5312.24
Rotation (°) ↓3.012.864.282.983.642.92
Gripper (normalized) ↓0.03950.01850.10130.02340.07040.0209
Progress consistency (0–100) ↑90.3490.8980.3688.5385.3589.71
MAE ↓0.03910.02810.07270.05320.05590.0407
MSE ↓0.00940.00600.02320.01570.01630.0108
Frame FID ↓34.8324.0773.3849.6454.1036.85

After controlling for data quantity and training budget, the two-task Evaluation averages improve across visual, action, and progress metrics. Mean progress consistency increases by 4.36 points, with a 95% confidence interval of [1.94, 6.84] from paired trajectory resampling. This supports the persistence of Play-data gains under matched training exposure on the tested tasks.

Table R3. Long rollout results for the same models.

MetricWipe whiteboard / Task-centeredWipe whiteboard / PlayPour balls / Task-centeredPour balls / PlayTwo-task mean / Task-centeredTwo-task mean / Play
PSNR (dB) ↑16.3418.7314.7316.1415.5417.44
SSIM ↑0.72480.76540.64470.64660.68470.7060
FVD (R3D-18) ↓52.2629.6857.4636.1054.8632.89
Position (mm) ↓22.1918.4817.4113.6719.8016.07
Rotation (°) ↓4.383.934.813.354.593.64
Gripper (normalized) ↓0.14790.02960.14380.04910.14580.0393
Progress consistency (0–100) ↑83.8784.4178.0176.1480.9480.27
MAE ↓0.07270.05120.09390.07510.08330.0632
MSE ↓0.02340.01350.03370.02450.02850.0190
Frame FID ↓64.7150.7198.3876.0881.5563.39

Long rollout visual and action metrics also improve. The mean change in progress consistency is −0.67 points, with a 95% confidence interval of [−5.71, 3.92], leaving uncertainty about long-horizon progress gains. Results are first aggregated within each task and then averaged with equal weights for the two tasks. The progress-difference intervals use 10,000 paired trajectory resamples within each task and reflect test-trajectory uncertainty conditional on the fixed tasks and trained checkpoints.

W2.

We thank the reviewer for raising evaluator accuracy and out-of-distribution reliability concerns. We validated the frozen IDMs on 1,480 original recordings across nine platforms. A split audit confirmed that these recordings are disjoint from IDM training and checkpoint-selection data. Checkpoints and hyperparameters remained fixed. Our metric compares motion independently inferred from generated and reference videos by the same frozen IDM. The new validation assesses motion recovery and sensitivity to motion differences on real held-out recordings.

Table R4. Frozen-IDM errors against source motion labels, reported as means with 95% confidence intervals.

Platform / componentRecordingsPosition (mm) ↓Rotation (°) ↓Gripper (% of range) ↓
ALOHA28020.22 [19.74, 20.72]5.98 [5.84, 6.12]1.33 [1.28, 1.38]
Astribot36017.78 [17.44, 18.13]2.24 [2.17, 2.31]1.41 [1.36, 1.45]
Tianji40010.81 [10.56, 11.05]2.84 [2.76, 2.91]1.87 [1.79, 1.96]
Flexiv8017.41 [16.79, 18.04]3.00 [2.86, 3.15]4.03 [3.71, 4.36]
Gripper12019.24 [18.65, 19.83]2.19 [2.15, 2.24]2.96 [2.83, 3.10]
Finger8011.08 [10.61, 11.57]2.81 [2.71, 2.92]3.67 [3.36, 4.01]
UMI Single2013.75 [12.52, 15.16]2.08 [1.94, 2.22]1.78 [1.72, 1.84]
ARX (sim.)2020.98 [19.00, 22.91]7.99 [7.17, 8.87]2.69 [2.45, 2.96]
Ego camera12018.60 [17.37, 19.87]1.93 [1.84, 2.02]—
Ego wrists12033.15 [32.30, 34.00]8.61 [8.37, 8.87]—
Ego fingertips1209.67 [9.42, 9.94]——

Errors are first aggregated within recordings, then averaged within tasks, with equal task weights within each platform. The intervals are obtained from 20,000 resamples of original recordings within fixed tasks. UMI Single and ARX use held-out Long rollout recordings. Ego uses reconstructed motion labels, and its three components share the same 120 recordings, counted once in the total.

We compare distances between IDM motion estimates from non-overlapping windows within each recording with the corresponding source-label distances. We use up to 128 predetermined window pairs per recording, compute correlations within recordings, and aggregate across recordings and tasks.

Table R5. Position errors of the IDM and a zero-motion baseline (mm), and Spearman correlations between motion distances.

PlatformIDM position errorZero-motion baseline errorMotion-distance Spearman ρ\rho
ALOHA20.2246.520.910
Astribot17.7836.580.855
Tianji10.8123.450.882
Flexiv17.4130.990.869
Gripper19.2453.350.932
Finger11.0827.610.942
UMI Single13.7534.580.871
ARX (sim.)20.9836.250.762

IDM position errors are lower than the zero-motion baseline on all eight platforms. Correlations between motion distances range from 0.762 to 0.942, supporting motion recovery and sensitivity to motion differences on real held-out videos.

We added a blinded human evaluation of Robometer, used zero-shot. We sampled two trajectories per task from 26 tasks spanning nine platforms and ten interaction categories, independently of automatic scores. Three annotators independently judged which generated video in each of 52 anonymized prediction pairs better matched the reference execution’s task progression, focusing on progress timing, stalls, failures, and recovery.

Table R6. Agreement between progress-consistency rankings and human judgments.

Judgment sourceDecisive A/B preferencesTiesMetric agreements / decisive preferencesAgreement
Annotator 1371533/3789.2%
Annotator 244840/4490.9%
Annotator 345740/4588.9%
Majority judgment371535/3794.6%

There were 37 pairs with a decisive human majority preference, of which 35 agreed with the metric ranking, yielding 94.6% agreement and a 95% confidence interval of [86.8%, 100%] obtained by resampling tasks. The other 15 pairs received majority tie judgments. Fleiss’ κ\kappa across the three annotators was 0.496, and pairwise exact agreement ranged from 55.8% to 75.0%. These results support progress-consistency ranking on samples with decisive preferences.

The human question directly matches our progress-consistency definition. Scores of 89–92 describe agreement between generated and reference progress curves. Task success rates are computed separately from execution outcomes. To address Robometer’s additional role as the RL reward, we validate progress rankings with independent human judgments and assess RL transfer through real-robot success rates.

Q1.

We thank the reviewer for asking about our contributions beyond PlayWorld. RoboWorld100’s core contributions are a task-relevant human-play dataset spanning nine platforms, ten interaction categories, and real and simulated environments, together with a world-model evaluation benchmark. Fixed splits, paired testing, and evaluation of visual quality, action consistency, task progress, and long-horizon stability provide consistent conditions for comparing models across platforms and interaction settings.

Cross-platform results reveal that model comparisons depend on both platform and evaluation dimension. On Flexiv, GigaWorld achieves the highest PSNR but the largest position error and lowest progress consistency. On ALOHA and Tianji, Cosmos Edge performs better in PSNR and FVD, while Cosmos Nano has higher motion and progress consistency. These observations suggest that visual-quality advantages can coexist with action-response or task-progression discrepancies. Multidimensional evaluation helps identify these differences and informs model selection.

PlayWorld focuses on automatically collected Play data. We study task-relevant play collected by humans using task knowledge to actively explore diverse interactions around task objectives. With matched frame exposure and optimizer-update budgets on two Tianji tasks, human-collected Play improved visual and motion prediction on both Evaluation and Long rollout. Mean Evaluation progress consistency increased by 4.36 points, mainly from the ball-pouring task. The mean Long rollout change was −0.67 points, with a 95% confidence interval of [−5.71, 3.92] (Tables R2–R3). These results support the predictive value of human-collected, task-relevant Play on the tested tasks.

Training and Evaluation have different collection objectives. Training emphasizes open-ended interaction exploration. Evaluation uses separately collected, task-execution-oriented trajectories covering successful completion, errors, and failures. Table 4 therefore examines how Play helps predict task execution.

The proposed policy-rollout test asks whether Play’s advantages extend to policy-produced behavior. Our policy evaluation executes the same policy in closed loop in the Play simulator, the Task-centered simulator, and on the real robot. Actions depend on each environment’s current observations, and real-robot success rates assess both simulators’ estimates. The Play simulator’s smaller average gap indicates more accurate performance estimates for the tested policies, providing evidence for Play in policy-driven execution. We agree that an independent policy-rollout test set would more directly test the effect of the test distribution on Table 4. We thank the reviewer for this suggestion and will make every effort to add this evaluation as time and compute permit, with timely result updates.

Q2.

We thank the reviewer for suggesting more policy checkpoints and trials, together with ranking correlations. Our original experiment assessed success-rate estimation for a given policy across four Tianji tasks, using 10 trials per task in each environment. The Play simulator reduced the mean absolute gap to real-robot success from 40 to 15 percentage points, supporting its estimates for the tested policies.

Multiple checkpoints within each task would assess whether simulators correctly distinguish and rank policies of different ability. More trials would help reduce estimation uncertainty. We are adding this comparison in both simulators and on the real robot, reporting correlations and ranking consistency. We will complete the experiment before the rebuttal deadline and provide the results as soon as it is complete.

Q3.

We thank the reviewer for suggesting the same DSRL procedure in the Task-centered simulator. Our original study compared data sources for world-model prediction and examined downstream utility through policy optimization in the Play simulator. Mean real-robot success across three Tianji tasks increased from 26.7% to 63.3%, supporting its utility for policy optimization. Assessing the data source’s effect on RL transfer requires comparing both simulators.

We are adding this comparison with matched base policies, DSRL settings, simulator-training budgets, and interaction budgets. Given the available compute and time, the Task-centered comparison remains incomplete. We will report the results when it is complete.

Q4.

We thank the reviewer for asking about IDM accuracy. W2, Tables R4–R5, reports per-platform real held-out video errors and 95% confidence intervals, together with validation of sensitivity to motion differences.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

Dear Area Chair and Reviewers,

We sincerely thank the reviewers for their careful assessments and constructive suggestions, and for recognizing RoboWorld100’s broad interaction coverage and the value of evaluating world models beyond visual quality. The discussion has helped us further examine the benchmark’s applicability, the interpretation of its metrics, and the downstream value of task-relevant human-play data.

Our responses and follow-up experiments focused on the questions raised by the reviewers:

  • Broader model coverage. Reviewer CoFb asked whether the benchmark could evaluate additional world-model baselines. Following this suggestion, we post-trained DreamDojo-2B on RoboWorld100 and completed evaluation across all nine platforms under the same protocol. The results extend the model comparison and reinforce the importance of evaluating motion and task progress alongside visual quality.

  • Agreement with independent evidence. Reviewers Q8GM and DKsc asked how the learned evaluators compare with recorded motion and human judgments. We validated the frozen IDMs on 1,480 held-out recordings across nine platforms and conducted a blinded human study of progress-consistency rankings. Robometer agreed with 35 of 37 decisive majority preferences (94.6%) among 52 comparisons, with the other 15 receiving majority tie judgments.

  • Play benefits under controlled training exposure. Reviewers Q8GM and DKsc asked whether Play’s benefits persist when training exposure is controlled. We therefore compared Play and Task-centered data on two Tianji tasks with matched frame exposure and optimizer updates. Visual and motion-prediction improvements persisted, and mean Evaluation progress consistency increased by 4.36 points, supporting Play’s benefits under the tested budget controls.

  • Policy ranking and real-robot transfer. Reviewer DKsc suggested evaluating multiple policy checkpoints with more trials. We evaluated five checkpoints on Pour, with 20 trials per checkpoint in each environment. The Play simulator tracked real policy rankings more closely than the Task-centered simulator, with Spearman correlations of 0.8721 versus 0.3162. Reviewers Q8GM and DKsc also asked how the same DSRL procedure performs in a Task-centered simulator. Our three-task comparison found mean real-robot success of 63.3% for Play DSRL versus 33.3% for Task-centered DSRL, with higher observed success for Play on all three tasks.

Reviewer feedback: We are encouraged that all three reviewers give positive overall ratings. Reviewer CoFb acknowledged RoboWorld100 as a good benchmark for better evaluation of robotics world models, while Reviewer Q8GM recognized the contribution of our dataset and benchmark to action-conditioned world-model research. Reviewer DKsc thanked us for the detailed experiments and explanations and expressed that they are now “much more supportive of the comprehensiveness and contribution of this benchmark.”

We are grateful for the constructive discussion and the reviewers’ positive follow-up assessments. The full experimental results and clarifications are available in our rebuttal and supplementary responses. We welcome any further questions and sincerely appreciate the reviewers’ and Area Chair’s time and consideration.

Best regards, The Authors of RoboWorld100

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.