- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ProWorld evaluates frozen robot policies by generating multi-view video and proprioceptive states for repeated closed-loop interaction. It combines shared-backbone video-state prediction, Shared-Time View Encoding (STVE), and simulated failure data. Across three policies and four local tasks, the reported Pearson and Spearman correlations with real success rates are 0.95 and 0.94. The paper also evaluates policies on RoboChallenge and video prediction on two benchmarks.
Critical strengths
-
Providing both visual observations and predicted robot states addresses a practical requirement of closed-loop policy evaluation. Modeling failed interactions is also well motivated, since hallucinated successes can distort estimated policy performance.
-
The experiments cover several evaluation settings. The video ablations support the benefits of joint state supervision and STVE, while the failure-data experiment shows a promising reduction in success-rate overestimation.
Critical weaknesses
-
The benefit of predicted state feedback is not directly established for the main evaluation objective. Table 4 measures video quality, video-state agreement, motion direction, and gripper accuracy; Table 6 measures video quality. These results support state supervision during training, but do not show whether feeding predicted states to the policy improves agreement with real-robot outcomes. A comparison against command-derived state feedback in the same trained model would help separate the benefit of joint training from that of the feedback itself. VSC also measures agreement between generated video and predicted states, rather than position accuracy against the recorded states.
-
The evidence for reliable policy evaluation needs stronger statistical support. The headline correlations pool 12 policy-task pairs, so they may partly reflect differences in task difficulty. Within-task policy rankings and absolute success-rate errors would be more directly informative. Appendix A.5 reports total trajectory counts, but not the counts for each policy-task pair or uncertainty estimates. Failure data raises Pearson correlation from 0.93 to 0.95 and improves the fitted calibration line, but the stability of these gains is unclear. Local generated rollouts are scored by a VLM, without sufficient detail about the judge or validation against human labels.
-
The methodological advance over closely related systems is not fully established. Ctrl-World already supports multi-view closed-loop evaluation and trains on successful and failed trajectories. GE-Sim 2.0 already provides predicted proprioceptive feedback through a state expert trained over a frozen visual model. The Cosmos 2.5 technical report, Section 6.3.1, describes independent per-view encoding, view embeddings, and separately constructed per-view RoPE. ProWorld's shared-backbone training, rotary view coordinate, and simulation-based failure augmentation are meaningful differences. However, Table 5 does not clearly establish the gain over the existing per-view RoPE and view-embedding design, and GE-Sim 2.0 is absent from the central closed-loop comparison.
-
Practical evaluation cost is insufficiently documented. The Wan2.1-14B version is used for RoboChallenge, where most tasks require over 80 interaction rounds. The paper mentions distillation but does not report per-round latency or total rollout time. The backbone and inference configuration behind each reported evaluation should also be made explicit. This is a secondary concern, but it limits assessment of the savings over physical evaluation.
I choose weak rejection, because the central benefit of predicted state feedback remains untested at the policy-evaluation level, and the reliability of the reported success-rate agreement is not yet sufficiently established.
Questions for authors
-
With the world-model weights, policies, and initial scenes held fixed, how do predicted-state and command-derived-state feedback compare in success-rate error and within-task policy ranking? For paired prediction, could you also report position and rotation errors against the reference states?
-
Could you report per-policy/task trial counts, uncertainty estimates, and absolute success-rate errors? Please also identify the local VLM judge and provide a human-annotated check of its success labels.
-
Does the time-concatenation baseline in Table 5 reset temporal RoPE coordinates for each view and include view embeddings, as in the Cosmos design? If not, how much of STVE's gain remains against that baseline?
-
For each closed-loop setting, could you specify the backbone, hardware, resolution, sampling steps, action/video chunk lengths, and per-round and complete-rollout times?
4 comments
Response to Reviewer oXq7 (Part 2/2)
(Continued from the previous comment.)
W3 & Q3. Advance over related systems
- Ctrl-World is multi-view but returns no robot state, so the policy is fed its own commands. Under identical Cobot fine-tuning it reaches r = 0.33, and the RoboChallenge ablation above shows what the missing state feedback costs over long horizons.
- GE-Sim 2.0 trains a state expert on a frozen visual model, so state supervision cannot shape the visual dynamics. In ProWorld the state loss trains the shared backbone (removing it costs 1.36 dB PSNR, Table 8 of the revision), and asymmetric attention keeps state a readout of the video (VSC 20.3 to 14.2 mm against full attention). We could not include GE-Sim 2.0 in the closed-loop study: adapting it to a new robot requires forward kinematics and calibrated camera poses, which our Cobot and RoboChallenge data do not have. In Table 3, ProWorld is ahead of it on trajectory accuracy (62.2 vs. 58.6) and photometric consistency (35.3 vs. 7.1).
- Cosmos 2.5. We do not claim per-view encoding as new. The difference is where view identity lives: Cosmos adds it as a token offset, while STVE makes it a fourth rotary coordinate inside joint attention, with no added tokens or parameters.
Table 5. The original time-concatenation baseline does not reset the temporal coordinate and has no view embedding. We have now trained the variant you describe under the same setting: each view's temporal coordinate restarts at zero and a learned per-view embedding is added, as in Cosmos 2.5.
| Encoding | PSNR↑ | SSIM↑ | LPIPS↓ | VSC↓ | Dir.↑ | Grip.↑ |
|---|---|---|---|---|---|---|
| Pixel tiling | 17.95 | 0.619 | 0.2843 | 23.2 | 0.318 | 0.969 |
| Time concatenation | 21.56 | 0.809 | 0.1266 | 13.3 | 0.416 | 0.974 |
| + time reset, view embedding | 21.88 | 0.812 | 0.1197 | 13.3 | 0.412 | 0.972 |
| STVE (ours) | 22.60 | 0.829 | 0.1069 | 14.2 | 0.464 | 0.974 |
The stronger baseline recovers about a third of the PSNR and LPIPS gap. STVE keeps most of its gain over it: +0.72 dB PSNR, −0.0128 LPIPS, and directional accuracy of 0.464 vs. 0.412. As in the original table, the time-concatenation variants have lower VSC. The new row is in Table 5 and the variant is described in Sec. 3.3 and Appendix B.8.
The gain also holds in closed loop: replacing STVE with time concatenation in the full model lowers Cobot agreement from 0.95/0.94 (Pearson/Spearman) to 0.93/0.86 and raises the MAE from 7.4 to 8.5 points.
W4 & Q4. Evaluation cost
Both settings run on a single H800 GPU and generate three views at 384×288. Times exclude model loading.
| Cobot | RoboChallenge | |
|---|---|---|
| Backbone | MiniMax-H3 (distilled) | Wan2.1-14B (not distilled) |
| Denoising steps per round | 4 | 50 |
| Action chunk | 50 actions at 30 Hz | first 16 actions (π0.5 outputs 16, DM0.5 30) |
| Video per round | 17 frames at 10 fps | 8 frames at 5 fps, first 4 kept |
| Fed back to the policy | last frame + last state | 4th frame + its state |
| Rounds per rollout | 25 | 120 (96 s of robot time) |
| Time per round | 4.8 to 10 s, growing with the history cache | about 12 s |
| Full rollout | 3 to 4 min | about 24 min |
The two settings differ mainly in distillation. The Cobot model is distilled, and a rollout costs a few GPU-minutes. The RoboChallenge cost comes mostly from 50-step sampling with the undistilled 14B model, which the same distillation would reduce. In both settings a rollout needs no robot, operator or scene reset, and rollouts from different initial frames can run in parallel. This is Table 17 in Appendix E.4; Appendix A.1 now states which backbone each study uses.
After reading the rebuttal, I change my rating to "weak acceptance", since the authors answered my concerns.
Thank you for reading our response and for raising your score. We are glad our rebuttal addressed your concerns, and all the discussed changes are reflected in the revised paper.
Response to Reviewer oXq7 (Part 1/2)
Thank you for the thorough review. We appreciate that you find predicted-state feedback and failure modeling well motivated. You give two reasons for the weak rejection: state feedback is untested at the policy-evaluation level, and the reliability of the success-rate agreement is not established. We have now run both: a closed-loop state-feedback ablation (W1) and per-pair statistics with a human-validated judge (W2). The remaining two points follow in the second comment. The new results are in the revised PDF (Table 6 and Appendix E; changes in blue).
W1 & Q1. Benefit of predicted-state feedback
Closed-loop comparison. We ran the comparison you ask for: same world-model weights, policies and initial scenes, with the policy fed either the predicted state or the command-derived state (the last command of its previous chunk). We used RoboChallenge because the effect is a long-horizon one. A command-derived state drifts from what the generated images show as rounds accumulate, and RoboChallenge rollouts run 120 rounds, against 25 on Cobot.
MAE is the mean absolute success-rate error. On within-task ranking, DM0.5 beats π0.5 on all six tasks on the real platform; ProWorld reproduces this on five tasks with predicted state (the sixth is a 0% tie) and on four with command-derived state. The gap is largest on long bimanual tasks: for DM0.5 on Hold the Tray, real success is 43%, and ProWorld gives 60% with predicted state and 10% without.
Errors against recorded states (last three rows of Table 4):
The recorded-state errors are close to VSC (20.5 / 20.3 / 14.2 mm) and rank the variants the same way. We keep VSC as the primary metric because once the policy departs from the demonstration there is no recorded trajectory to match; what the policy needs is a state that agrees with the images it is shown. These numbers are in Appendix B.2 (Table 7).
W2 & Q2. Statistical support
Per-pair results (Cobot). The real column gives successes/trials; each real rollout provides the initial frame of one generated rollout, so the generated trial counts are the same. There are 221 real rollouts in total. "w/o F" is ProWorld without failure data. This is Table 14 of the revision.
Summary over the 12 pairs.
The one pair ProWorld misorders has real success rates 2 points apart (43% vs. 41%). A bootstrap over the 12 pairs gives a 95% interval of [0.85, 0.98] for the full model's Pearson r.
Failure data. The gain is in calibration: on the pairs with real success at or below 30%, over-estimation drops from 11.1 to 3.0 points. As a direct test, we replayed 50 missed-grasp segments through both models. Without failure data 35 are rendered as successes, with it 2, while successful grasps are preserved in both cases (47 and 46 of 50).
VLM judge. Qwen3.8-27B, run locally at its highest reasoning setting on a tiled three-view video, with a prompt that states the view layout and the task and asks for a binary success decision. Against annotators who did not see its output, it agrees on 90% of 50 real and 84% of 50 generated rollouts (precision/recall 0.88/0.92 for successes, 0.85/0.78 for failures).