acceptodds
Under review as a conference paper at ICLR 2027

ProWorld: Multi-View World Model with Proprioceptive Feedback for Closed-Loop Policy Evaluation

Abstract

Evaluating robot manipulation policies through repeated physical trials is costly. We study whether an action-conditioned world model can serve as a closed-loop test environment whose policy success rates track those on real robots. Beyond realistic video, the model must return the camera views and proprioceptive state needed for the policy's next action and preserve failed-action outcomes to avoid overestimating policy capability. We introduce ProWorld, a world model that jointly predicts multi-view video and robot state, then feeds both back to a frozen policy for repeated interaction. ProWorld comprises a shared transformer for joint video–state flow prediction and Shared-Time View Encoding to distinguish cameras while preserving temporal synchronization. To reduce interaction hallucinations that turn missed grasps into successful lifts, we augment training with failed interactions collected in high-fidelity simulation, reducing inflated success rates in the generated environment. ProWorld achieves the highest overall TriWorldBench score among the compared state-of-the-art methods. Across three policies and four local manipulation tasks, it attains a Pearson correlation of 0.95 and a Spearman correlation of 0.94 between generated and real success rates. These results support joint visual-proprioceptive feedback and failure-aware training as a basis for world model policy evaluation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Model, Policy Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_oXq7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ProWorld evaluates frozen robot policies by generating multi-view video and proprioceptive states for repeated closed-loop interaction. It combines shared-backbone video-state prediction, Shared-Time View Encoding (STVE), and simulated failure data. Across three policies and four local tasks, the reported Pearson and Spearman correlations with real success rates are 0.95 and 0.94. The paper also evaluates policies on RoboChallenge and video prediction on two benchmarks.

Critical strengths
  1. Providing both visual observations and predicted robot states addresses a practical requirement of closed-loop policy evaluation. Modeling failed interactions is also well motivated, since hallucinated successes can distort estimated policy performance.

  2. The experiments cover several evaluation settings. The video ablations support the benefits of joint state supervision and STVE, while the failure-data experiment shows a promising reduction in success-rate overestimation.

Critical weaknesses
  1. The benefit of predicted state feedback is not directly established for the main evaluation objective. Table 4 measures video quality, video-state agreement, motion direction, and gripper accuracy; Table 6 measures video quality. These results support state supervision during training, but do not show whether feeding predicted states to the policy improves agreement with real-robot outcomes. A comparison against command-derived state feedback in the same trained model would help separate the benefit of joint training from that of the feedback itself. VSC also measures agreement between generated video and predicted states, rather than position accuracy against the recorded states.

  2. The evidence for reliable policy evaluation needs stronger statistical support. The headline correlations pool 12 policy-task pairs, so they may partly reflect differences in task difficulty. Within-task policy rankings and absolute success-rate errors would be more directly informative. Appendix A.5 reports total trajectory counts, but not the counts for each policy-task pair or uncertainty estimates. Failure data raises Pearson correlation from 0.93 to 0.95 and improves the fitted calibration line, but the stability of these gains is unclear. Local generated rollouts are scored by a VLM, without sufficient detail about the judge or validation against human labels.

  3. The methodological advance over closely related systems is not fully established. Ctrl-World already supports multi-view closed-loop evaluation and trains on successful and failed trajectories. GE-Sim 2.0 already provides predicted proprioceptive feedback through a state expert trained over a frozen visual model. The Cosmos 2.5 technical report, Section 6.3.1, describes independent per-view encoding, view embeddings, and separately constructed per-view RoPE. ProWorld's shared-backbone training, rotary view coordinate, and simulation-based failure augmentation are meaningful differences. However, Table 5 does not clearly establish the gain over the existing per-view RoPE and view-embedding design, and GE-Sim 2.0 is absent from the central closed-loop comparison.

  4. Practical evaluation cost is insufficiently documented. The Wan2.1-14B version is used for RoboChallenge, where most tasks require over 80 interaction rounds. The paper mentions distillation but does not report per-round latency or total rollout time. The backbone and inference configuration behind each reported evaluation should also be made explicit. This is a secondary concern, but it limits assessment of the savings over physical evaluation.

I choose weak rejection, because the central benefit of predicted state feedback remains untested at the policy-evaluation level, and the reliability of the reported success-rate agreement is not yet sufficiently established.

Questions for authors
  1. With the world-model weights, policies, and initial scenes held fixed, how do predicted-state and command-derived-state feedback compare in success-rate error and within-task policy ranking? For paired prediction, could you also report position and rotation errors against the reference states?

  2. Could you report per-policy/task trial counts, uncertainty estimates, and absolute success-rate errors? Please also identify the local VLM judge and provide a human-annotated check of its success labels.

  3. Does the time-concatenation baseline in Table 5 reset temporal RoPE coordinates for each view and include view embeddings, as in the Cosmos design? If not, how much of STVE's gain remains against that baseline?

  4. For each closed-loop setting, could you specify the backbone, hardware, resolution, sampling steps, action/video chunk lengths, and per-round and complete-rollout times?

4 comments
AuthorsOpenReview ↗

Response to Reviewer oXq7 (Part 1/2)

Thank you for the thorough review. We appreciate that you find predicted-state feedback and failure modeling well motivated. You give two reasons for the weak rejection: state feedback is untested at the policy-evaluation level, and the reliability of the success-rate agreement is not established. We have now run both: a closed-loop state-feedback ablation (W1) and per-pair statistics with a human-validated judge (W2). The remaining two points follow in the second comment. The new results are in the revised PDF (Table 6 and Appendix E; changes in blue).

W1 & Q1. Benefit of predicted-state feedback

Closed-loop comparison. We ran the comparison you ask for: same world-model weights, policies and initial scenes, with the policy fed either the predicted state or the command-derived state (the last command of its previous chunk). We used RoboChallenge because the effect is a long-horizon one. A command-derived state drifts from what the generated images show as rounds accumulate, and RoboChallenge rollouts run 120 rounds, against 25 on Cobot.

RoboChallenge, 12 policy-task pairsPearsonSpearmanMAE (pts)
Command-derived state0.740.6318.9
Predicted state (ours)0.890.8613.5

MAE is the mean absolute success-rate error. On within-task ranking, DM0.5 beats π0.5 on all six tasks on the real platform; ProWorld reproduces this on five tasks with predicted state (the sixth is a 0% tie) and on four with command-derived state. The gap is largest on long bimanual tasks: for DM0.5 on Hold the Tray, real success is 43%, and ProWorld gives 60% with predicted state and 10% without.

Errors against recorded states (last three rows of Table 4):

Abs. stateFull attn.Ours
Predicted vs. video (IDM): rotation4.8°4.2°3.0°
Predicted vs. recorded: position (mm)21.419.714.0
Predicted vs. recorded: rotation5.0°4.7°3.3°

The recorded-state errors are close to VSC (20.5 / 20.3 / 14.2 mm) and rank the variants the same way. We keep VSC as the primary metric because once the policy departs from the demonstration there is no recorded trajectory to match; what the policy needs is a state that agrees with the images it is shown. These numbers are in Appendix B.2 (Table 7).

W2 & Q2. Statistical support

Per-pair results (Cobot). The real column gives successes/trials; each real rollout provides the initial frame of one generated rollout, so the generated trial counts are the same. There are 221 real rollouts in total. "w/o F" is ProWorld without failure data. This is Table 14 of the revision.

TaskPolicyRealProWorldw/o F
Assemble Partsπ0.5-RL47% (14/30)43%43%
π0.518% (2/11)27%27%
π00% (0/10)0%10%
Cook Vegetableπ0.5-RL100% (30/30)93%93%
π0.571% (10/14)64%79%
π050% (6/12)50%50%
Put Cube into Drawerπ0.5-RL97% (29/30)77%83%
π0.554% (7/13)62%69%
π060% (6/10)70%60%
Pack and Pour Fruitπ0.5-RL43% (13/30)50%67%
π0.541% (7/17)59%59%
π029% (4/14)29%43%

Summary over the 12 pairs.

PearsonSpearmanMAE (pts)Within-task pairs ordered correctly
Ctrl-World0.330.2623.27/12
Cosmos3-Nano0.670.5418.010/12
ProWorld w/o failure data0.930.9110.011/12
ProWorld0.950.947.411/12

The one pair ProWorld misorders has real success rates 2 points apart (43% vs. 41%). A bootstrap over the 12 pairs gives a 95% interval of [0.85, 0.98] for the full model's Pearson r.

Failure data. The gain is in calibration: on the pairs with real success at or below 30%, over-estimation drops from 11.1 to 3.0 points. As a direct test, we replayed 50 missed-grasp segments through both models. Without failure data 35 are rendered as successes, with it 2, while successful grasps are preserved in both cases (47 and 46 of 50).

VLM judge. Qwen3.8-27B, run locally at its highest reasoning setting on a tiled three-view video, with a prompt that states the view layout and the task and asks for a binary success decision. Against annotators who did not see its output, it agrees on 90% of 50 real and 84% of 50 generated rollouts (precision/recall 0.88/0.92 for successes, 0.85/0.78 for failures).

AuthorsOpenReview ↗

Response to Reviewer oXq7 (Part 2/2)

(Continued from the previous comment.)

W3 & Q3. Advance over related systems

  • Ctrl-World is multi-view but returns no robot state, so the policy is fed its own commands. Under identical Cobot fine-tuning it reaches r = 0.33, and the RoboChallenge ablation above shows what the missing state feedback costs over long horizons.
  • GE-Sim 2.0 trains a state expert on a frozen visual model, so state supervision cannot shape the visual dynamics. In ProWorld the state loss trains the shared backbone (removing it costs 1.36 dB PSNR, Table 8 of the revision), and asymmetric attention keeps state a readout of the video (VSC 20.3 to 14.2 mm against full attention). We could not include GE-Sim 2.0 in the closed-loop study: adapting it to a new robot requires forward kinematics and calibrated camera poses, which our Cobot and RoboChallenge data do not have. In Table 3, ProWorld is ahead of it on trajectory accuracy (62.2 vs. 58.6) and photometric consistency (35.3 vs. 7.1).
  • Cosmos 2.5. We do not claim per-view encoding as new. The difference is where view identity lives: Cosmos adds it as a token offset, while STVE makes it a fourth rotary coordinate inside joint attention, with no added tokens or parameters.

Table 5. The original time-concatenation baseline does not reset the temporal coordinate and has no view embedding. We have now trained the variant you describe under the same setting: each view's temporal coordinate restarts at zero and a learned per-view embedding is added, as in Cosmos 2.5.

EncodingPSNR↑SSIM↑LPIPS↓VSC↓Dir.↑Grip.↑
Pixel tiling17.950.6190.284323.20.3180.969
Time concatenation21.560.8090.126613.30.4160.974
+ time reset, view embedding21.880.8120.119713.30.4120.972
STVE (ours)22.600.8290.106914.20.4640.974

The stronger baseline recovers about a third of the PSNR and LPIPS gap. STVE keeps most of its gain over it: +0.72 dB PSNR, −0.0128 LPIPS, and directional accuracy of 0.464 vs. 0.412. As in the original table, the time-concatenation variants have lower VSC. The new row is in Table 5 and the variant is described in Sec. 3.3 and Appendix B.8.

The gain also holds in closed loop: replacing STVE with time concatenation in the full model lowers Cobot agreement from 0.95/0.94 (Pearson/Spearman) to 0.93/0.86 and raises the MAE from 7.4 to 8.5 points.

W4 & Q4. Evaluation cost

Both settings run on a single H800 GPU and generate three views at 384×288. Times exclude model loading.

CobotRoboChallenge
BackboneMiniMax-H3 (distilled)Wan2.1-14B (not distilled)
Denoising steps per round450
Action chunk50 actions at 30 Hzfirst 16 actions (π0.5 outputs 16, DM0.5 30)
Video per round17 frames at 10 fps8 frames at 5 fps, first 4 kept
Fed back to the policylast frame + last state4th frame + its state
Rounds per rollout25120 (96 s of robot time)
Time per round4.8 to 10 s, growing with the history cacheabout 12 s
Full rollout3 to 4 minabout 24 min

The two settings differ mainly in distillation. The Cobot model is distilled, and a rollout costs a few GPU-minutes. The RoboChallenge cost comes mostly from 50-step sampling with the undistilled 14B model, which the same distillation would reduce. In both settings a rollout needs no robot, operator or scene reset, and rollouts from different initial frames can run in parallel. This is Table 17 in Appendix E.4; Appendix A.1 now states which backbone each study uses.

After reading the rebuttal, I change my rating to "weak acceptance", since the authors answered my concerns.

Thank you for reading our response and for raising your score. We are glad our rebuttal addressed your concerns, and all the discussed changes are reflected in the revised paper.

Reviewer_Z6r6OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper builds an action-conditioned video world model meant to be used as a closed-loop test environment for frozen manipulation policies. A shared flow-matching transformer predicts multi-view video and proprioceptive state together, with state queries reading the video stream through an asymmetric mask. Camera identity gets its own rotary coordinate (STVE). Missed-grasp failures are collected in GenieSim by replaying successful trajectories with a Cartesian offset, and mixed into training so that failed actions don't get rendered as successes. On 3 policies x 4 local tasks the generated success rates correlate with real ones at r = 0.95 (Ctrl-World 0.33, Cosmos3-Nano 0.67), on 2 policies x 6 RoboChallenge tasks at r = 0.89, and the model is first on TriWorldBench.

Critical strengths
  • Strong evaluation. The paper reports the correlation between generated and real success rates across policies and tasks, rather than reporting PSNR alone. The results directly shows real-world utility of a world-model evaluator, and the paper does this on two separate setups including minute-long RoboChallenge rollouts.
  • The failure-data idea is simple and interesting. Perturbed replay in simulation is a cheap way to get supervision on missed contact, and the intercept shift from 18.4 to 10.5 seem promising.
  • The appendix is thorough. Action/state interface, alignment, VSC calibration, the lambda sweep, noise scheduling, and the full benchmark tables are all there, and the reasons for dropping the WorldArena motion metrics are stated.
Critical weaknesses
  1. Gains from the proposed components vs. gains from scaling. The proposed approach and evaluation seem solid to me. My key concern about this work is that the data-side improvement is not discussed clearly enough, so it's hard to assess whether the gains come from the proposed improvements or simply from scaling up a strong backbone with more data. For example, Figure 6 shows that the proposed world model predicts real-world success much better than Ctrl-World and Cosmos3-Nano (0.95, 0.33, and 0.67, respectively), and that removing failure data from pretraining hurts this score marginally from 0.95 to 0.93. However, this does not tell us how much of the gain comes from scaling up with successful data. For example, if training Cosmos3-Nano or Ctrl-World with the same pretraining data used in this work can get >0.85, the claimed contributions (state prediction, STVE, failure data) combined would only have helped <0.1, which is quite a bit weaker than it currently reads and would need a change in the emphasis of the article to address. It is also unclear whether the baselines were fine-tuned on the Cobot platform. I would assume yes, but if not the experiments wouldn't be a fair comparison, and I think this is worth a clarification. I appreciate the video-space evaluations on state prediction and STVE, and they seem promising. But they do not guarantee utility when used to score policies. The failure-data ablation does get the closed-loop treatment (Figure 6, third vs fourth panel), so the pipeline exists; running the same 12-pair correlation with the policy fed its last command instead of the predicted state, and with time concatenation instead of STVE, would show whether these two components matter for evaluation. Otherwise I'd ask the authors to soften those claims to what Tables 4 and 5 actually show.

  2. Possible leakage in the adaptation data, and thin protocol details? A.5 says ProWorld is fine-tuned on trajectories from the same platform and tasks before evaluation, and Section 3.4 lists "robot-policy rollouts" as training data. I couldn't tell whether those rollouts come from the same three policies being scored. If they do, part of the correlation could be the model having seen each policy's typical outcome, and the adaptation set should exclude the evaluated policies.

Questions for authors
  1. Were Ctrl-World and Cosmos3-Nano fine-tuned on the same pretraining mixture and/or the Cobot data, and what state did the policy receive from them each round?

  2. Does the target-platform adaptation data include rollouts of pi0, pi0.5, or pi0.5-RL on the four evaluated tasks?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Z6r6

Thank you for the careful review. We are glad you found the real-vs-generated success correlation, the failure-data idea and the appendix valuable. Your main concern is whether the gains come from the proposed components or from data scale, and you suggested concrete experiments to separate the two. We ran them. The results are below and in the revised PDF (Table 6 and Appendix E; changes in blue).

W1 & Q1. Component gains vs. scaling; baseline protocol

Baseline protocol. Ctrl-World and Cosmos3-Nano were fine-tuned from their released checkpoints on the same Cobot data as ProWorld, for the same number of steps and without failure data. The like-for-like comparison is therefore with ProWorld without failure data. Neither baseline outputs robot state, so in each round the policy receives the last state it commanded in the previous chunk.

Data vs. method. We removed our pretraining entirely (public video backbone, fine-tuned on Cobot data only, same steps), and ran the closed-loop STVE ablation you asked for.

World model (Cobot, 12 pairs)PearsonSpearmanMAE (pts)
Ctrl-World0.330.2623.2
Cosmos3-Nano0.670.5418.0
ProWorld, no pretraining, no failure data0.790.4715.0
ProWorld, no failure data0.930.9110.0
ProWorld, time concatenation instead of STVE0.930.868.5
ProWorld (full)0.950.947.4

MAE is the mean absolute gap between generated and real success rates.

  1. Pretraining data matters, and the revised Sec. 4.4 says so plainly: it moves Pearson from 0.79 to 0.93 and Spearman from 0.47 to 0.91.
  2. Data is not the whole story. With no robot pretraining, ProWorld already beats both baselines on Pearson and MAE, although they start from their own large-scale pretrained checkpoints and see the same Cobot data. (Its Spearman is slightly below that of Cosmos3-Nano, 0.47 vs. 0.54.)
  3. STVE and failure data each add a smaller gain on top.

Tables 4 and 5 point the same way: all variants there share one backbone and train on AgiBotWorld-Alpha only, so their differences cannot come from our data.

State feedback in closed loop. We ran the experiment you describe: same weights, policies and initial frames, with the policy fed its last command instead of the predicted state. We used RoboChallenge because the effect is a long-horizon one. A commanded state drifts from what the generated images show as rounds accumulate, and RoboChallenge rollouts run 120 rounds, against 25 on Cobot.

RoboChallenge, 12 policy-task pairsPearsonSpearmanMAE (pts)
Last command as state0.740.6318.9
Predicted state (ours)0.890.8613.5

Without state feedback the model under-estimates the stronger policy on long bimanual tasks. Averaged over the six tasks, DM0.5 succeeds 58% of the time on the real platform, 55% with predicted state and 38% without.

Failure data. We also tested the mechanism directly, replaying 50 missed-grasp and 50 successful-grasp segments through both models. Without failure data, 35 of the 50 missed grasps are rendered as successes; with it, 2. Successful grasps are preserved in both cases (47 and 46 of 50).

W2 & Q2. Possible leakage in the adaptation data

There is none in either study.

  • Cobot. Adaptation uses only human-teleoperated trajectories of the four tasks, with no rollouts of any policy. The real policy rollouts serve only to measure the real success rates and to provide the initial frame of each world-model rollout.
  • RoboChallenge. Adaptation uses the Table30v2 demonstrations plus rollouts that other participants submitted to the benchmark. Rollouts of the two evaluated policies (π0.5 and DM0.5) are excluded. This is what "robot-policy rollouts" in Sec. 3.4 refers to.

So the world model never sees how the evaluated policies behave on the evaluated tasks. A.5 now states both adaptation sets and the baseline protocol above.

Protocol details. The revision also fills in the protocol (Appendix E). The Cobot study has 221 real rollouts: 30 per task for π0.5-RL and 10 to 17 for π0 and π0.5 (per-pair counts in Table 14). Within tasks, ProWorld orders 11 of 12 policy pairs as the real robot does. The VLM judge is Qwen3.8-27B; on 50 real and 50 generated rollouts labeled by annotators who did not see its output, it agrees with the human label on 87 (Table 15).

Dear Reviewer Z6r6,

Thank you again for suggesting concrete experiments. We ran all of them, and as the discussion period ends in two days, we would appreciate knowing whether they address your concerns.

In brief: (1) to separate data from method, ProWorld without any robot pretraining already reaches r = 0.79 vs. 0.67 for Cosmos3-Nano and 0.33 for Ctrl-World under the same Cobot fine-tuning, and pretraining raises it to 0.93 (Table 6, left); (2) the closed-loop state-feedback ablation you proposed shows r dropping from 0.89 to 0.74 without predicted state on RoboChallenge; (3) the closed-loop STVE ablation and the missed-grasp analysis (70% → 4% of missed grasps rendered as success) are included; (4) neither adaptation set contains rollouts of the evaluated policies, so there is no leakage.

We would be happy to clarify anything further before the deadline. If these results resolve your concerns, we would be grateful if you could consider updating your score.

Best regards,

The Authors

Reviewer_jvwBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces a world model that can predict multi-view images and robot state simutaneously. Additionally, authors propose Shared-Time View Encoding (STVE) to organize multi-view inputs and encourage the multi-view consistency. Authors show results on multiple video prediction examples and policy evaluation applications.

Critical strengths
  • The code is included in the submission
  • The results are comprehensive, including cross-embodiment adapation, multiview consistency, and application of the world models
  • Qualitative results are encouraging
Critical weaknesses
  • The quantitative results are mixed. According to Table 3, ProWorld is not the SOTA on 6/10 metrics. This makes it a bit hard to accept this paper, as the result is not SOTA
  • While there are more and more world model papers, it is not super clear what message this paper tries to convey. The major difference in this paper is the state prediction and multi-view prediction. However, it is not clear how much benefits do state prediction and multi-view prediction bring. According to Table 3, state prediction does not bring too many benefits.
Questions for authors
  • What is R in Equation 5?
  • One of the highlights in the experiment section is multi-robot adaptation. However, it is not clear how the proposed method is connected to this result. Is it simply because of the comprehensive data?
  • For baseline Ctrl-World and Cosmos3-Nano in Figure 6, do they use failure data as well?
2 comments
AuthorsOpenReview ↗

Response to Reviewer jvwB

Thank you for the thoughtful review. We are glad you found the experiments comprehensive and the qualitative results encouraging. Your two concerns are the WorldArena numbers and how much state and multi-view prediction help; we answer both below, the second with a new closed-loop ablation. The new experiments are in the revised PDF, with changes marked in blue.

W1. Mixed results in Table 3

Table 3 (WorldArena) is a single-view, open-loop benchmark. ProWorld is trained on multi-view data and is run here from the head camera alone, its least favorable setting. Even so:

  • No model dominates. ProWorld is best on 4 of the 10 metrics and second on depth accuracy (98.36 vs. 98.44); every other model is best on at most 2. Its average over the 10 metrics is also the highest (75.6; GigaWorld-1 has 75.2, the rest are below 73).
  • The metrics we trail on are led by ABot-PhysWorld and Xiaomi-Robotics-U0, which fall far behind on trajectory accuracy (31.5 and 41.9 vs. our 62.2). For a model that a policy acts in, following the commanded motion matters most.

In the three-view setting the model is built for (Table 2), ProWorld has the best overall score and is first or second on every sub-metric. On our actual target, agreement with real-robot success, it reaches Pearson r = 0.95, against 0.33 for Ctrl-World and 0.67 for Cosmos3-Nano (Fig. 6).

W2. The message, and the benefit of state and multi-view prediction

The message: to stand in for a real robot in policy evaluation, a world model must return everything the policy consumes (all camera views and the proprioceptive state) and must show failure when the policy fails. Each component serves one of these requirements.

State prediction. Table 4 (which we believe the comment refers to) is open-loop: actions are replayed from recordings, so nothing depends on the predicted state. Its benefit appears in closed loop, especially over long horizons. Without it the policy is fed its own last command as its state, and this input drifts away from what the generated images show. We tested this on RoboChallenge (120 rounds per rollout, against 25 on Cobot), changing only the state given to the policy:

RoboChallenge, 12 policy-task pairsPearsonSpearmanMAE (pts)
Last command as state0.740.6318.9
Predicted state (ours)0.890.8613.5

This is Table 6 of the revision; MAE is the mean absolute gap between generated and real success rates. For example, DM0.5 succeeds on Hold the Tray 43% of the time on the real robot; ProWorld gives 60% with predicted state and 10% without.

Multi-view prediction. The evaluated policies take head and wrist cameras as input, because the head view is occluded during grasping. A single-view world model cannot close the loop with them, so multi-view output is a requirement. Among the ways of providing it, STVE gives the best video quality (Table 5), and the gain carries over to closed loop: replacing it with time concatenation lowers Cobot agreement from 0.95/0.94 (Pearson/Spearman) to 0.93/0.86.

Q1. R in Eq. 5

R(t, h, w, v) is the rotary position matrix applied to each token's query and key. It is block-diagonal: R_t, R_h, R_w and R_v are standard RoPE rotations for the time, height, width and view indices, each acting on its own group of channels. The revision defines it right after the equation.

Q2. How the method connects to multi-robot adaptation

Data is part of the reason, and the method is what makes the data usable. Pooling more than 20 embodiments relies on the interface of Sec. 3.2 and Appendix A: a shared 32-d action/state layout with per-embodiment normalizers; velocity-style control, which ties a command to the motion it produces instead of a robot-specific absolute frame (Table 4, rows 1-2); metadata conditioning on embodiment, cameras and sampling rate; and STVE, which lets robots with different camera sets share one backbone. With this interface, one recipe (pooled pretraining, then a short fine-tune) adapts ProWorld to both Cobot and RoboChallenge.

We also measured what the pretraining contributes: fine-tuning the public video backbone directly on Cobot data gives r = 0.79, against 0.93 with pretraining.

Q3. Do the baselines in Fig. 6 use failure data?

No. Both were fine-tuned on the same Cobot data for the same number of steps, without failure data. Panels 1-2 should therefore be compared with panel 3 (ours without failure data, 0.93/0.91), and panels 3-4 isolate the effect of failure data. The caption now says so.

Dear Reviewer jvwB,

Thank you again for your review. As the discussion period ends in two days, we wanted to check whether our response has addressed your concerns.

In brief: (1) on the mixed results in Table 3, ProWorld ranks first on the average over the 10 WorldArena metrics despite being run single-view, and is best overall in the three-view setting it is designed for (Table 2); (2) on the value of state prediction, a new closed-loop experiment on RoboChallenge shows that replacing predicted state with the last command lowers real-robot agreement from r = 0.89 to 0.74 (Table 6, right); (3) we defined R in Eq. 5 and clarified that the Fig. 6 baselines are not given failure data.

If anything remains unclear or you would like further analysis, we would be glad to provide it before the deadline. If our response has resolved your concerns, we would be grateful if you could consider updating your assessment.

Best regards,

The Authors

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to Reviewer 3B4d

Thank you for the detailed review, and for recognizing the complete feedback interface (images plus state, with failures represented) and the design ablations. Both of your points ask for numbers, so we give them directly. The corresponding tables are in the new Appendix E of the revised PDF (changes in blue).

W1 & Q1. Evaluation protocol and statistics

Per-pair results (Cobot). "w/o F" is ProWorld without failure data. Each real rollout provides the initial frame of one generated rollout, so the generated trial counts equal the real ones.

TaskPolicyRealProWorldw/o F
Assemble Partsπ0.5-RL47% (14/30)43%43%
π0.518% (2/11)27%27%
π00% (0/10)0%10%
Cook Vegetableπ0.5-RL100% (30/30)93%93%
π0.571% (10/14)64%79%
π050% (6/12)50%50%
Put Cube into Drawerπ0.5-RL97% (29/30)77%83%
π0.554% (7/13)62%69%
π060% (6/10)70%60%
Pack and Pour Fruitπ0.5-RL43% (13/30)50%67%
π0.541% (7/17)59%59%
π029% (4/14)29%43%

There are 221 real rollouts in total (A.5 had rounded this to 200).

Within-task ranking. ProWorld reproduces the real ordering of the three policies on three tasks, including π0 beating π0.5 on the drawer task. On Pack and Pour Fruit it swaps π0.5-RL and π0.5, whose real rates are 2 points apart. Over all within-task policy pairs it orders 11 of 12 correctly (Cosmos3-Nano 10, Ctrl-World 7).

Error, calibration, uncertainty. The mean absolute error between generated and real success rates is 7.4 points for ProWorld (10.0 without failure data), against 18.0 for Cosmos3-Nano and 23.2 for Ctrl-World. The calibration line is y = 0.82x + 10.53 (in %), and a bootstrap over the 12 pairs gives a 95% interval of [0.85, 0.98] for Pearson r.

VLM judge. Qwen3.8-27B, deployed locally with vLLM at its highest reasoning setting. The three views are tiled into one video (left wrist, head, right wrist); the prompt states the layout and the task instruction and asks for a binary success decision. Real and generated rollouts use the same judge and prompt.

Agreement with human labels. On 50 real and 50 generated rollouts labeled by annotators who did not see the VLM output, the judge agrees on 45 of the real and 42 of the generated ones. Precision/recall are 0.88/0.92 for successes and 0.85/0.78 for failures (confusion matrices in Table 15).

RoboChallenge. As you note, the protocol differs (human-judged generated rollouts, official platform success rates as reference), so we report it as a separate study: Pearson 0.89, Spearman 0.86 and an MAE of 13.5 points over its 12 pairs. DM0.5 beats π0.5 on all six tasks on the real platform; ProWorld gives the same ordering on five and a 0% tie on Tie a Knot (real: 10% vs. 0%).

W2 & Q2. Incremental benefit of failure data

Matching. Both runs use the same number of adaptation steps and the same batch size, hence the same number of training samples; neither completes one epoch. In the control, the 10% share of failures is filled with real Cobot trajectories from the same pool, so the control sees more in-domain data, not less.

Missed grasps. We took the second option you propose, replaying 50 missed-grasp and 50 successful-grasp action segments through both models:

Missed grasp rendered as successSuccessful grasp rendered as success
Without failure data35/50 (70%)47/50 (94%)
With failure data2/50 (4%)46/50 (92%)

So the model has not simply become more pessimistic; it has learned what a missed grasp looks like.

Effect on policy evaluation. This explains the calibration change in Fig. 6. On the three pairs with real success at or below 30%, over-estimation drops from 11.1 to 3.0 points; over all 12 pairs, the MAE drops from 10.0 to 7.4 and the fitted intercept from 18.41 to 10.53. The correlation changes little (0.93 to 0.95) because the model without failure data already orders the pairs well; what failure data fixes is the optimism at the low end. Sec. 4.2 now describes the benefit in these terms.

Area_Chair_TxKCOpenReview ↗

Please review the authors’ rebuttal and update your assessment

Dear reviewers,

Thank you for your reviews, and thank you to the authors for their responses. Please read the authors’ rebuttal and discuss whether it addresses your main concerns. Please update your review and rating if warranted, and briefly explain your current assessment, including any concerns that remain unresolved. If your assessment is unchanged, please explain why.

Thank you for your continued engagement.

Dear Reviewer 3B4d,

Thank you again for the detailed review. As the discussion period ends in two days, we wanted to follow up on whether our response answers your questions.

In brief: (1) we added per-pair results with exact trial counts (221 real rollouts), within-task rankings (11 of 12 pairs ordered correctly), MAE, and a bootstrap 95% CI of [0.85, 0.98] for Pearson r; (2) we validated the VLM judge against blind human labels (87/100 agreement); (3) for failure data, the matched control and a focused missed-grasp analysis show that missed grasps rendered as success drop from 70% to 4% while real successes are preserved, which reduces the MAE from 10.0 to 7.4 points. All of this is in the revised PDF (Appendix E).

If any point needs further detail, we would be glad to provide it before the deadline. If our response has addressed your concerns, we would appreciate it if you could consider updating your assessment.

Best regards,

The Authors

AuthorsOpenReview ↗

Summary of Revisions and Thanks to Reviewers and the Area Chair

Dear Area Chair and Reviewers,

We sincerely thank all four reviewers for the time and care they put into reviewing our paper, and the Area Chair for overseeing the discussion. The reviews were detailed, constructive and technically sharp, and several of them proposed concrete experiments that we ran in full. We believe these have made the paper substantially stronger and its claims more precise. We are also grateful that the reviewers found the problem well motivated, the evaluation against real-robot outcomes valuable, and the failure-data idea simple and effective.

To make the discussion easier to follow, we briefly summarize the main concerns and where each is addressed in the revised PDF (changes marked in blue).

1. Is closed-loop state feedback actually useful for policy evaluation? (oXq7, jvwB, Z6r6)
We ran a new closed-loop ablation on RoboChallenge, keeping the world-model weights, policies and initial frames fixed. Replacing the predicted state with the policy’s last command lowers agreement with real success from r = 0.89 to 0.74 and raises MAE from 13.5 to 18.9 points (Table 6). We also report position and rotation errors of the predicted state against recorded states (Table 7).

2. Gains from the method vs. from data scale; baseline protocol (Z6r6)
All baselines were fine-tuned on the same Cobot data for the same number of steps. Without any robot pretraining, ProWorld still reaches r = 0.79, vs. 0.67 for Cosmos3-Nano and 0.33 for Ctrl-World. Pretraining raises it to 0.93, and STVE and failure data each add a further gain in closed loop. We now state the role of pretraining data plainly in Sec. 4.4. Neither adaptation set contains rollouts of the evaluated policies, so there is no leakage (A.5).

3. Statistical reliability of the evaluation (3B4d, oXq7)
We added per-pair results with exact trial counts (221 real rollouts, Table 14), within-task rankings (11 of 12 pairs ordered correctly), MAE (7.4 points), and a bootstrap 95% CI of [0.85, 0.98] for Pearson r. We also identify the VLM judge and validate it against blind human labels (87/100 agreement, Table 15).

4. Mechanism of failure data (3B4d, oXq7)
In a matched control and a focused missed-grasp replay, the share of missed grasps rendered as success drops from 70% to 4%, while successful grasps are preserved (94% vs. 92%). This explains the calibration gain at the low end, and Sec. 4.2 now describes the benefit in these terms.

5. Mixed WorldArena results (jvwB)
Table 3 is a single-view, open-loop setting, the least favorable one for a multi-view model. ProWorld still has the highest average over the 10 metrics, and it is best overall in the three-view setting it is designed for (Table 2).

6. Related work, design clarity and cost (oXq7, jvwB)
We added a stronger Cosmos-style baseline to Table 5, against which STVE keeps most of its gain. We clarified how our design differs from Ctrl-World, GE-Sim 2.0 and Cosmos 2.5, defined R in Eq. 5, clarified the Fig. 6 baseline protocol, and report per-round and full-rollout cost in Table 17.

We are very thankful for this constructive review process. The reviewers’ suggestions directly shaped the new experiments and the revised claims, and we have tried to reflect every point raised in the paper itself, not only in our replies. If there is anything that could be clarified further, we would be happy to respond within the remaining time.

With our sincere appreciation,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.