- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies video world models for robot policy evaluation when supplied actions can lead to failure. It constructs simulator-verified, scene-matched success-failure trajectories through controlled action perturbations and trains a Wan2.2 video model with the KVAF action representation from EA-WM. Its main modeling contribution, SAFF, uses feedback-derived correction signals and semantic-motion gradients to construct detached weights for the flow-matching loss. It also introduces a six-dimensional fidelity benchmark with confidence and missing-data rules. Experiments report improved video prediction and lower policy success-rate estimation errors in simulation and on real robots.
Critical strengths
-
The simulator-verified failure corpus is a useful practical contribution. Same-scene success–failure pairs make controlled action deviations and their consequences traceable (Section 3.1; Appendix A.1).
-
The experiments support improved video prediction and policy success-rate estimation in simulation and on real robots. The simulation ablation supports SAFF over standard flow matching, with auxiliary computations used only during training (Table 5; Tables 15-16).
Critical weaknesses
-
The evidence for predicting specific failure consequences needs strengthening. The controlled corpus and aggregate results are useful, and Figures 6-9 illustrate improved appearance and post-contact motion. However, these examples do not clearly connect a named action perturbation, its observed failure consequences, and the model's prediction. Representative simulation case studies would clarify which failure modes are reproduced faithfully and where prediction errors remain. This connection is central to the claim of faithful policy evaluation from controlled failure data.
-
The collection and use of real-world failure data are unclear. Section 4.1 describes teleoperation on three tasks and synchronized video/state recording, but does not specify how failed trajectories were obtained or their composition in world-model training and evaluation. The real-world results are encouraging, yet this missing protocol makes it difficult to assess failure coverage and the relationship between collected failures and evaluated policy rollouts. The detailed simulation failure-construction protocol does not resolve this real-world data question.
-
The confidence-aware trajectory score needs independent validation. The appendix provides detailed support and missing-data rules, including abstention when GT evidence is insufficient. For supported track pairs, Equation (19) adds a penalty when mean paired SAM 3 confidence is below . Such low confidence could reflect a distorted prediction or difficulty tracking an otherwise faithful prediction, for example under partial occlusion. The penalty may therefore mix prediction error with tracking difficulty. The paper does not show whether it improves agreement with independently assessed trajectory fidelity. Better policy success-rate estimates support the full system, but do not validate this specific scoring choice.
Questions for authors
-
Please provide representative simulation cases connecting a specific action perturbation, its observed failure consequences, and the model's prediction. Which failure modes are reproduced faithfully, and where do prediction errors remain? Existing rollouts could help clarify what is learned beyond the aggregate gains.
-
How were real-world failed trajectories collected: deliberate teleoperation errors, naturally occurring failures, policy rollouts, or another procedure? Please clarify the success/failure composition of world-model training and evaluation data and its relationship to the evaluated policy rollouts.
-
For supported track pairs, do the low-confidence penalties improve agreement with independently assessed trajectory fidelity? How do you distinguish low confidence caused by prediction distortions from difficult tracking of an otherwise faithful prediction? Existing cases or other validation evidence would help justify the scoring choice.
-
Please provide representative videos of successful and failed executions, showing GT, your method, and Ctrl-World side by side. The sampled frames in Figures 6-9 are useful; videos would help assess the continuity and plausibility of contact, release, and subsequent object motion. If videos were uploaded separately, please indicate where they can be accessed.
4 comments
2. Real-world failure collection and composition
Thank you for pointing out the missing collection details. We clarify the sources of real-world failures, the training and evaluation composition, and their relationship to policy evaluation.
Collection procedure. Successful trajectories come from teleoperated demonstrations, while failed trajectories come from naturally occurring failures during autonomous executions of a fully trained π0.5 policy on three tasks: fruit placement into a black box, mug hanging, and duck placement onto a shelf. A human initializes the environment; execution then proceeds autonomously without takeover or injected errors. Success/failure labels are assigned manually. Videos and robot states are recorded synchronously at 30 Hz.
World-model training and video evaluation. The unique training set contains 1,076 successful and 527 failed recordings. We balance the training mixture across both tasks and outcomes through stratified repeated sampling: each task contributes 618 successful and 618 failed entries per epoch, giving 3,708 training entries with a 1:1 success/failure ratio. The composition is:
| Task | Unique training videos (success / failure) | Training entries per epoch (success / failure) | Held-out video-test recordings (success / failure) |
|---|---|---|---|
| Fruit placement | 171 / 271 | 618 / 618 | 29 / 30 |
| Mug hanging | 287 / 190 | 618 / 618 | 42 / 21 |
| Duck placement | 618 / 66 | 618 / 618 | 74 / 10 |
| Total | 1,076 / 527 | 1,854 / 1,854 | 145 / 61 |
The 206-video prediction test set uses unique recordings without repeated sampling. Splitting is performed by collection groups before training resampling, and we verified that training and this test set share neither source recordings nor collection groups. RGB and KVAF use the same temporal sampling.
Relationship to policy evaluation. Failure collection uses a fully trained π0.5 policy, while the policy-evaluation experiment assesses five π0.5 variants trained on 10%, 20%, 30%, 50%, and 100% of the policy-training data. These percentages refer to policy training; the world-model training mixture follows the task and outcome balance above. The video-prediction test evaluates fidelity against recorded futures, while policy evaluation compares world-model–VLM success-rate estimates with execution-based rates.
3. Independent validation of the confidence penalty
Thank you for raising this concern. We added independent trajectory-fidelity ratings and an ablation that isolates the explicit low-confidence penalty.
Independent reference and controlled comparison. Three additional evaluators independently rated 100 existing GT–SAFF video pairs on a 1–5 trajectory-fidelity scale, without prior access to automatic scores or other evaluators' ratings. Three videos have no available automatic trajectory score for either role under the original support rules; the remaining 97 videos cover 50 tasks. Pairwise inter-rater Spearman correlations are 0.9024–0.9601.
We compare the original setting, , with , keeping tracking, instance matching, support rules, confidence-weighted localization error, and missing-observation penalties fixed. The Spearman correlations with human ratings are:
| Human reference | Without explicit penalty | With explicit penalty | Δρ (with − without) | 95% CI for Δρ |
|---|---|---|---|---|
| Evaluator 1 | 0.8350 | 0.8355 | +0.00046 | [−0.00656, 0.00692] |
| Evaluator 2 | 0.7994 | 0.7964 | −0.00303 | [−0.01214, 0.00448] |
| Evaluator 3 | 0.7515 | 0.7485 | −0.00299 | [−0.01204, 0.00444] |
| Per-video mean rating (supplementary) | 0.8056 | 0.8036 | −0.00208 | [−0.01034, 0.00498] |
Both score variants correlate strongly with human ratings, with only small changes after adding the explicit term. Confidence intervals use 10,000 paired task-cluster bootstrap resamples.
Why the two score variants are close. Both variants retain confidence-weighted localization error and missing-observation penalties; the ablation removes only the additional low-confidence term. This term affects 18 of the 97 videos, and the Spearman correlation between the two automatic score variants is 0.9984. Their nearly identical rankings leave limited room for differences in correlation with the same human ratings: for these fixed scores, the absolute difference is bounded by 0.0571. In the 45 videos with supported arm and object tracks, the rankings are identical, so the human correlations are identical as well.
Distinguishing prediction error from tracking difficulty. The paired confidence uses the lower GT/prediction detection confidence, so a low value can also originate on the GT side. We inspected selected GT/prediction frames and their segmentation masks to check whether trajectory discrepancies arose from the visible motion or from track extraction. This identified a cross-arm instance mismatch and a segmentation-drift case. Both cases received no explicit confidence penalty, and their original scores remain in the analysis. These checks show why visible motion, instance identity, and segmentation masks need to be considered together when interpreting low confidence.
Effect on the reported scores. Across all 315 SAFF predictions, the explicit penalty affects 58 videos. Arm and object scores are first averaged separately and combined with weights of 0.4/0.6; the resulting trajectory dimension contributes one sixth of the final benchmark score. Removing only the explicit penalty changes the trajectory dimension from 65.62 to 65.81 and the overall score from 73.17 to 73.20, a difference of about 0.03 points on the 0–100 scale.
4. Full videos of successful and failed executions
Thank you for the suggestion. We provide complete videos for the cube-placement scene shown in Figure 6, covering successful and failed GT executions, each with side-by-side GT/Ours/Ctrl-World comparisons:
- Failed GT execution, with a grasp-position offset: watch the comparison.
- Successful GT execution, without the perturbation, from the same scene: watch the comparison.
The full videos retain all 81 aligned frames, allowing inspection of contact, separation, and subsequent object motion throughout the execution. For the failed execution, we also provide an annotated version marking the intervention and subsequent events, together with half-speed clips around and after the intervention.
Both pages support direct browser playback. Select “Without event annotations” to view the full comparison without event annotations; no ZIP download or login is required. The video index also includes the premature-release and suppressed-release cases discussed in our response to Question 1.
We thank the reviewer for recognizing the practical value of our simulator-verified failure corpus and the improvements in video prediction and policy success-rate estimation. We appreciate the suggestions to connect specific action perturbations to their predicted consequences, clarify the real-world failure-data protocol, validate the confidence-aware trajectory score, and provide comparative videos. Below, we address these points individually with annotated case studies, protocol details, scoring analyses, and side-by-side videos.
1. Perturbations and their predicted consequences
Thank you for the suggestion. We randomly selected three simulation case studies from existing rollouts, explicitly connecting the executed action perturbation, its observed consequence in GT, and the corresponding model prediction. The cases use the epoch-13 SAFF checkpoint. Intervention frames below refer to the aligned 81-frame sequences, indexed from 0 to 80.
1. Grasp-position offset: unsuccessful cube pickup (intervention frame 28). We offset the grasp target when placing a cube onto a stand. In GT, the offset contact tilts the cube without securing a grasp; the empty gripper moves away, leaving the cube on the table and the stand empty. Our prediction reproduces this sequence. The remaining errors concern the cube's contact pose, rotation, and local geometry. Ctrl-World exhibits cube deformation and motion toward the image boundary, with the cube no longer visible at the end.
2. Premature release: the object separates before cabinet placement (intervention frame 56). We open the gripper early while transporting a toy car. In GT, separation becomes visible around frame 59: the car returns to the table while the empty gripper continues toward the drawer. Our prediction reproduces this separation and subsequent object–gripper motion. Ctrl-World also captures the broad release outcome.
3. Suppressed release: failed block handover (intervention frame 54). We override the planned release so that the source-side gripper remains closed. In GT, the block returns with that gripper, leaving the receiving gripper empty. Our prediction preserves this return direction and object–gripper association, whereas Ctrl-World associates the block with the receiving side in the later frames.
The anonymous video page provides side-by-side GT/Ours/Ctrl-World playback for all three cases, including annotated full clips, half-speed clips around and after the intervention, and versions without event annotations.