- Rating
- 1
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates failure insensitivity in robot world models, showing that released models can produce success-like predictions on execution-verified failures. The authors propose CureWM, which constructs alternative actions from successful demonstrations across a severity grid, verifies their outcomes through execution, and fine-tunes released models on the resulting failures and surviving successes alongside nominal demonstrations, without changing their architecture or training objective. Experiments in simulation and on a physical robot arm show that CureWM reduces optimism.
Critical strengths
-
Performance. The paper separates optimism from discrimination. The distinction is valuable and is supported by the paper's own evidence. On LIBERO-10, optimism is only 2.5% while the false-positive rate reaches 99.4%. The unverified-label ablation does lower training optimism, but it does so by also pushing the values of successful actions down to 0.147.
-
Experiments. The comparison between counterfactual failures and policy failures matches the number of failures, the per-task counts, the successful replays, and the training steps. It is the most informative experiment in the paper.
-
Experiments. The hardware evaluation includes blinded assessments by three non-author raters, and the supplementary materials provide anonymized code and analysis scripts.
Critical weaknesses
-
Unclear presentation of experimental findings. The manuscript contains several statements indicating that a result “does not establish,” “does not guarantee,” or is “not statistically established.” These qualifications appear across multiple simulation experiments. I would like the authors to clarify which conclusions are supported by the reported results and provide more conclusive experimental evidence.
-
Insufficient evidence for the central claim. The authors do not convincingly demonstrate whether repairing failure prediction makes world models more useful for policy evaluation, planning, or synthetic data generation, which is the central motivation of this work. Carefully qualifying conclusions is important, but it cannot substitute for empirical evidence.
-
Missing related word and limited novelty. The motivation, experimental setup, models, and platforms used in this paper are similar to those in VLAW [1], yet the paper neither cites nor compares against this work. VLAW explicitly identifies the excessive optimism of world models trained only on demonstrations. On the DROID Franka platform, it fine-tunes Ctrl-World using real rollouts labeled as success or failure, reports fewer false-positive interaction predictions, and demonstrates policy improvement across five real-world task categories. The authors should therefore clearly articulate the differences from VLAW and provide evidence of improvements over it.
-
Limited original methodological contributions. The perturbation families used in this paper come from MiraBench. I consider “without changing the architecture or training objective” a constraint, which VLAW also satisfies. Setting these existing elements aside, the paper’s original contributions appear limited to two aspects: the specific construction of counterfactual failures through perturbations that preserve a shared action prefix with the source demonstration, and the comparison between counterfactual failures and policy failures.
[1] VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Model.
Questions for authors
The paper positions itself as repairing released checkpoints, so the most direct comparison should be with the released model before repair. Across the experiments that measure control performance, CureWM does not improve average performance, and performance declines in most cases. For example, applying CureWM to the released checkpoint lowers average task success on LIBERO, while every CureWM variant performs worse than the released model in RoboCasa. The authors acknowledge a “repair–control trade-off” on RoboCasa in their Limitations section. In other words, improvements in prediction metrics come at the expense of control performance. I consider this the most important aspect to improve, how to repair failure prediction while preserving the world model’s existing capabilities as much as possible.
4 comments
Dear Reviewer 7cNh,
Thank you again for your thoughtful review and constructive comments on our paper. We have posted our response to your review and hope it addresses your concerns.
We can totally understand this is a busy period. When you have a moment, we would greatly appreciate it if you could take a look and let us know whether anything remains unclear. We would be happy to provide further clarification or additional results if needed.
Thank you for your time and consideration.
Best, The authors
Thank you for your response, and I apologize for not reading and replying to it sooner. If time permits, I would appreciate a further response before the discussion deadline. I will reassess my rating in the next phase.
I appreciate the candor of your rebuttal and the additional experiments and analyses. My first two weaknesses have been addressed. Regarding Weakness 3, is there evidence that the proposed method offers advantages over VLAW?
Regarding my question about the loss of control performance during prediction repair, could this issue be mitigated by isolating parameter updates for prediction repair from those for action generation, or by addressing gradient conflicts between prediction and control objectives?
We thank the reviewer for the follow-up questions.
- Evidence regarding VLAW (W3). With the two failure sources matched (140 unique failures each, the same 580 successes, 7,500 and 17,500 steps, two seeds), the evidence indicates complementary strengths rather than a uniform advantage.
(a) Counterfactual failures repair sensitivity to action contrasts, and the policy's own failures (the VLAW recipe) do not: AUROC 0.57 to 0.61 against 0.52 on held-out counterfactuals, 0.55 to 0.58 against 0.50 on unseen severities, and 0.57 to 0.64 against 0.51 on counterfactuals built around the policy's own trajectories. The VLAW-style arm also lowers the value of successes (0.38 to 0.39 against 0.46 for the baseline, while the counterfactual arm stays at 0.43 to 0.45), and counterfactual data requires no policy failures (822 replays from 27 demonstrations, before any rollout exists).
(b) On the policy's own natural failures the VLAW-style arm is stronger (value 0.04 to 0.09 at step 75 against 0.27 to 0.36), as it was on the real robot (generated-end AUROC 0.937 against 0.866, with the scene confound stated in Section 5).
(c) The arm trained on both sources retains both properties (0.54 to 0.60 on the counterfactual sets, and 97 to 98 % of natural failures detected at the midpoint at a 10 % false-alarm budget).
The revision will state the comparison accordingly. Counterfactual repair is the less expensive, policy-independent and more transferable source, the VLAW recipe is stronger on the policy's own rollouts, and the two combine. A hardware comparison without the scene confound (counterfactuals and rollouts in the same scene, four arms from one binding checkpoint) is in progress and will be reported regardless of its direction.
- Isolating the updates to mitigate the control cost. Two reported arms bear on this. Masking the action loss on failed replays preserves the prediction repair (optimism 32.9 % against 30.2 %) but not the control performance (Goal 94.6 %, −1.87 [−4.00, −0.00] points), so the cost is not imitation of failed actions but a shift of the shared representation. Linear weight interpolation between the baseline and CureWM shows that the objectives are not irreconcilable: at weight 0.75 the model retains about three quarters of the repair (two seed pairs, 0.74 and 0.77) with no measurable closed-loop cost over 40 tasks (+0.12 [−0.45, +0.70] against the baselines and +0.70 [+0.10, +1.30] against CureWM). Explicit parameter isolation (a separate value and prediction pathway, adapters for the repair with the action pathway frozen, or gradient projection) is the natural next step.
We thank the reviewer. The control question is where most of the new compute went. New arms are at 17,500 steps like the paper's and intervals are bootstraps clustered by source trajectory (prediction) or paired by task (closed loop).
W1. Supported conclusions. Established over seeds (intervals exclude zero): (1) released and step-matched baselines are failure-insensitive on verified counterfactual failures (AUROC 0.495 to 0.499); (2) CureWM repairs this on the held-out, off-grid and unseen-source sets (+0.15 to +0.26 AUROC, four seeds); (3) verified labels are necessary (relabeled survivors: 0.545 to 0.583); (4) the repair transfers to the released policy's natural failures once under way (value drop 0.13 to 0.21, successes at most 0.03), not before; (5) the policy's own failures do not repair counterfactual sensitivity but are the stronger data on policy rollouts. Not established, said so in one limitations paragraph: same-demonstration survivors, cup-to-cube transfer, early anticipation, planning gains, hardware gains over the adaptation point.
W2. Usefulness. Monitoring the released policy (818 rollouts, 238 failures unseen by both arms), at 10 % false alarms: the baseline catches 41 % of failures at the midpoint and 79 to 81 % at three quarters, CureWM 61 to 65 % and 90 to 92 % (two seeds each). Ranking 30 perturbed controller variants (success 27 to 100 %) by mean predicted value: Spearman 0.31 to 0.66 (+0.345 [+0.236, +0.440]), calibration unchanged; variants of the trained families, not learned policies. Hardware policy evaluation (31 real policy conditions, no new robot time): counterfactual repair recovers part of the loss from success-only fine-tuning (+0.05 [-0.00, +0.10] AUROC), not beyond the adaptation point. Planning: no established gain. So: a late failure monitor and a ranker of perturbed variants, no planning or hardware evaluation gain by itself.
W3. VLAW. The revision cites it. VLAW fine-tunes Ctrl-World on the policy's own real rollouts labelled by outcome, without perturbation, counterfactuals or severity. Our policy-failure arm is that recipe under matched failures, per-task counts, successes and steps (7,500 for these 140-failure pools); the comparison cuts both ways. Counterfactual failures repair sensitivity on every counterfactual set (AUROC 0.606 / 0.584 / 0.635 against 0.518 / 0.498 / 0.510), need no policy in the loop and cost 0.33 GPU-minutes per verified failure against 7.9. The policy's own failures are stronger on policy rollouts: midpoint AUROC 0.925 against 0.755 on natural failures held out from both arms; on the robot, an arm trained on other policies' real rollouts predicts unseen policies at AUROC 0.94 against 0.83 (+0.11 [+0.03, +0.19]); counterfactual repair adds +0.04, interval containing zero (different camera placement, not a clean head-to-head). We claim no improvement over VLAW: the two sources repair different things, and an arm trained on both keeps both properties.
W4. Novelty. The revision states the contribution as those two elements plus the controlled finding above and the measured cost.
Q. Control cost. The reviewer is right. New: two seeds per arm on all 40 LIBERO tasks, 500 episodes per suite, task-paired intervals:
The submission's single seed (-0.25 [-1.30, +0.75]) hid a small, suite-dependent cost: Goal and LIBERO-10, not Spatial and Object. Fine-tuning itself costs more (baseline minus released -1.85 [-2.85, -0.90]; CureWM -2.07 [-3.00, -1.20]). The RoboCasa trade-off stands (OpenDrawer 17/30 at 5,000 steps against 28/30 released; 26/30 at 20,000). Mitigations: masking the action loss on failed replays keeps the prediction repair (optimism 32.9 % against 30.2 %) but not the control (Goal 94.6, -1.87 [-4.00, -0.00]); and interpolating weights between baseline and CureWM, one model per point:
The repair grows with the weight, the cost does not: the 0.75 point keeps about three quarters of the repair (+0.18 [+0.10, +0.27] over the baseline, -0.06 [-0.13, -0.02] under CureWM) at the baseline's success rate; with single-seed endpoints, a candidate trade-off, not a result. The revision reports the cost as a result and the interpolation as open.