- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This authors introduce a test-time planning method for deciding which parts of a world-model-predicted failed manipulation plan should be preserved rather than restarting the entire plan. The method is built around three main contributions. First, candidate action edits are scored according to their effect on the same diagnosed downstream failure, allowing the algorithm to identify useful repairs that occur before the first visible symptom. Second, paired world-model predictions provide evidence about whether an edited prefix should be retained, while a value model decides whether collecting additional evidence is worth the search candidates that would be sacrificed to obtain it. Finally, the method is evaluated under identical prediction budgets with a frozen World Action Planner, reaching 84.6% macro success compared with 80.7% for the strongest baseline, with the largest gains on long-horizon tasks. The paper also studies failure localization, adaptive versus fixed-depth confirmation, calibration transfer, world-model counterfactual fidelity, natural failures, and coupled two-fault cases.
Critical strengths
S1: The paper defines a well motivated formulation of plan reuse as a counterfactual problem. The method goes beyond simply identifying a likely failure source but rather it evaluates candidate action interventions according to their effect on a common diagnosed downstream failure, treats each successful intervention as defining a constrained search branch, and explicitly trades off additional evidence about that branch against the search capacity consumed by obtaining it.
S2: The evaluation section compares against multiple strong baselines, offers useful ablations that isolate different pieces of their method, and tests under different budgets.
S3: The authors provide calibration quality and sensitivity analysis. They provide metrics in cases where their assumptions fall apart (two-fault vs single-fault) and they are frank about the very large calibration-time of the method.
Critical weaknesses
W1: The large calibration cost is problematic. The same-family experiment reports 54,000 full-rollout equivalents when using 80 development origins per family. While this price is paid offline, it does weaken the interpretation that the proposed method simply makes better use of “the same compute”.
W2: While the evaluation is robust, the experiments rely on specific manipulation structures: aquisition/transport/release, observable object geometry, calibrated cameras etc. The authors do not establish that the core method transfers to substantially different world models.
W3: The authors mention this as a weakness: the method nominates a single action edit, interacting early faults cannot be represented this way and therefore their method suffers on the two-fault cohort. (84.6% macro success vs 75.0%)
Questions for authors
Q1: Ablation Table 21 is very interesting and i think should be in the main paper
Q2: Table 3 should probably add the full rollout equivalents that are reported in table 16 in the appendix. The authors mention it in text, but it would probably be clearer to add these values to the table
4 comments
Response to Reviewer PGRe (2/3): other world models and the manipulation structure (W2)
W2: While the evaluation is robust, the experiments rely on specific manipulation structures … The authors do not establish that the core method transfers to substantially different world models.
We agree: the paper does not establish this. The experiments use three released checkpoints of one video architecture and four manipulation families. The limitations will say that transfer to another world-model architecture, and to tasks without this structure, is untested. What we can do is state which part of the method depends on what, and what the paper's evidence covers for each part:
| Part of the method | What it assumes | What the paper's evidence covers |
|---|---|---|
| World-model interface | A black box that predicts a plan prefix from an observation history. No gradients, latent states or fine-tuning | Three checkpoints of one architecture (Table 5). Shared noise in the paired checks is not essential: independent noise changes success by 0.7 points [−0.7, 2.0] (Table 21) |
| Failure measure and edit proposer | Acquisition, transport and release phases, object geometry from RGB, calibrated cameras | Specific to this setting. The same perception is used by every planner in Table 2 |
| Decision layer (value model, stopping rule) | Only check outcomes, the nomination margin, the retained fraction and the candidate count | Fitted on three families and frozen for the fourth: 84.6%, with a better Brier score than the prevalence baseline in every held-out family (Table 17) |
Two further facts bear on your question.
- Unseen checkpoints. For both LIBERO families, the held-out family uses a checkpoint on which the value model was never fitted (Table 5). The margins over the strongest baseline there are 1.3 points (LIBERO-Object) and 8.7 points (LIBERO-Long). This is transfer across checkpoints of one architecture, and no more.
- What a new world model must provide. The method relies on one property: an edited action must change the prediction the way it changes the execution. The fidelity audit measures this without any planner (Tables 1 and 23). On our model, 92–95% of predictions are valid, the direction of the response agrees with execution in 82–87% of cases, and delayed failures are recalled in 78–83%. For a new world model this audit is the test to run first, because extra checks cannot repair a systematic mismatch.
The revision will add a "requirements and scope" paragraph with these three dependencies and the audit as the entry test.
Response to Reviewer PGRe (3/3): the two-fault cohort (W3); Table 21 (Q1)
W3: The authors mention this as a weakness: the method nominates a single action edit, interacting early faults cannot be represented this way and therefore their method suffers on the two-fault cohort. (84.6% macro success vs 75.0%)
Agreed: this is the boundary of the method. Two facts from the paper put the drop in context (Tables 2, 24 and 25).
The cohort is harder for every planner (success in %):
| Planner | Single fault (400 origins) | Two coupled faults (80 origins) | Difference | Joint-edit failures (of 240 runs) |
|---|---|---|---|---|
| MPPI | 74.8 | 65.0 | −9.8 | 28 |
| Geo-gated selection | 80.7 | 68.8 | −11.9 | 30 |
| ROOTCAUSE-F2 | 80.1 | 67.5 | −12.6 | 21 |
| ROOTCAUSE | 84.6 | 75.0 | −9.6 | 20 |
ROOTCAUSE loses 9.6 points. MPPI and Geo-gated selection, which may change every action, lose 9.8 and 11.9. ROOTCAUSE remains the best of the six planners tested. The two cohorts differ and the second has 80 origins, so we read these differences as indicative only.
The single-edit limit accounts for a third of our failures there. Of the 60 failed runs, 20 need coordinated edits before the symptom, which one nomination cannot express (Table 25). The others are wrong nominations (12), world-model mismatch (11), ranking errors (10), budget exhaustion (4) and unresolved cases (3). The planners that may change every action have more joint-edit failures on this cohort (28 and 30 runs), so freeing all actions does not solve these cases at this budget either.
What we will do. Nominating a pair of edits changes only the nomination step: the branch definition, the paired checks and the decision rule stay as they are. We have not evaluated this, and the paper does not claim it. The limitation is stated in the conclusion. The revision will also state it in Sec. 3.3 next to the two-fault result, with the failure breakdown of Table 25.
Q1: Ablation Table 21 is very interesting and i think should be in the main paper
We will move it. A condensed Table 21 will sit in Sec. 3.2, next to the paragraph it supports. The additional page allowed for the revision makes room for it. Its main message is that the nomination is the one large effect: replacing it by local scoring costs 6.2 points [4.2, 8.3]. Removing the margin, the retained fraction or the budget input from the value model changes success by at most 0.5 points, within noise.
We are glad to clarify anything further. If these answers address your concerns, we would be grateful if you would consider updating your assessment.
The authors rebuttal have convinced me to keep my positive score.
Response to Reviewer PGRe (1/3): the offline calibration cost (W1, Q2)
Thank you for the careful reading, and for separating what the experiments establish from what they do not. We agree with all three weaknesses as statements of scope. Part 1 covers the calibration cost (W1, Q2), Part 2 other world models (W2), and Part 3 the two-fault cohort (W3) and Q1. The changes we describe will be in the revised PDF.
You are right, and the wording will change. "Matched budget" refers to the online prediction budget per decision. The value model adds a one-time offline cost on top. The abstract, the contributions and Sec. 3 will say "matched online budget", and the limitations will state the offline cost.
We will add them. The revised Table 3 will report the fit cost next to the success it buys. The same information at 32 rollouts per decision:
Same-family costs are from Table 16. A leave-one-family-out model is fitted on 300 origins, at the per-origin cost implied by Table 16 (about 169). These figures count world-model generation; the simulator executions that label the development plans are accounted separately, as in the paper. The strongest baseline, Geo-gated selection, has no offline cost and reaches 80.7% with 32 rollouts and 86.5% with 64.
What the table says.
When "the same compute" holds. Charge the offline fit to our method: if one fitted model serves N decisions, each decision costs 32 + 50,700/N rollouts. Give Geo-gated selection the same total:
The claim that holds is therefore narrower than "the same compute". The planners use equal online compute, and the offline investment pays off only when one fitted model serves more than roughly 2,500 decisions. With fewer decisions, the strongest baseline with the same total compute is ahead (86.5% against 84.6% at 1,600 decisions). The middle row is an interpolation, not a measurement. The revision will state this in Sec. 3.2 and in the limitations.