acceptodds
Under review as a conference paper at ICLR 2027

What Should Survive a Failed Plan? Counterfactual Reuse with World Models

Abstract

World models can reject a plan before execution, but a failed prediction does not say which earlier decisions the next search should keep. We formulate this problem as counterfactual plan reuse: nominate a single action edit, decide whether to retain the prefix it defines, and search for a completion under a shared prediction budget. ROOTCAUSE scores every candidate edit by its effect on the same diagnosed failure, so a repair can be placed before the first visible symptom, then decides with paired world-model predictions whether the resulting branch deserves the search it would concentrate. Its central quantity is the value of one more check against the candidates that check displaces, so confirmation depth follows the budget instead of being fixed in advance. On four manipulation families with a frozen World Action Planner, ROOTCAUSE raises matched-budget success from 80.7% for the strongest baseline to 84.6%, with the largest gains on long-horizon tasks and a margin over fixed-depth confirmation that grows from 0.3 to 4.6 points as the budget quadruples. Held-out audits of action-response fidelity and calibration transfer locate the regime in which retaining actions pays and where it stops.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
world models, test-time planning, robot manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PGReOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This authors introduce a test-time planning method for deciding which parts of a world-model-predicted failed manipulation plan should be preserved rather than restarting the entire plan. The method is built around three main contributions. First, candidate action edits are scored according to their effect on the same diagnosed downstream failure, allowing the algorithm to identify useful repairs that occur before the first visible symptom. Second, paired world-model predictions provide evidence about whether an edited prefix should be retained, while a value model decides whether collecting additional evidence is worth the search candidates that would be sacrificed to obtain it. Finally, the method is evaluated under identical prediction budgets with a frozen World Action Planner, reaching 84.6% macro success compared with 80.7% for the strongest baseline, with the largest gains on long-horizon tasks. The paper also studies failure localization, adaptive versus fixed-depth confirmation, calibration transfer, world-model counterfactual fidelity, natural failures, and coupled two-fault cases.

Critical strengths

S1: The paper defines a well motivated formulation of plan reuse as a counterfactual problem. The method goes beyond simply identifying a likely failure source but rather it evaluates candidate action interventions according to their effect on a common diagnosed downstream failure, treats each successful intervention as defining a constrained search branch, and explicitly trades off additional evidence about that branch against the search capacity consumed by obtaining it.

S2: The evaluation section compares against multiple strong baselines, offers useful ablations that isolate different pieces of their method, and tests under different budgets.

S3: The authors provide calibration quality and sensitivity analysis. They provide metrics in cases where their assumptions fall apart (two-fault vs single-fault) and they are frank about the very large calibration-time of the method.

Critical weaknesses

W1: The large calibration cost is problematic. The same-family experiment reports 54,000 full-rollout equivalents when using 80 development origins per family. While this price is paid offline, it does weaken the interpretation that the proposed method simply makes better use of “the same compute”.

W2: While the evaluation is robust, the experiments rely on specific manipulation structures: aquisition/transport/release, observable object geometry, calibrated cameras etc. The authors do not establish that the core method transfers to substantially different world models.

W3: The authors mention this as a weakness: the method nominates a single action edit, interacting early faults cannot be represented this way and therefore their method suffers on the two-fault cohort. (84.6% macro success vs 75.0%)

Questions for authors

Q1: Ablation Table 21 is very interesting and i think should be in the main paper

Q2: Table 3 should probably add the full rollout equivalents that are reported in table 16 in the appendix. The authors mention it in text, but it would probably be clearer to add these values to the table

4 comments
AuthorsOpenReview ↗

Response to Reviewer PGRe (1/3): the offline calibration cost (W1, Q2)

Thank you for the careful reading, and for separating what the experiments establish from what they do not. We agree with all three weaknesses as statements of scope. Part 1 covers the calibration cost (W1, Q2), Part 2 other world models (W2), and Part 3 the two-fault cohort (W3) and Q1. The changes we describe will be in the revised PDF.

W1: The large calibration cost is problematic. … While this price is paid offline, it does weaken the interpretation that the proposed method simply makes better use of “the same compute”.

You are right, and the wording will change. "Matched budget" refers to the online prediction budget per decision. The value model adds a one-time offline cost on top. The abstract, the contributions and Sec. 3 will say "matched online budget", and the limitations will state the offline cost.

Q2: Table 3 should probably add the full rollout equivalents that are reported in table 16 in the appendix.

We will add them. The revised Table 3 will report the fit cost next to the success it buys. The same information at 32 rollouts per decision:

Value-model fitDevelopment originsOffline cost (full-rollout eq.)Same cost in online decisionsMacro success (%)
None: ROOTCAUSE-NoChk00074.8
None: ROOTCAUSE-F2 (fixed depth)00080.1
Same family, four models4 × 53,36910576.5
Same family, four models4 × 106,74621178.1
Same family, four models4 × 2013,49842280.2
Same family, four models4 × 4027,05384582.5
Same family, four models4 × 8054,1041,69184.9
Leave-one-family-out, one model (headline)300, none from the evaluation familyabout 50,700about 1,60084.6

Same-family costs are from Table 16. A leave-one-family-out model is fitted on 300 origins, at the per-origin cost implied by Table 16 (about 169). These figures count world-model generation; the simulator executions that label the development plans are accounted separately, as in the paper. The strongest baseline, Geo-gated selection, has no offline cost and reaches 80.7% with 32 rollouts and 86.5% with 64.

What the table says.

  • Calibration is what buys the margin over the strongest baseline. Without it, ROOTCAUSE-F2 is level with Geo-gated selection (80.1% and 80.7%). With it, the margin is 3.9 points [2.0, 5.8].
  • Too little calibration hurts. With 5 or 10 origins per family, the calibrated controller is worse than the calibration-free rule (76.5% and 78.1% against 80.1%).
  • The cost is not paid per task family. A leave-one-family-out model sees no data from the family it is evaluated on and reaches 84.6%, on par with 80 in-family origins (84.9%).
  • At a small budget it is not worth paying. With 16 rollouts, calibration adds 0.3 points [−1.9, 2.6] over the calibration-free rule (Table 7).

When "the same compute" holds. Charge the offline fit to our method: if one fitted model serves N decisions, each decision costs 32 + 50,700/N rollouts. Give Geo-gated selection the same total:

One fitted model servesOur cost per decision, offline share includedGeo-gated selection with that budgetROOTCAUSE
1,600 decisions64 rollouts86.5% (measured)84.6%
2,500 decisionsAbout 52 rolloutsAbout the same (interpolated between its 80.7% and 86.5%)84.6%
Many more32 rollouts80.7% (measured)84.6%

The claim that holds is therefore narrower than "the same compute". The planners use equal online compute, and the offline investment pays off only when one fitted model serves more than roughly 2,500 decisions. With fewer decisions, the strongest baseline with the same total compute is ahead (86.5% against 84.6% at 1,600 decisions). The middle row is an interpolation, not a measurement. The revision will state this in Sec. 3.2 and in the limitations.

AuthorsOpenReview ↗

Response to Reviewer PGRe (2/3): other world models and the manipulation structure (W2)

W2: While the evaluation is robust, the experiments rely on specific manipulation structures … The authors do not establish that the core method transfers to substantially different world models.

We agree: the paper does not establish this. The experiments use three released checkpoints of one video architecture and four manipulation families. The limitations will say that transfer to another world-model architecture, and to tasks without this structure, is untested. What we can do is state which part of the method depends on what, and what the paper's evidence covers for each part:

Part of the methodWhat it assumesWhat the paper's evidence covers
World-model interfaceA black box that predicts a plan prefix from an observation history. No gradients, latent states or fine-tuningThree checkpoints of one architecture (Table 5). Shared noise in the paired checks is not essential: independent noise changes success by 0.7 points [−0.7, 2.0] (Table 21)
Failure measure and edit proposerAcquisition, transport and release phases, object geometry from RGB, calibrated camerasSpecific to this setting. The same perception is used by every planner in Table 2
Decision layer (value model, stopping rule)Only check outcomes, the nomination margin, the retained fraction and the candidate countFitted on three families and frozen for the fourth: 84.6%, with a better Brier score than the prevalence baseline in every held-out family (Table 17)

Two further facts bear on your question.

  • Unseen checkpoints. For both LIBERO families, the held-out family uses a checkpoint on which the value model was never fitted (Table 5). The margins over the strongest baseline there are 1.3 points (LIBERO-Object) and 8.7 points (LIBERO-Long). This is transfer across checkpoints of one architecture, and no more.
  • What a new world model must provide. The method relies on one property: an edited action must change the prediction the way it changes the execution. The fidelity audit measures this without any planner (Tables 1 and 23). On our model, 92–95% of predictions are valid, the direction of the response agrees with execution in 82–87% of cases, and delayed failures are recalled in 78–83%. For a new world model this audit is the test to run first, because extra checks cannot repair a systematic mismatch.

The revision will add a "requirements and scope" paragraph with these three dependencies and the audit as the entry test.

AuthorsOpenReview ↗

Response to Reviewer PGRe (3/3): the two-fault cohort (W3); Table 21 (Q1)

W3: The authors mention this as a weakness: the method nominates a single action edit, interacting early faults cannot be represented this way and therefore their method suffers on the two-fault cohort. (84.6% macro success vs 75.0%)

Agreed: this is the boundary of the method. Two facts from the paper put the drop in context (Tables 2, 24 and 25).

The cohort is harder for every planner (success in %):

PlannerSingle fault (400 origins)Two coupled faults (80 origins)DifferenceJoint-edit failures (of 240 runs)
MPPI74.865.0−9.828
Geo-gated selection80.768.8−11.930
ROOTCAUSE-F280.167.5−12.621
ROOTCAUSE84.675.0−9.620

ROOTCAUSE loses 9.6 points. MPPI and Geo-gated selection, which may change every action, lose 9.8 and 11.9. ROOTCAUSE remains the best of the six planners tested. The two cohorts differ and the second has 80 origins, so we read these differences as indicative only.

The single-edit limit accounts for a third of our failures there. Of the 60 failed runs, 20 need coordinated edits before the symptom, which one nomination cannot express (Table 25). The others are wrong nominations (12), world-model mismatch (11), ranking errors (10), budget exhaustion (4) and unresolved cases (3). The planners that may change every action have more joint-edit failures on this cohort (28 and 30 runs), so freeing all actions does not solve these cases at this budget either.

What we will do. Nominating a pair of edits changes only the nomination step: the branch definition, the paired checks and the decision rule stay as they are. We have not evaluated this, and the paper does not claim it. The limitation is stated in the conclusion. The revision will also state it in Sec. 3.3 next to the two-fault result, with the failure breakdown of Table 25.

Q1: Ablation Table 21 is very interesting and i think should be in the main paper

We will move it. A condensed Table 21 will sit in Sec. 3.2, next to the paragraph it supports. The additional page allowed for the revision makes room for it. Its main message is that the nomination is the one large effect: replacing it by local scoring costs 6.2 points [4.2, 8.3]. Removing the margin, the retained fraction or the budget input from the value model changes success by at most 0.5 points, within noise.

We are glad to clarify anything further. If these answers address your concerns, we would be grateful if you would consider updating your assessment.

The authors rebuttal have convinced me to keep my positive score.

Reviewer_CuAvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates how to reuse portions of failed plans when planning with a world model. In particular, the proposed method identifies the point responsible for a failure, rather than merely the point at which the failure manifests, and replans accordingly. Across four experimental environments, the method achieves higher success rates than the baselines, with particularly large improvements in long-horizon planning.

Critical strengths
  • Sound motivation: Improving a plan by identifying the root cause of a failure, rather than simply reacting to where the failure becomes apparent, is a conceptually well-motivated approach.
  • Detailed methodological description: The proposed method is explained thoroughly and concretely, with accompanying mathematical formulations.
  • Extensive empirical evaluation: Beyond comparisons with baselines, the experiments demonstrate that the method can achieve better performance under limited computational budgets.
Critical weaknesses
  • Overly specific modeling assumptions: The setting assumed in this paper appears rather specialized. For example, identifying where to revise a plan based on valid geometry and deviations from it, as well as the algorithm for determining how to revise plans using those deviations, could be viewed as a specific instantiation of a more general search algorithm.
  • The sequential nature of the search may incur longer wall-clock times: Identifying problematic portions of a plan and subsequently replanning introduces sequential dependencies in generation, which could lead to longer planning times.
Questions for authors
  • What are the strengths of the proposed method compared with traditional tree search? Could it be viewed as a particular instantiation of tree search?
  • The score in Equation (1) appears to identify the time step with the greatest influence on the endpoint, rather than necessarily identifying the root cause of the failure. Is it justified to interpret this as identifying the root cause?
  • In Table 26, all methods have similar mean latencies. Were these measurements obtained with parallelization wherever possible? For example, full-root could generate 32 candidates in parallel.
4 comments
AuthorsOpenReview ↗

Response to Reviewer CuAv (1/3): relation to tree search

Thank you for the careful review and the accurate summary of the paper. We answer your points in three parts: the relation to tree search in this part, the specificity of the method and the "root cause" interpretation in Part 2, and wall-clock time and presentation in Part 3. The changes we describe will be in the revised PDF.

What are the strengths of the proposed method compared with traditional tree search? Could it be viewed as a particular instantiation of tree search?

Yes, it can be viewed as a tree search, and the revised Sec. 2 will present it this way:

Traditional tree searchROOTCAUSE
NodeA partial plan, extended step by stepA set of retained actions. The root retains nothing. A child fixes the actions before one edit and the edit itself (Eq. 2)
Which children are createdSuccessors in temporal order, or a restart at the last verified stepOne child, proposed by a geometric replay that makes no world-model call. It may lie before the visible symptom (Eq. 1)
How a child is evaluatedRollouts or a learned value functionPaired world-model predictions up to the symptom, with shared noise, charged to the budget
How much evaluationA fixed number of visits, or a bandit ruleZero to four pairs, chosen by the value of one more check against the candidates it displaces (Eqs. 6–7)
Below the chosen nodeFurther expansionSample complete candidates, rank them, execute the best

Table 2 does not include MCTS itself. It includes several simpler searches of this kind, under the same budget (32 rollouts per decision, success in %):

PlannerChildEvaluation before committingSuccess
Full-rootNone: root onlyNone63.5
TBS-style adapterRestart at the last verified boundaryNone75.5
ROOTCAUSE-NoChkBest edit by Eq. 1None74.8
Paired branch searchTwo nominated anchorsTwo paired rounds, fixed78.9
ROOTCAUSE-F2Best edit by Eq. 1At most two pairs; keep while they pass80.1
ROOTCAUSEBest edit by Eq. 1Zero to four pairs; decide by calibrated value84.6

The strongest baseline, Geo-gated selection (80.7%), reranks complete candidates and is not a search over retained actions.

What the method adds to a traditional tree search.

  • It creates a child without spending budget, and before the symptom. A traditional search finds good children by evaluating them. Here one evaluation is a video prediction at about 7 s per block, a decision can afford 32 rollouts, and one level of single-edit children already holds up to 24 proposals at each of up to 18 steps. Restarting where the failure becomes visible avoids this cost but cannot reach an earlier cause: for faults before the symptom it yields a useful branch in 49.2% of cases, against 68.6% for our nomination (Table 12). Replacing the nomination by local scoring costs 6.2 points [4.2, 8.3] (Table 21).
  • It decides on calibrated value, not on pass counts. After each check, the planner compares the calibrated success of the search that would remain if it keeps the child and if it reopens the plan. It buys another check only when the information is worth the candidates that the check displaces (Eq. 6). This is value-of-computation reasoning, which has also been used to select computations in Monte Carlo tree search (Russell & Wefald, 1991; Hay et al., 2012). Rules that keep the child while its checks pass give 78.8%, 80.1% and 79.4% with at most one, two and four pairs. Our rule gives 84.6%, which is 4.5 points [2.7, 6.4] above the two-pair rule (Table 7).
  • Neither part is enough alone. Committing to the nominated child unchecked gives 74.8%, no better than restarting at the last verified boundary (75.5%).
AuthorsOpenReview ↗

Response to Reviewer CuAv (2/3): what is general and what is specific; what Eq. 1 identifies

Overly specific modeling assumptions: The setting assumed in this paper appears rather specialized. … could be viewed as a specific instantiation of a more general search algorithm.

We agree with this view. The revision will separate the general search of Part 1 from its two manipulation-specific modules:

ModuleSpecific to manipulation?What another domain would supply
Failure measureYes: acquisition centering, object–gripper displacement, clearance and support error from RGB geometryAny scalar failure measure over time, with a threshold
Edit proposer (geometric replay)Yes: the object follows the gripper and is tested against the supportAny cheap way to score an edit by its effect on the diagnosed failure
Branch, paired checks, value model, stopping rule (Eqs. 2–7)No: the inputs are check outcomes, the nomination margin, the retained fraction and the candidate countNothing new

The evidence for the last row is that Table 2 uses leave-one-family-out calibration: the decision layer is fitted on three task families and applied unchanged to the fourth. We do not claim more. All four families are manipulation tasks with one world-model architecture, and the limitations will say so.

The score in Equation (1) appears to identify the time step with the greatest influence on the endpoint, rather than necessarily identifying the root cause of the failure. Is it justified to interpret this as identifying the root cause?

Your reading is correct, and the wording will change. Eq. 1 does not identify a cause in a strict sense. For each step up to the symptom and each legal replacement, it measures how much the diagnosed failure would drop if only that action changed and all other commands stayed fixed. It returns the edit with the largest drop. This is the most effective single-action repair. It can differ from the cause, for example when an early error can be compensated by a later action.

How often the two coincide. With single injected faults the cause is known (Table 12, %):

Rule for choosing the siteFault before the symptom: injected action foundFault before the symptom: useful branchFault at the symptom: injected action foundFault at the symptom: useful branch
Eq. 172.168.680.365.8
Local scoring at each edit's own endpoint38.556.079.256.4
Symptom boundary0.0 by construction49.296.150.8

A branch is useful when at least 20% of its sampled completions succeed in execution. Eq. 1 finds the injected action in 72.1% of the cases in which it lies before the symptom. When the fault is at the symptom, it is less precise than the symptom boundary (80.3% against 96.1%).

Why the planner does not need the cause.

  • It needs a prefix worth keeping. What matters for the search is whether the retained actions still admit successful completions. By that measure Eq. 1 is the best rule in both cases (68.6% and 65.8% useful branches).
  • The nominated edit is a hypothesis, not a conclusion. Paired world-model checks test it before the planner commits. Committing to it unchecked gives 74.8% success, and testing it gives 84.6%. The difference of 9.8 points [7.7, 11.9] is what it would cost to assume that Eq. 1 has found the root cause.

Changes in the revision.

  • The text will say "failure-conditioned repair site" where it implied identification, and Sec. 2.1 will state that Eq. 1 selects a repair that is then tested.
  • The introduction says that ROOTCAUSE retains "the earliest edit that removes" the failure. Eq. 1 selects the highest-scoring edit, and the sentence will be corrected.
  • The name will be explained as what the nomination can reach, a site before the symptom, and not as a guarantee.
AuthorsOpenReview ↗

Response to Reviewer CuAv (3/3): wall-clock time with parallel workers; presentation

In Table 26, all methods have similar mean latencies. Were these measurements obtained with parallelization wherever possible? For example, full-root could generate 32 candidates in parallel.

No, not in that sense. All planners were timed with the same inference settings, and for every planner the latency in Table 26 equals the number of generated blocks times about 6.9 s: 2,444 s for 351.9 blocks (Full-root) and 2,418 s for 348.2 blocks (ROOTCAUSE). The latencies are similar because all planners receive the same block budget. None of them shows what a planner could gain by generating its candidates on parallel workers. The revised caption will say this.

The sequential nature of the search may incur longer wall-clock times: Identifying problematic portions of a plan and subsequently replanning introduces sequential dependencies in generation, which could lead to longer planning times.

Yes, with parallel workers it does. The table gives the wall-clock time of one decision with G workers, in units of one complete rollout (42 to 125 s, depending on the task family). Predictions within a stage run in parallel, and stages run in sequence. The table is computed from each planner's stages and, for ROOTCAUSE, from the logged stopping depths (Table 9: 18.2%, 36.6% and 45.2% of decisions stop after one, two and four pairs). It is not a new measurement.

PlannerStages in sequenceG = 1G = 8G = 32Success (%)
Full-rootOne324163.5
MPPIFour generations of 8324474.8
ROOTCAUSE, at most two pairsDiagnosis, up to 2 check batches, final search32at most 6.5at most 3.583.3
ROOTCAUSEDiagnosis, 1 to 3 check batches, final search32at most 7.3at most 4.384.6

The ROOTCAUSE entries are averages over decisions. They are upper bounds because a check predicts only up to the symptom and is shorter than a complete rollout.

  • With one worker there is no penalty. All planners generate the same number of blocks.
  • With 32 workers the penalty is real. A one-shot sampler finishes in the time of one rollout. ROOTCAUSE needs up to 4.3, about as many as MPPI.
  • The depth is bounded and can be traded for success. A decision has at most five stages. The default schedule already runs the last two pairs as one batch, with the same success as checking one pair at a time (84.6%, Table 11). Capping at two pairs removes a stage and changes success by −1.2 points [−2.9, 0.4] (Table 11).

What we do not claim. The paper fixes the compute budget. If workers are plentiful and only wall-clock time matters, another planner can spend a larger budget in the time our method needs. Full-root with 64 rollouts reaches 75.3%, still below ROOTCAUSE with 32 (84.6%), but Geo-gated selection with 64 rollouts reaches 86.5% (Table 6). The revision will state this limit in Sec. 3.3 and in the limitations.

Presentation. You rated the presentation 1, and we take this seriously. The revision will make four changes:

  1. The introduction will open with the setting (a learned video world model and a fixed budget of predictions) and the motivation.
  2. Sec. 2 will open with the overview below and a worked example, before any equation.
  3. Sec. 2 will add the tree-search view of Part 1 and the separation of general and specific modules of Part 2.
  4. The appendix will add a notation table.

The overview: (1) Diagnose. Predict the nominal plan twice and find the earliest point at which a geometric failure measure crosses its threshold in both predictions. (2) Nominate. A geometric replay, with no world-model call, scores up to 24 replacements at each step up to that point by how much the failure would drop. (3) Branch. Keep the actions before the best edit, fix the edit, and free all later actions. (4) Decide. Commit to the branch, reopen the whole plan, or buy a paired check, whichever gives the final search the highest calibrated chance of success. (5) Search. Spend the remaining budget on complete candidates, rank them, and execute the best.

The example: a 12-step plan with 32 rollouts has 384 blocks. Diagnosis costs 24. If the symptom is at step 8, one check costs 16 blocks, and four checks leave 24 complete candidates instead of 30.

We are glad to clarify anything further. If these answers address your concerns, we would be grateful if you would reconsider your assessment.

Thank you for the detailed response.

Let me first explain why I asked about the comparison with tree search. Tree search can also go back to the point where a problem arises and find a branch that works. For example, if one searches for an alternative to an existing plan with a method such as MCTS, the search would find the point responsible for the failure along the way. From this perspective, I wanted to see what distinguishes the proposed method from existing tree search.

The response answered my questions clearly. It clarified that Eq. 1 selects the most effective repair site, not a cause in a strict sense, that the latencies in Table 26 were measured without parallelization, and that the sequential structure has a wall-clock cost with parallel workers. I appreciate that these points will be reflected in the paper.

However, the main reason for my assessment remains. As the response also acknowledges, the proposed method can be viewed as a specific form of tree search, and the part that distinguishes it within that view, the choice of the branching point, is specific to manipulation. My assessment of the scope of the contribution has therefore not changed, and I will keep my current score.

Reviewer_K7gjOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The authors propose analysing a plan that is predicted to fail by what they term a "world model" to guess which parts of the plan are (not) responsible for the failure.

Critical strengths

None that I am able to identify.

Critical weaknesses

World models used for planning provide the cause of a plan's failure (such as unsatisfied preconditions, no collision-free trajectory, etc). The authors appear to assume the model does not: it only accepts or rejects a complete plan (whereas actual planning models can also be queried for executability of plan prefixes). No motivation for this problem setting is provided. The introduction has only a simple example, then the paper dives into details. There is no explanation of why the problem is of interest.

Questions for authors

What, if any, is the motivation for this problem?

4 comments
AuthorsOpenReview ↗

Response to Reviewer K7gj (1/2): which kind of world model, and what it does and does not tell the planner

Thank you for the review. It shows us that the introduction never says which kind of world model the paper is about, so a reader can reasonably take it to be a symbolic planning model. We have rewritten the introduction. We answer your points in the order you raised them: the setting in this part, the motivation and the introduction in Part 2.

Short answer to your question. The paper is about learned video world models, which are slow to query and report no cause of failure. After such a model predicts a failure, the planner must still decide which earlier decisions to keep. Under the same prediction budget, reopening everything succeeds in 63.5% of cases, the strongest baseline in 80.7%, and our method in 84.6% (Table 2).

World models used for planning provide the cause of a plan's failure (such as unsatisfied preconditions, no collision-free trajectory, etc).

That is true of the planning models you describe. It is not true of the model in this paper. We use "world model" in the sense common in machine learning (Ha & Schmidhuber, 2018; Hafner et al., 2019–2025): a learned model that predicts future observations from past observations and actions. Ours predicts future camera images from past images and robot commands.

Symbolic planning modelLearned video world model (this paper)
What it isOperators with preconditions and effects, with a motion plannerA frozen video generator conditioned on robot commands
What a query returnsWhether a plan or prefix is executable, and which condition failsPredicted camera frames for a plan or prefix
How a failure shows upAn unsatisfied precondition, or no collision-free trajectoryA geometric error that a perception pipeline measures on the predicted frames
Is the cause given?Largely, through the violated conditionNo. The frames show when the failure becomes visible, not which action caused it
Cost of a queryLow for symbolic checks, higher for motion planningAbout 7 s per predicted block; a decision with a budget of 32 rollouts takes about 40 minutes (Table 26)

In task-and-motion planning, a motion-planning failure is indeed passed back to the task planner in symbolic form, for example as the object that obstructs the motion (e.g., Srivastava et al., 2014). A video model has nothing of this kind to report.

The authors appear to assume the model does not: it only accepts or rejects a complete plan (whereas actual planning models can also be queried for executability of plan prefixes).

We do not assume this, and the paper should have said so plainly. The video model predicts one block per plan step and joins the blocks in sequence, so it can be queried on any prefix, at a cost proportional to the prefix length (Sec. 2.1). The method relies on this twice. It examines the prediction of the nominal plan from the start to find the earliest point at which the predicted failure becomes visible, and its later checks predict only up to that point (Sec. 2.2).

What a prefix query cannot give is the action responsible. Take the example of the introduction: the robot predicts that a cube will land beside a stack. An off-center grasp and a misplaced release produce the same visible outcome, but the repairs differ. If the grasp is stable, the acquisition and transport commands are worth keeping and only the placement should change. If it is not, the repair has to happen before the cube leaves the table. The visible symptom is the same in both cases, so it cannot tell the planner which part of the plan to keep. Part 2 shows how much that decision matters.

AuthorsOpenReview ↗

Response to Reviewer K7gj (2/2): the motivation; the rewritten introduction

No motivation for this problem setting is provided. … There is no explanation of why the problem is of interest.

What, if any, is the motivation for this problem?

Why this setting. Learned world models are now used to test robot plans before execution (Finn & Levine, 2017; Hansen et al., 2024; Zhou et al., 2025; Zhang & Du, 2026). They need no hand-written operators or preconditions: they are learned from recorded robot data, and the video model we use works directly from camera images. The price is the one in the table of Part 1: the model reports no cause of failure, and each query is slow.

Why the problem. After such a model predicts that a plan will fail, the planner has a fixed number of further predictions in which to find a better plan. It can reopen the whole plan, or keep the part that was sound and search only the rest. Keeping the right part concentrates the search where it is needed. Keeping a wrong part removes the solutions from the search. The paper is about making this choice.

How much the choice matters. All planners below receive the same prediction budget (Table 2, 32 rollouts per decision, success in %):

What the planner does after a predicted failurePlannerAll four task familiesLong-horizon family
Reopens every actionFull-root63.539.0
Re-optimizes the whole sequenceMPPI74.856.0
Restarts at the last verified boundaryTBS-style adapter75.557.0
Filters and reranks complete candidates by geometric evaluationGeo-gated selection (strongest baseline)80.765.0
Proposes a repair, tests it within the same budget, and keeps the plan up to that repair if the test supports itROOTCAUSE (ours)84.674.0
  • Queries are scarce. One decision takes about 40 minutes (Table 26), so how the budget is spent after a rejected plan decides the outcome.
  • Restarting where the failure becomes visible is what prefix queries offer, and it is not enough. When the faulty action lies before the visible symptom, restarting at the symptom yields a useful branch in 49.2% of cases, against 68.6% for the repair our method proposes (Table 12). A branch is useful when at least 20% of its sampled completions succeed in execution.
  • The result does not depend on injected faults. When nominal plans are produced with untargeted retargeting noise and no fault is injected, 30 to 62 of 100 fail, and ROOTCAUSE recovers 61–78% of those failures (Table 14).

The same question exists for symbolic models. Even there, a violated precondition says where the plan breaks. Which earlier choices to keep is a separate question with its own literature: plan repair and reuse versus replanning (Nebel & Koehler, 1995; Fox et al., 2006) and dependency-directed backtracking (Stallman & Sussman, 1977). We ask it for learned video models, where there is no dependency structure to follow and every query is expensive. The revised related work makes this connection, and we thank you for prompting it.

The introduction has only a simple example, then the paper dives into details.

We agree. The introduction now reads in this order: the setting, the contrast with symbolic planning models, the motivation with the numbers above, the example followed from failure to repair, and only then the method. Its first paragraph now states:

Throughout, a world model is a learned, action-conditioned video predictor: given past camera frames and a sequence of robot commands, it generates future frames. It is not a symbolic planning model. It reports no preconditions and no cause of failure; a failure is something measured on predicted frames. It can be queried on any plan prefix, at a cost proportional to the prefix length.

We hope this clarifies the setting and the motivation, and we are glad to answer further questions. If it resolves your concern, we would be grateful if you would reconsider your assessment.

References added in the revision. Nebel & Koehler, "Plan reuse versus plan generation: a theoretical and empirical analysis", Artificial Intelligence, 1995. Fox, Gerevini, Long & Serina, "Plan stability: replanning versus plan repair", ICAPS, 2006. Stallman & Sussman, "Forward reasoning and dependency-directed backtracking in a system for computer-aided circuit analysis", Artificial Intelligence, 1977. Srivastava, Fang, Riano, Chitnis, Russell & Abbeel, "Combined task and motion planning through an extensible planner-independent interface layer", ICRA, 2014.

(note: I added this comment as a comment to the wrong response... deleted that and reposting it in what I hope is the right place.)

Thank you. It certainly would help the paper's readability, at least to an outsider, if the introduction was more elaborate on these points.

Thank you for the follow-up, and for reposting it in the right place. We agree, and the revised PDF, now uploaded, does this. The changes are marked in blue:

  • Sec. 1, first paragraph. It now states what "world model" means in this paper (a learned video predictor, not a symbolic planning model), how the two differ in what they report about a failure, and what a query costs.
  • Sec. 1, second and third paragraphs. The motivation is given with numbers, and the cube example is followed from the predicted failure to its repair.
  • Sec. 4, "Plan repair and reuse". A new paragraph relates the problem to plan repair, plan reuse and dependency-directed backtracking, with the four references from our response.

Your review listed two critical weaknesses, the assumption about what the model provides and the missing motivation in the introduction, and no concerns about soundness. If the revised introduction resolves these two points, we would be grateful if you would consider updating your rating. If anything on the first page is still unclear to a reader from outside this area, please tell us and we will revise it.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: revised PDF and summary of changes

We thank the three reviewers. A revised PDF is now uploaded. This note says how to read it and where each change is. Detailed answers are under each review.

How to read the revision.

  • Changes are marked in blue. The main text is 10 pages, the limit for this phase.
  • Every table, figure, equation and section of the submitted version keeps its number, so the references in our responses still hold. New tables are numbered R1 to R9.
  • No previously reported number has changed.
  • The revision contains no new experiment. The new Appendix C adds analyses computed from the reported results, and marks each computed or interpolated quantity as such.

What changed.

ChangeWhereRaised by
The setting comes first: the world model is a learned video predictor, not a symbolic planning model; motivation with numbersSec. 1, first three paragraphsK7gj, CuAv
Link to plan repair, plan reuse and dependency-directed backtracking; four references addedSec. 4, "Plan repair and reuse"K7gj
Five-step overview and a worked example before any equation; notation tableSec. 2 opening; Table R2CuAv
The method as a tree search; general and manipulation-specific modulesSec. 2 opening; Tables R3–R5CuAv
Eq. 1 selects a failure-conditioned repair site that is then tested, not a cause; the sentence in the introduction is correctedSec. 1; Sec. 2.1CuAv
"Matched budget" now reads "matched online budget"; the offline fit cost is in Table 3Abstract; Sec. 1; Sec. 3.1; Table 3PGRe
Comparison at equal total compute: the offline fit pays off only when one fitted model serves roughly 2,500 decisions or moreSec. 3.2; Tables R6–R7PGRe
Condensed ablations in the main textTable R1PGRe
Reported latency is compute, not parallel wall-clock time; time with parallel workersTable 26 caption; Sec. 3.3; Table R8CuAv
The two-fault cohort in context, with the failure breakdownSec. 3.3; Table R9PGRe
Requirements and scope: one world-model architecture, manipulation tasks, the fidelity audit as the entry testSec. 5; Table R5PGRe, CuAv

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.