acceptodds
Under review as a conference paper at ICLR 2027

Separating Detection from Repair in Long-Horizon Manipulation

Abstract

A common architecture for long-horizon manipulation pairs a high-level planner with a low-level executor. The planner is expected to notice execution failures and put the task back on course. Research has focused on when to consult the planner, whether at subtask boundaries, on a schedule, or after signs of failure. We test their shared assumption that a planner consulted at the right moment can repair what it finds. Using a monitoring layer built for this study, we compare nine combinations of failure monitoring, planner consultation, and physical retry on identical scenes in RoboCasa365. We evaluate five types of failure signals and find that stalls, or prolonged periods without progress, provide a selective trigger for planner consultation. With ground-truth poses supplied as a diagnostic control, the stall monitor triggers consultations in most episodes that ultimately fail and rarely in those that succeed. Yet detecting trouble does not translate to successful repair. Through systematic experiments and analyses, we find no run-time intervention improves task completion over the executor alone. A review before execution provides a revealing contrast. It corrects known ordering defects and raises completion on some of them, showing that the planner can repair a faulty plan. The persistent stalls we analyze expose a different problem. In half of them, the executor never grasps the object, even though an applicable instruction usually exists. Consulting the planner at a stall does not detectably increase the rate at which the stalled segment resumes, and retrying the same skill rarely succeeds. In the perceptual configuration, the monitors trigger planner consultation in nearly every episode, including successful ones. These comparisons locate the bottleneck in the hands, not the plan. In this setting, the findings point toward training recovery skills for failure states, rather than further changes to when or how the planner is consulted.

Primary area:
applications to robotics, autonomy, planning
Keywords:
long-horizon manipulation, failure detection, failure recovery

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GL7zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether a planner can help a robot recover after a failure is detected. Nine configurations are tested on 350 shared RoboCasa365 scenes. The tested runtime interventions show no clear improvement. Reviewing plans before execution fixes some ordering errors. The authors argue that recovery is mainly limited by the executor.

Critical strengths
  1. The repeated executor baseline is a useful control. It shows how much outcomes can change without any intervention. Testing on the same scenes also makes the comparisons easier to interpret.

  2. The authors openly report unsuccessful experiments, implementation problems, and limitations. This deserves credit. The plan repair experiments also show why correcting a plan does not necessarily lead to successful execution.

Critical weaknesses
  1. The results show that the tested interventions did not help, but the reason remains unclear. Could the authors compare different responses from the same saved stall state? These could include continuing execution, consulting the planner, and giving a trained instruction chosen by an expert. Repeating these trials would help distinguish poor recovery decisions from limited execution ability. This distinction seems necessary to support the main conclusion.

  2. Toast, Stir, and Store are absent from fine-tuning. Across the baseline and five monitored configurations, they produce only 5/900 successes. This leaves little room to assess whether planning helps on these tasks. Could the authors report intervention effects separately on the four covered tasks? It would also help to know whether the conclusion holds with an executor trained on all seven tasks.

Questions for authors

See weakness.

2 comments
AuthorsOpenReview ↗

Response to W1 (1/2)

We thank the reviewer for the constructive review, and especially for proposing the same-state comparison. We report it in this rebuttal. This first reply answers W1. A second reply answers W2.

W1: We first run the executor alone on our 350 scenes and save the simulator state at the first stall of each episode. We check every saved state by restoring it. Object poses, the robot's pose, subgoal status, and rendered images all have to match the original. From 160 checked states, spread across the seven tasks, we compare five responses. The first lets the executor continue. The second consults the planner. The third consults the planner with the executor's 49 training instructions in view. The fourth issues the best training instruction for that state. The fifth runs the scripted retry. Each response is tried five times from each state, with a different executor seed each time.

The planner is consulted as in Stall only (GT poses). In the third response, it is also told to use the 49 instructions word for word. In the fourth response, every training instruction that names the stalled object or fixture is issued, and the one that works best from that state is kept. Because it is picked after seeing the outcomes, this response favors instruction choice. It stands in for an expert's choice. The scripted retry moves the arm back along its path to the segment's starting point and issues the segment's instruction again.

The outcome is resumption, meaning that the stalled subgoal is completed within a fresh segment budget. For each response, we average the resumption rate over states and compute a 95% interval by resampling states. We compare each response with continuing through paired Wilcoxon tests, Holm-corrected over the four comparisons.

ResponseResumed95% intervalDifference from continuingHolm-corrected pp
Continue12.9%[8.5%, 17.6%]
Planner8.7%[5.5%, 12.4%]−4.1 points0.053
Planner with the 49 instructions12.9%[8.7%, 17.5%]0.0 points1
Best training instruction, on 105 states18.9%[12.6%, 25.5%]0.0 points against continuing on the same states1
Scripted retry10.2%[6.5%, 14.4%]−2.6 points0.33

Only 105 of the 160 states have a training instruction that names the stalled object or fixture, so the best training instruction is compared with continuing on those 105 states. Continuing also resumes 18.9% there.

Before running, we prescribe how the result is to be read. If the best training instruction stays close to continuing, the stall is limited by execution even with the best instruction. If it is clearly above continuing while the planner is not, decision quality matters. We observe the first pattern.

Two further results agree. In 141 of 160 states whose stall never resolves in the original run, the five responses resume between 3.7% and 10.2% of stalled segments. The highest rate comes from the best training instruction, on the 86 of these states that have one. Episodes started from the saved states succeed between 3.4% and 4.9% of the time under the other four responses.

Therefore, poor recovery decisions do not explain the null result. Even the best training instruction does not do better than letting the executor continue.

We repeat the test with the second executor described in our second reply, from 143 of its own saved stall states in sub-task mode. Continuing resumes 36.2% of stalled segments. Consulting the planner resumes 30.2%, and the scripted retry 32.2%. On the 61 states where a training-language wording validated for this executor exists, issuing that wording resumes 3.3 points less than continuing, with a 95% interval of [−9.2, +2.6] points. None of these differences is significant after Holm correction. The pre-registered reading is again that the stall is limited by execution.

Appendix N of the revised paper reports this test, on pages 26–28, lines 1377–1461, with Tables 9 and 10 on page 27.

AuthorsOpenReview ↗

Response to W2 (2/2)

W2: We restrict the main study to the 200 scenes of the four fine-tuned tasks, which are Steam, Load, Stack, and Pack. The executor alone completes 71 of them, and no configuration improves on it. A scene counts as rescued when only the configuration succeeds, and as harmed when only the executor alone succeeds.

ConfigurationRescuedHarmedSign test ppHolm-corrected pp
Stall only (GT poses)37410.7341
Any monitor (GT poses)22390.0400.159
Clock (GT poses)36420.5721
Clock, belief withheld40411.0001
Any monitor (perception)32540.0230.115

No Holm-corrected pp is below 0.05. The two uncorrected pp-values below 0.05 both point toward harm. With five comparisons, each tested at a corrected level of 0.01, the minimum detectable paired effect is 13 to 16 points.

If you meant one result per task, Table 13 of the revised paper gives every configuration's successes on each task, on page 30, lines 1583–1594. Appendix Q describes it on pages 29–30, lines 1562–1580. Per task, the two runs of the executor alone differ by up to 6 out of 50 episodes. Among the five monitored configurations, only Any monitor (perception) gains more than that on any task, with 27 Stack episodes against 16. It also completes 1 Steam episode against 21 and 13 Load episodes against 27. Stall only (GT poses) and Clock (GT poses) complete 8 and 9 Stack episodes against 16. These per-task counts are descriptive.

We do not fine-tune our executor on all seven tasks. Our pretrained executor, the benchmark's own π0.5\pi_{0.5} checkpoint, has all seven tasks in its training data, but even with its whole-task instruction it completes only 14 of 350 episodes. We therefore test a second executor that already succeeds on all seven. This is Xiaomi-Robotics-1 (XR-1), a more powerful 5B-parameter VLA model. Its public checkpoint is fine-tuned for RoboCasa365, and its authors evaluate it on all seven of our tasks. Our installation reproduces their StoreLeftoversInBowl result, with 38 of 50 successes against 42 of 50 reported.

With its native whole-task instruction, XR-1 completes 172 of our 350 scenes, including 86 of the 150 Toast, Stir and Store episodes. Our executor completes 1 of those 150. Our planner edits one instruction per subgoal, so the planner comparison drives XR-1 one subgoal at a time. In that mode, XR-1 completes 82 of 350 episodes, and 33 of the 150 on these three tasks.

That mode differs from the 172-episode run in two ways. XR-1 receives one instruction per subgoal. As in our main study, each episode also ends once every subgoal has been completed or has used up its own time budget. The 172-episode run instead continues each episode until the task succeeds or its time limit is reached. On Store, all three subgoals keep the whole-task instruction, so only the episode end differs. Store alone falls from 36 to 5 of 50 episodes, which is 31 of the 53-episode gap on these three tasks.

Over all 350 scenes in that mode, XR-1 completes 89 episodes with the planner consulted as in Stall only (GT poses). The planner rescues 49 scenes and harms 42, and the sign test gives p=0.530p = 0.530. A second run of XR-1 alone completes 91 episodes.

On Toast, Stir and Store alone, XR-1 completes 36 of 150 episodes with the planner and 33 without. The planner rescues 24 of these scenes and harms 21, and the sign test gives p=0.766p = 0.766. On the four tasks our executor is fine-tuned on, XR-1 completes 53 of 200 episodes with the planner and 49 without. There, the planner rescues 25 scenes and harms 21, again with no detectable change.

We also test the planner with XR-1 at its full strength, giving it its whole-task instruction and the same episode end as its 172-episode run. With the planner, XR-1 completes 162 of 350 episodes against 172 without it. The sign test gives p=0.353p = 0.353, so again there is no detectable change. At this strength, XR-1 completes 75 of the 150 Toast, Stir and Store episodes with the planner and 86 without. On the four fine-tuned tasks, it completes 87 of 200 with the planner and 86 without. The conclusion therefore does not rest on the three tasks where our executor is at the floor.

Appendix Q of the revised paper reports the four-task analysis, on page 29, lines 1554–1560, with Table 12 on page 30. Appendix O reports the second executor, on pages 28–29, lines 1464–1528, with per-task results in Table 11 on page 28.

Reviewer_CrkSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates failure detection and repair in long horizon manipulation. The authors keep the Planner and Executor fixed in RoboCasa365. They conduct controlled experiments across nine configurations involving monitoring, planner consultation, and physical retry.

Critical strengths
  1. The paper provides a meaningful diagnostic perspective. It explicitly separates Failure Detection from Repair. The authors use controlled experiments with a fixed Planner and Executor to examine the relationship between these two processes.
  2. The experimental design is systematic. It covers Stall Monitoring, Planner Consultation, Preexecution Review, Physical Retry, Perception Calibration, and Second Planner Ablation. These experiments provide a relatively comprehensive analysis.
Critical weaknesses
  1. The main contribution focuses on controlled empirical diagnosis. The paper systematically examines the distinction between Failure Detection and Repair. However, it introduces relatively limited innovations in recovery mechanisms.
  2. Scripted Physical Retry shows limited effectiveness for grasp acquisition stalls. However, this finding only considers a single retreat and repeat strategy. How does this result generalize to broader learned recovery behaviors?
  3. The Planner’s interventions are mainly limited to language-level plan edits, and it cannot provide skills that the Executor does not possess. Therefore, does this mechanism inherently limit Planner-based Repair when failures arise from insufficient low-level execution capabilities?
  4. The complete Detection and Repair intervention study relies on a single fixed Executor. Does the observed repair bottleneck change with different levels of Executor competence or additional recovery training?
Questions for authors

Please refer to the Critical Weaknesses.

5 comments
AuthorsOpenReview ↗

Response to W1 and W2 (1/2)

We thank the reviewer for the careful review. This first reply answers W1 and W2. A second reply answers W3 and W4.

W1: We agree that the paper proposes no new recovery mechanism. Its contribution is a controlled measurement of where repair fails, and the new experiments make this measurement more direct.

In a new run of the executor alone on our 350 scenes, we save the simulator state at the first stall of each episode. We then restart from 160 of these states, and from each state we compare five responses. The first lets the executor continue. The second consults the planner. The third consults the planner with the executor's 49 training instructions in view. The fourth issues the best training instruction for that state. The fifth runs the scripted retry. Each response is tried five times from each state, with a different executor seed each time.

None of these responses resumes more stalled segments than letting the executor continue. Our second reply gives the numbers under W3. We also test a different consultation design. In a REFLECT-style consultation, the planner first explains the failure and then chooses a correction. On all 350 scenes, it completes 71 episodes, against 72 for the executor alone. Diagnosing the failure before choosing a correction does not help either. Appendix P of the revised paper reports this consultation, on page 29, lines 1543–1549.

As Section 2 of the paper reviews, most work on planner-in-the-loop repair changes when and how the planner is consulted. Our results show that this lever has no headroom here, because the failures lie in the executor's skills. New recovery mechanisms should therefore target those skills, for example through recovery skills trained on failure states.

W2: We retest the scripted retry from identical states. A stalled segment counts as resumed when its subgoal is completed within a fresh segment budget. From the same 160 saved stall states, the scripted retry resumes 10.2% of stalled segments. Letting the executor continue resumes 12.9%. The difference is −2.6 points, with a 95% interval of [−6.0, +0.7] points, so the retry is not detectably different from continuing.

The comparison is similar for both causes of stall. In the 97 states stalled in grasp acquisition, retry resumes 5.6% and continuing 8.9%. In the other 63 states, the rates are 17.5% and 19.0%, respectively.

We also combine the retry with a better instruction. After the retreat, we issue the training instruction that resumes the segment most often from that state in our same-state test. We run this on the 105 states that have such an instruction, with fresh trials. Pooled over 513 trials, it resumes 13.6% of stalled segments. In the same states, continuing resumes 18.5% and the plain retry 15.6%. Only in the 12 contact-stall states does it beat continuing, with 43.3% against 38.3%. The plain retry does better still there, at 56.7%. This agrees with the paper's finding that the retry helps mainly on contact stalls. Appendix N of the revised paper reports these results in Table 9 and on page 27, lines 1446–1454.

We do not train a learned recovery policy, and we do not claim that the retry result generalizes to one. Our results do reveal what such a policy needs. It needs demonstrations collected from failure states like these, such as a re-grasp after a missed grasp or a re-approach after a contact stall. The instructions for these demonstrations need to be in the executor's training language. The saved stall states also provide a direct test, because a learned recovery skill can be scored as one more response from the same states.

AuthorsOpenReview ↗

Response to W3 and W4 (2/2)

W3: Yes. A planner that edits language can only choose among skills the executor already has. It cannot repair what the executor cannot execute. The open question is how many natural failures are of this kind, and we measure this directly.

Of the 160 saved stall states, 105 have at least one training instruction that names the stalled object or fixture. From each of these states, we issue every such instruction five times and keep the one that works best. This is an optimistic stand-in for an expert's choice. The best instruction resumes 18.9% of stalled segments, with a 95% interval of [12.6%, 25.5%]. Letting the executor continue from the same states also resumes 18.9%. The difference is 0.0 points, with a 95% interval of [−3.8, +3.6] points. In 77 of the 105 states, no candidate instruction resumes the segment in any of its trials. On the 141 of 160 states whose stall never resolves in the original run, the five responses resume between 3.7% and 10.2% of stalled segments. The highest rate comes from the best training instruction, on the 86 of these states that have one. The same holds for the second executor described under W4. From 61 of its saved stall states, the wording validated for it resumes 3.3 points less than continuing, with a 95% interval of [−9.2, +2.6] points.

Choosing a better instruction is the planner's only lever, and here it has no headroom. As the paper’s Table 2 shows, the same planner does repair plan-level defects through pre-execution review. We therefore regard these stalls as limited by execution. Appendix N of the revised paper reports this test, on pages 26–27, lines 1400–1429.

W4: We add a second, independently trained executor, Xiaomi-Robotics-1 (XR-1). It is a 5B-parameter VLA model, and we use its public RoboCasa365 checkpoint. Our installation reproduces its published StoreLeftoversInBowl result, with 38 of 50 successes against 42 of 50 reported.

With its native whole-task instruction, XR-1 completes 172 of our 350 scenes. Our executor completes 72. Our planner edits one instruction per subgoal, so we also drive XR-1 one subgoal at a time, with wordings validated on held-out start states. In this mode, XR-1 alone completes 82 of 350 episodes. With the planner consulted as in Stall only (GT poses), it completes 89.

This mode differs from the 172-episode run in two ways. XR-1 receives one instruction per subgoal. As in our main study, each episode also ends once every subgoal has been completed or has used up its own time budget. The 172-episode run instead continues each episode until the task succeeds or its time limit is reached. On Store, all three subgoals keep the whole-task instruction, so only the episode end differs. Store alone falls from 36 to 5 of 50 episodes, which is 31 of the 90-episode gap.

A scene counts as rescued when XR-1 alone fails and the planner configuration succeeds, and as harmed in the reverse case. The planner rescues 49 scenes and harms 42, and the sign test gives p=0.530p = 0.530. A second run of XR-1 alone completes 91 episodes. The planner therefore does not detectably help this executor either. From XR-1's own saved stall states, consulting the planner resumes 30.2% of stalled segments, against 36.2% for continuing, a difference that is not significant after Holm correction.

The sub-task mode keeps XR-1 near our executor's level, so we also test the planner at XR-1's full strength. We give XR-1 its whole-task instruction and the same episode end as the 172-episode run. With the planner consulted as in Stall only (GT poses), it completes 162 of 350 episodes, with 42 rescued and 52 harmed scenes against the 172. The sign test gives p=0.353p = 0.353. A second run of XR-1 alone completes 158, with 44 rescued and 58 harmed scenes against the first. The planner's effect is therefore within run-to-run variation. The test has 80% power to detect a change of 7.7 points. Planner consultation thus does not detectably raise completion at any of the competence levels we test, from 72 to 172 of 350 episodes.

At the low end of competence, we find that Table 1's pretrained and single-skill executors receive the composite chain's instructions. These instructions lie outside the pretrained checkpoint's training language for 15 of the 23 subgoals. They lie outside the single-skill executor's training language for 21 of them. We rerun both executors with instructions from their own training data. The single-skill executor still completes none of the 350 episodes. The pretrained executor completes 14 with its whole-task instruction and 3 with its best sub-task wordings. Both stay far below the 72 of our executor, so the ordering in Table 1 holds. Appendix O of the revised paper reports the second executor, on pages 28–29, lines 1464–1528. Appendix B reports the reruns on pages 15–16, lines 806–813.

Thank you to the authors for conducting additional experiments with different executors. However, the differences in execution protocols appear to have a considerable impact on the final performance, and the current results still make it difficult to clearly distinguish the effects of the planner's capabilities, the executor's capabilities, and the interface design. Therefore, I will keep my original score for now.

Reviewer_jjh9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether a high-level planner can escalate the request at the right moment can actually repair failures in long-horizon manipulation, and it measures this separately from whether failures can be detected. The interesting part is that between the executor (pi0.5) and planner (claude opus 5.5) there is a monitoring layer with an explicit object belief and five switches - belief source, consultation trigger, planner context, review timing, and stall response. After that, the method using 9 configurations run on RoboCasa365.

Critical strengths
  1. The question matters and is well posed: replanning papers usually report end-to-end success, which mixes noticing a failure with fixing it, and this paper measures the two separately.
  2. Experiments are carefully designed.
  3. The decomposition is informative: injected plan defects show that the planner can fix errors visible in the plan, while the post-stall and retry analyses point to grasp acquisition as the main natural failure.
Critical weaknesses
  1. All Planner comparisons use one executor on seven simulated tasks, three of which almost never succeed, and no published replanning system such as REFLECT [1] is compared, so it is unclear how far the conclusions generalize.
  2. Rewriting the active instruction is the planner's most common edit, but only 2.3% of the rewritten instructions are among the 49 instructions the executor was trained on. So I think the null result may simply mean that the executor cannot follow the planner's edits. That would be an issue with how the planner is consulted, which the paper suggests is not worth changing further. The Sonnet 5 run has more in-vocabulary rewrites (17.3%), but this is still low and was only tested on 105 scenes.

[1] Liu, Z., Bahety, A., & Song, S. (2023). REFLECT: Summarizing robot experiences for failure explanation and correction (arXiv:2306.15724). arXiv. https://doi.org/10.48550/arXiv.2306.15724

Questions for authors
  1. Do rewrites that match the executor's trained instructions resume stalled segments more often than the ones that don't? Is the planner ever shown the list of 49 trained instructions? As far as I can tell, it is not part of the consultation input.
  2. How was it decided that a trained instruction applies in the persistent stalls?
  3. Could the planner's edits sometimes hurt? Any monitor (GT poses) calls the planner the most and has the lowest success among the GT configurations.
3 comments
AuthorsOpenReview ↗

Response to W1 (1/3)

We thank the reviewer for the constructive review. This first reply answers W1. The second reply addresses W2/Q1, and the third addresses Q2 and Q3.

W1: Our conclusion is that completion is limited by the executor's competence, not by when or how the planner is consulted. If this is right, a stronger executor should raise completion where planner consultation does not. The planner should also fail to repair a second executor's failures. We test both predictions with a second, independently trained executor.

We choose a more powerful second executor, Xiaomi-Robotics-1 (XR-1), which is a 5B-parameter VLA model. It demonstrates a much stronger performance on the RoboCasa365 benchmark, with an overall success rate of 57.4% versus 16.9% for π0.5\pi_{0.5}. We use its public RoboCasa365 checkpoint. To verify our installation, we rerun its published StoreLeftoversInBowl evaluation under its official protocol. It succeeds in 38 of 50 episodes, compared with 42 of 50 in their report.

Following XR-1's own assessment, the entire task is described with a single instruction. Run in this way on our 350 scenes, XR-1 completes 172 episodes. Our π0.5\pi_{0.5} executor completes 72. On the 150 episodes of Toast, Stir, and Store, XR-1 completes 86, while our π0.5\pi_{0.5} executor only completes 1. These three tasks are therefore no longer at the floor.

According to our design, the planner edits a chain with one instruction per subgoal. To pair XR-1 with the planner, we give it one subgoal at a time. For each of the 23 subgoals, we choose the wording that works best for XR-1 on held-out start states, following a rule fixed before the test. The candidates are RoboCasa's single-task instruction, RoboCasa's subtask label, and the whole-task instruction. Run in this way, XR-1 alone completes 82 of 350 episodes. With the planner consulted as in Stall only (GT poses), it completes 89.

This mode differs from the 172-episode run in two ways. XR-1 receives one instruction per subgoal, chosen as above. As in our main study, each episode also ends once every subgoal has been completed or has used up its own time budget. In the 172-episode run, the episode instead continues until the entire composite task succeeds or its time limit is reached. On Store, all three subgoals keep the whole-task instruction, so only the episode end differs. Completion on Store still falls from 36 to 5 of 50 episodes, so Store alone accounts for 31 of the 90-episode gap.

A scene counts as rescued when XR-1 alone fails and the planner configuration succeeds, and as harmed in the reverse case. The planner rescues 49 scenes and harms 42, and the sign test gives p=0.530p = 0.530. A second run of XR-1 alone completes 91 episodes, with 51 rescued and 42 harmed scenes against the first. The planner therefore changes outcomes no more than rerunning the executor does. The null result holds with a second executor. It also holds on Toast, Stir and Store alone, where XR-1 completes 33 of 150 episodes without the planner and 36 with it.

In this sub-task mode, XR-1 is not detectably stronger than our executor, with 82 completed episodes against 72. We therefore also test the planner with XR-1 at its full strength, giving it its whole-task instruction and the same episode end as the 172-episode run. With the planner, XR-1 completes 162 episodes, with 42 rescued and 52 harmed scenes against the 172. The sign test gives p=0.353p = 0.353. A second run of XR-1 alone completes 158, with 44 rescued and 58 harmed scenes. The planner's effect is therefore within run-to-run variation. The test has 80% power to detect a change of 7.7 points. An earlier version of this test uses a different runner, which holds XR-1 alone to 77 of 350 episodes. There, XR-1 completes 91 episodes with the planner, again with no detectable change.

For REFLECT, we add a REFLECT-style consultation to our interface. At a stall, the planner first explains the failure from keyframes and the event log, and then chooses a correction. On all 350 scenes, this configuration completes 71 episodes. The executor alone completes 72. Against it, the REFLECT-style consultation rescues 37 scenes and harms 38, so the two do not differ in a detectable way. Stall only (GT poses), which uses the same trigger without the explanation step, completes 69. Explaining the failure before choosing a correction therefore does not raise completion. The null result is not specific to how our planner reasons at a stall. This is what our conclusion predicts, because any correction still has to be carried out by the same executor.

Appendix O of the revised paper reports the second executor, on pages 28–29, lines 1464–1528. Appendix P reports the REFLECT-style consultation, on page 29, lines 1543–1549.

AuthorsOpenReview ↗

Response to W2/Q1 (2/3)

W2/Q1: Matched edits resume stalled segments slightly more often than unmatched edits. They do not resume them more often than simply letting the executor continue.

We first correct Appendix K. Its last paragraph compares the edits of two planners on 105 scenes. It reports that 2.3% of the primary planner's edits fall within the executor's training vocabulary, against 17.3% for the second planner. Both percentages contain two errors. First, they count two kinds of edits together. A rewrite replaces the instruction of the active step, while an insert adds a new step with its own instruction. Second, they are checked against the training dataset's full task table, which has 219 entries. The executor's fine-tuning episodes use only 49 distinct instructions. Checked against these 49, only 2 of the primary planner's 131 edits and 4 of the second planner's 127 are training instructions, and all six are inserts. Over the six main-study configurations that consult the planner and this 105-scene comparison, the planners apply 2,042 rewrites and 1,486 inserts. None of the rewrites and only 39 of the inserts are training instructions. The planner's edits are therefore rarely in the executor's training language, even less often than Appendix K reports. The revised Appendix K gives these counts on page 25, lines 1332–1338.

The planner is never shown the 49 instructions. Its prompt offers insert templates taken from the 13 single-skill tasks instead. Only three of these templates can produce one of the 49 instructions, which is why every match is an insert. The mismatch is thus built into the planner's prompt, so the reviewer's hypothesis could explain the null result.

We then test the reviewer's hypothesis directly in two ways. The first test starts from saved stall states. In a new run of the executor alone on our 350 scenes, we save the simulator state at the first stall of each episode. We keep the states that restore exactly and draw a sample of 160 across the seven tasks, a size we fix in advance. From each sampled state, we compare five responses. The first lets the executor continue. The second consults the planner. The third consults the planner with the executor's 49 training instructions in view. The fourth issues the best training instruction for that state. The fifth runs the scripted retry. Each response is tried five times from each state, with a different executor seed each time. A stalled segment counts as resumed when its subgoal is completed within a fresh segment budget.

When the planner is shown the 49 instructions and told to use them word for word, it resumes 12.9% of stalled segments. Without the instructions, it resumes 8.7%. Showing the training vocabulary therefore raises resumption by 4.1 points, a significant difference with p=0.004p = 0.004. Letting the executor continue without consulting the planner also resumes 12.9%. The planner with the vocabulary therefore does no better than continuing, with a difference of 0.0 points and a 95% interval of [−1.8, +1.8] points. The scripted retry resumes 10.2%, which is also no better than continuing.

For the fourth response, we issue every training instruction that names the stalled object or fixture five times, and keep the best one for each state. On the 105 states that have such an instruction, the best one resumes 18.9% of stalled segments. Continuing resumes the same 18.9% from these states.

The second test runs a vocabulary-aware planner on all 350 scenes, in the Stall only (GT poses) configuration. The planner complies. Of its 62 applied rewrites, 53 are exact training instructions. In the main study, however, none of the 264 rewrites is. Completion does not rise. The configuration completes 61 of 350 episodes, against 69 for the original planner and 72 for the executor alone. Neither difference is significant. Edited segments also rarely complete. In-vocabulary edits lead to a completed segment in 3 of 53 cases, and other edits in 1 of 36 cases.

Finally, we extend the Sonnet 5 run from 105 to all 350 scenes. Sonnet 5 in Stall only (GT poses) completes 66 episodes. The executor alone completes 72. Against it, Sonnet 5 rescues 33 scenes and harms 39, which is not a detectable difference. Opus 5 completes 69. None of the 158 rewrites by Sonnet 5 is a training instruction.

Edits phrased in the training vocabulary raise the planner's resumption at the stall to the level of continuing. They do not make stalled segments resumable, and they do not detectably raise completion.

The revised paper reports the same-state test in Appendix N, on pages 26–28, lines 1377–1461. The vocabulary-aware planner is in Appendix P, on page 29, lines 1532–1541. The Sonnet 5 run on all 350 scenes is in Appendix K, on page 25, lines 1326–1330.

AuthorsOpenReview ↗

Response to Q2 and Q3 (3/3)

Q2: A fixed rule decides whether a trained instruction applies. The rule assigns each stall a class from its state, and a fixed table marks some classes as having an applicable instruction. It never compares the stall with the 49 instruction strings. Rerunning the rule on the logs reproduces the paper's 141 of 193, so the count itself is correct. The paper's wording is the problem, because it says an instruction applies when the rule only checks the stall's class.

We therefore check the 193 persistent stalls against the 49 strings themselves. In 123 of them, some training instructions name the stalled object or fixture. In 84 of them, some training instructions name both the object and its destination. These 84 consist of all 68 persistent stalls on the four fine-tuned tasks and 16 of the 38 on Store. No persistent stall on Stir or Toast meets this stricter test. Section 5.2 of the revised paper now gives these two counts and their definitions, on page 7, lines 353–356. The introduction gives the first count as 64%, on page 2, lines 89–90.

The same-state test in our reply answers the underlying question directly. Where a matching instruction exists, issuing the best one resumes no more stalls than letting the executor continue.

Q3: Possibly, but we cannot detect it. Four results bear on this. In the first three, consulting the planner resumes fewer stalled segments than continuing, but no tested difference is significant after Holm correction. Over whole episodes, the loss is also small and not significant. Section 5.2 of the revised paper now states this, on page 8, lines 404–408.

First, we compare responses from the same saved stall states, as described in our second reply. Consulting the planner resumes 8.7% of stalled segments. Letting the executor continue resumes 12.9%. The difference is −4.1 points, with a 95% interval of [−7.1, −1.5] points. After Holm correction over four comparisons, p=0.053p = 0.053.

Second, the same holds for the second executor, XR-1. We repeat the same-state comparison from 143 of XR-1's own saved stall states. Consulting the planner resumes 30.2% of stalled segments, against 36.2% for continuing. The difference is −6.2 points, with a 95% interval of [−12.1, −0.6] points, and the Holm-corrected p=0.065p = 0.065.

Third, with XR-1 at its full strength, the stalled segment resumes after the planner's first consultation in 64 of 152 cases. Without the planner, it resumes in 114 of 174 stalls. These are different sets of stalls, as explained in Section 5.2, so the comparison is indicative. Over all 350 scenes, XR-1 completes 162 episodes with the planner and 172 without it, which is not a detectable difference.

Fourth, in the main study, Any monitor (GT poses) completes 55 episodes, 17 fewer than the 72 of the executor alone. The sign test gives p=0.043p = 0.043 before correction, and the difference is not significant after Holm correction. This configuration harms 40 scenes, about as many as the 41 harmed by a plain rerun of the executor. It rescues 23, against 36 for the rerun. Its lower completion therefore comes from fewer rescues rather than from more harms.

We then look for a specific harmful edit in the 40 harmed scenes. In 27 of them, no consultation happens at or after the point where the episode last makes progress. In the other 13, we check two candidate mechanisms. One is flushing the action queue while an object is being carried, which occurs in 4 of the 13. The other is a later regression of a completed subgoal, which occurs in 1 of the 13. Neither is more frequent here than across all 1,554 consultations, where they occur 455 and 74 times, respectively. We find no single harmful edit.

Appendix N of the revised paper reports the first two results, on pages 26–28, lines 1377–1461. Appendix O reports the third, on pages 28–29, lines 1501–1513. Appendix Q reports the fourth and the search for a harmful edit, on page 30, lines 1596–1603.

Comments on the paper

2 comments
Area_Chair_daPtOpenReview ↗

Reviewers-Authors Discussion Reminder

Dear Reviewers,

The review process for the paper has reached the rebuttal stage, and the authors have submitted their responses. Reviewers are encouraged to engage actively with these responses. Please clarify whether your initial concerns have been addressed and indicate if any new questions have arisen. Your meaningful participation is essential to maintaining a healthy ecosystem for ICLR.

Your AC.

Dear Reviewers and Area Chair,

We thank the reviewers for their careful reading and for the discussion. As the rebuttal phase is approaching its end, we summarize this rebuttal below. Our replies to each reviewer and the appendices cited below give the full numbers.

Where the paper stands. The paper measures whether a frozen VLM planner consulted during execution repairs the failures that the monitors detect. Our new experiments leave its main result unchanged. Consulting the planner during execution does not detectably raise task completion, with the evaluated executor in the paper or with a second executor, under any protocol we test. On constructed plans with known ordering defects, pre-execution review still corrects most of them. The rebuttal narrows one conclusion. For our executor, stalls are limited by execution even with the best instruction. When the second executor runs at full strength, with its own whole-task instruction, a better wording resumes more stalls than continuing, while consulting the planner shows no detectable gain. Even that wording does not detectably raise episode success. Section 5.2 now states the execution-limited conclusion for our executor, on page 8, lines 407–408.

The questions the reviewers share.

  1. Does repair fail because the planner decides poorly or because the executor cannot act? GL7z (W1), CrkS (W3) and jjh9 (W2) raise this. We restart from 160 saved stall states of our executor and compare responses from identical states. On the 105 states where a training instruction names the stalled object or fixture, even the best one, chosen after seeing the outcomes, resumes no more stalls than letting the executor continue. Both resume 18.9%. The same holds for the second executor driven one subgoal at a time. At that executor's full strength, a validated wording, where one exists, resumes stalls 10.4 points more often than continuing, while consulting the planner shows no detectable gain. Appendix N reports the first test, on pages 26–28. Appendix R reports the full-strength test, on pages 31–32.
  2. Does the result hold for a second executor and for the tasks absent from fine-tuning? All reviewers raise this, and CrkS further asks how the execution protocol affects it. We add Xiaomi-Robotics-1 (XR-1), which completes up to 172 of 350 episodes, against 72 for our executor. On the three tasks absent from our executor's fine-tuning, XR-1 completes up to 86 of 150 episodes, whereas ours completes only 1. The protocol does change absolute completion. We therefore compare the planner with the executor alone under the same protocol, and a second run of the executor alone measures run-to-run variation. In none of the four settings of executor and protocol does the planner detectably raise completion. Appendix O reports these runs, on pages 28–29. On the four fine-tuned tasks alone, no configuration improves on the executor alone, as Appendix Q reports on pages 29–30.
  3. Would a different planner or consultation do better? jjh9 (W1, W2, Q1) and CrkS (W1) raise this. Prompted to use the executor's 49 training instructions verbatim, the planner does so in 85.5% of its rewrites. It still completes 61 episodes, against 72 for the executor alone. A second planner, Claude Sonnet 5, completes 66. A REFLECT-style consultation, which explains the failure before correcting it, completes 71. Appendix P reports the vocabulary and REFLECT-style runs, on page 29. Appendix K reports the Sonnet 5 run, on page 25.

Other changes to the paper.

  • Table 1's pretrained and single-skill executors now also run with their own instructions. They complete 14 and 0 episodes, far below the 72 of our executor. Appendix B reports the reruns on pages 15–16, and the Table 1 caption on page 5, lines 219–220, now says so.
  • Appendix K now checks the planner's rewrites against the 49 instructions used in fine-tuning. None of the 2,042 rewrites in any configuration is a training instruction. The corrected text is on page 25, lines 1332–1338.
  • Section 5.2's "141 (73%)" becomes 123 of 193 persistent stalls in which a training instruction names the stalled object or fixture. The corrected text is on page 7, lines 353–356, and in the introduction on page 2, lines 89–90.

We welcome any follow-up questions and will gladly answer them in the remaining discussion period.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.