- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks whether a planner can help a robot recover after a failure is detected. Nine configurations are tested on 350 shared RoboCasa365 scenes. The tested runtime interventions show no clear improvement. Reviewing plans before execution fixes some ordering errors. The authors argue that recovery is mainly limited by the executor.
Critical strengths
-
The repeated executor baseline is a useful control. It shows how much outcomes can change without any intervention. Testing on the same scenes also makes the comparisons easier to interpret.
-
The authors openly report unsuccessful experiments, implementation problems, and limitations. This deserves credit. The plan repair experiments also show why correcting a plan does not necessarily lead to successful execution.
Critical weaknesses
-
The results show that the tested interventions did not help, but the reason remains unclear. Could the authors compare different responses from the same saved stall state? These could include continuing execution, consulting the planner, and giving a trained instruction chosen by an expert. Repeating these trials would help distinguish poor recovery decisions from limited execution ability. This distinction seems necessary to support the main conclusion.
-
Toast, Stir, and Store are absent from fine-tuning. Across the baseline and five monitored configurations, they produce only 5/900 successes. This leaves little room to assess whether planning helps on these tasks. Could the authors report intervention effects separately on the four covered tasks? It would also help to know whether the conclusion holds with an executor trained on all seven tasks.
Questions for authors
See weakness.
2 comments
Response to W2 (2/2)
W2: We restrict the main study to the 200 scenes of the four fine-tuned tasks, which are Steam, Load, Stack, and Pack. The executor alone completes 71 of them, and no configuration improves on it. A scene counts as rescued when only the configuration succeeds, and as harmed when only the executor alone succeeds.
| Configuration | Rescued | Harmed | Sign test | Holm-corrected |
|---|---|---|---|---|
| Stall only (GT poses) | 37 | 41 | 0.734 | 1 |
| Any monitor (GT poses) | 22 | 39 | 0.040 | 0.159 |
| Clock (GT poses) | 36 | 42 | 0.572 | 1 |
| Clock, belief withheld | 40 | 41 | 1.000 | 1 |
| Any monitor (perception) | 32 | 54 | 0.023 | 0.115 |
No Holm-corrected is below 0.05. The two uncorrected -values below 0.05 both point toward harm. With five comparisons, each tested at a corrected level of 0.01, the minimum detectable paired effect is 13 to 16 points.
If you meant one result per task, Table 13 of the revised paper gives every configuration's successes on each task, on page 30, lines 1583–1594. Appendix Q describes it on pages 29–30, lines 1562–1580. Per task, the two runs of the executor alone differ by up to 6 out of 50 episodes. Among the five monitored configurations, only Any monitor (perception) gains more than that on any task, with 27 Stack episodes against 16. It also completes 1 Steam episode against 21 and 13 Load episodes against 27. Stall only (GT poses) and Clock (GT poses) complete 8 and 9 Stack episodes against 16. These per-task counts are descriptive.
We do not fine-tune our executor on all seven tasks. Our pretrained executor, the benchmark's own checkpoint, has all seven tasks in its training data, but even with its whole-task instruction it completes only 14 of 350 episodes. We therefore test a second executor that already succeeds on all seven. This is Xiaomi-Robotics-1 (XR-1), a more powerful 5B-parameter VLA model. Its public checkpoint is fine-tuned for RoboCasa365, and its authors evaluate it on all seven of our tasks. Our installation reproduces their StoreLeftoversInBowl result, with 38 of 50 successes against 42 of 50 reported.
With its native whole-task instruction, XR-1 completes 172 of our 350 scenes, including 86 of the 150 Toast, Stir and Store episodes. Our executor completes 1 of those 150. Our planner edits one instruction per subgoal, so the planner comparison drives XR-1 one subgoal at a time. In that mode, XR-1 completes 82 of 350 episodes, and 33 of the 150 on these three tasks.
That mode differs from the 172-episode run in two ways. XR-1 receives one instruction per subgoal. As in our main study, each episode also ends once every subgoal has been completed or has used up its own time budget. The 172-episode run instead continues each episode until the task succeeds or its time limit is reached. On Store, all three subgoals keep the whole-task instruction, so only the episode end differs. Store alone falls from 36 to 5 of 50 episodes, which is 31 of the 53-episode gap on these three tasks.
Over all 350 scenes in that mode, XR-1 completes 89 episodes with the planner consulted as in Stall only (GT poses). The planner rescues 49 scenes and harms 42, and the sign test gives . A second run of XR-1 alone completes 91 episodes.
On Toast, Stir and Store alone, XR-1 completes 36 of 150 episodes with the planner and 33 without. The planner rescues 24 of these scenes and harms 21, and the sign test gives . On the four tasks our executor is fine-tuned on, XR-1 completes 53 of 200 episodes with the planner and 49 without. There, the planner rescues 25 scenes and harms 21, again with no detectable change.
We also test the planner with XR-1 at its full strength, giving it its whole-task instruction and the same episode end as its 172-episode run. With the planner, XR-1 completes 162 of 350 episodes against 172 without it. The sign test gives , so again there is no detectable change. At this strength, XR-1 completes 75 of the 150 Toast, Stir and Store episodes with the planner and 86 without. On the four fine-tuned tasks, it completes 87 of 200 with the planner and 86 without. The conclusion therefore does not rest on the three tasks where our executor is at the floor.
Appendix Q of the revised paper reports the four-task analysis, on page 29, lines 1554–1560, with Table 12 on page 30. Appendix O reports the second executor, on pages 28–29, lines 1464–1528, with per-task results in Table 11 on page 28.
Response to W1 (1/2)
We thank the reviewer for the constructive review, and especially for proposing the same-state comparison. We report it in this rebuttal. This first reply answers W1. A second reply answers W2.
W1: We first run the executor alone on our 350 scenes and save the simulator state at the first stall of each episode. We check every saved state by restoring it. Object poses, the robot's pose, subgoal status, and rendered images all have to match the original. From 160 checked states, spread across the seven tasks, we compare five responses. The first lets the executor continue. The second consults the planner. The third consults the planner with the executor's 49 training instructions in view. The fourth issues the best training instruction for that state. The fifth runs the scripted retry. Each response is tried five times from each state, with a different executor seed each time.
The planner is consulted as in Stall only (GT poses). In the third response, it is also told to use the 49 instructions word for word. In the fourth response, every training instruction that names the stalled object or fixture is issued, and the one that works best from that state is kept. Because it is picked after seeing the outcomes, this response favors instruction choice. It stands in for an expert's choice. The scripted retry moves the arm back along its path to the segment's starting point and issues the segment's instruction again.
The outcome is resumption, meaning that the stalled subgoal is completed within a fresh segment budget. For each response, we average the resumption rate over states and compute a 95% interval by resampling states. We compare each response with continuing through paired Wilcoxon tests, Holm-corrected over the four comparisons.
Only 105 of the 160 states have a training instruction that names the stalled object or fixture, so the best training instruction is compared with continuing on those 105 states. Continuing also resumes 18.9% there.
Before running, we prescribe how the result is to be read. If the best training instruction stays close to continuing, the stall is limited by execution even with the best instruction. If it is clearly above continuing while the planner is not, decision quality matters. We observe the first pattern.
Two further results agree. In 141 of 160 states whose stall never resolves in the original run, the five responses resume between 3.7% and 10.2% of stalled segments. The highest rate comes from the best training instruction, on the 86 of these states that have one. Episodes started from the saved states succeed between 3.4% and 4.9% of the time under the other four responses.
Therefore, poor recovery decisions do not explain the null result. Even the best training instruction does not do better than letting the executor continue.
We repeat the test with the second executor described in our second reply, from 143 of its own saved stall states in sub-task mode. Continuing resumes 36.2% of stalled segments. Consulting the planner resumes 30.2%, and the scripted retry 32.2%. On the 61 states where a training-language wording validated for this executor exists, issuing that wording resumes 3.3 points less than continuing, with a 95% interval of [−9.2, +2.6] points. None of these differences is significant after Holm correction. The pre-registered reading is again that the stall is limited by execution.
Appendix N of the revised paper reports this test, on pages 26–28, lines 1377–1461, with Tables 9 and 10 on page 27.