- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes using reusable skills to teach a policy failure-recovery behaviors. During policy rollouts, a failure detector identifies potential failures, and a VLM selects corrective skills based on the scene, task description, and available skills. Successful repair segments are then added to the training dataset to improve the policy's robustness to failures.
Critical strengths
None
Critical weaknesses
1. Unclear justification for using skills only for recovery.
While I agree that sharing skills across tasks can improve robustness, the paper does not clearly justify restricting skill execution to recovery. Would using skills throughout the entire task, or invoking them more frequently, yield better performance? An ablation on intervention frequency would help clarify this design choice.
2. Concerns about how the task policy improves over time.
- Frozen failure detector: My understanding is that the failure detector is trained on initial episodes and then frozen. As the task policy evolves, how can the detector identify failures on updated policy? If it continues to trigger at similar failure states, what benefit does updating the task policy provide?
- Benefit of policy updates: The newly collected recovery trajectories come from skills already available in the library. Why should training the task policy on these trajectories improve performance beyond directly invoking those skills? The paper should clarify what the task policy learns that the existing skill library cannot already provide.
Questions for authors
Suggestions for Clarity
Throughout the paper, I found several concepts and claims that could benefit from clearer definitions or explanations. While I could often infer the intended meaning from the context, explicitly clarifying these points would make the paper easier to follow. Please disregard if some of my suggestions are trivial.
- L154: Explicitly define the task policy (), its dataset (), and each skill policy () and their corresponding dataset. Clarify how these policies and datasets are constructed. This notation would help clarify that the framework maintains two separate sets of policies: a task policy and a library of skill policies. Initially, I found this distinction unclear and assumed the task itself was executed by composing skills.
- L155 "For simplicity": Explain why is it simplified.
- L161: Hard to grasp what "our own sharing experiments" refer to. Elaborate what it means and maybe refer to the corresponding section.
- L229: Clarify why using a fixed prompt makes it scalable.
- L252: Clarify what is excluded by "the policy's actions and rejected trials." (both task and retrial behaviors are "policy's action")
6 comments
Response to Reviewer 1NBY (2 / 3)
W2a: “As the task policy evolves, how can the detector identify failures on updated policy? If it continues to trigger at similar failure states, what benefit does updating the task policy provide?”
The frozen value model continues to respond to new behavior because:
-
The value model is action-conditioned (L206, Eq. 1), and is able to change its prediction when the policy changes its proposed action.
-
The value model was trained on common failures at every stage before the loop began (General Response, item 2).
A trigger at a familiar state means the updated policy still fails there, which is the intended and desirable behavior. If the subsequent invoked repair succeeds at that failed state, it becomes new supervision (at that state) for the next policy. Over consecutive iterations, these failure modes in the learned policy are gently “ironed out” via this trigger-and-train mechanism. Note that the value model is not used during our reported evaluation (L314–315); the policy executing independently is able to improve on every task: 14/30 to 25/30 on Stack-3, 0/20 to 12/20 on Drawer, 13.8 to 71.3 on Coffee and 18.1 to 83.8 on Sweeping.
W2b: “Why should training a policy on collected recoveries improve performance beyond directly invoking skills?”
Successful skill corrections teach the learned policy new actions in failure states, while skill failures are ignored, thereby preserving good policy behavior despite potential failures. We would like to re-emphasize how the policy imitates the skills. It keeps only accepted repairs from completed rollouts (L248–252), so the 23–30% of skill executions that fail never enter training (Table 2), and at stages it already performs successfully, it keeps its own behavior (L238–240). Thus the task policy learns the skills' successful behavior at its own failure states, without the skills' failures and without hand-offs. The result outperforms both i) invoking skills under human guidance (i.e. the CoP baseline), as well as ii) invoking them with a VLM. The former is Table 1 in the main paper, and the latter is included in our general response item 1. We reproduce both tables here for ease of reference:
| Task | Initial policy | CoP | Trained policy alone |
|---|---|---|---|
| Stack-3 | 14/30 | 23/30 | 25/30 |
| Drawer | 0/20 | 8/20 | 12/20 |
| Coffee | 13.8 | 38.8 | 71.3 |
| Sweeping | 18.1 | 55.0 | 83.8 |
| Test-time system | Drawer | Stack-3 | VLM calls per rollout | Latency per call |
|---|---|---|---|---|
| alone | 0/20 | 14/30 | 0 | – |
| + value model + skills + Gemini 2.5 Pro | 5/20 | 16/30 | 4.2 | 9.61s |
| + value model + skills + Astra | 8/20 | 19/30 | 3.8 | 4.52s |
| alone (ours) | 12/20 | 25/30 | 0 | – |
Note also that the policy’s learnable dataset grows each round while the skills stay fixed (Eq. 2). A natural next step is to add a validated task policy to the library as a longer skill. A reliable drawer-opening-and-placement policy, for example, could supply a correction within a more complex organization task. That higher-level learner would collect successful executions of the new skill at its own failure states and incorporate them through the same update procedure. Repeating this process could turn improvements within individual tasks into a growing library of reusable capabilities. The present experiments establish the first step (learning a stronger task policy from a fixed library) and the sharing results motivate testing this recursive extension, consistent with the library-expansion direction in Section 5, lines 457–462.
Response to Reviewer 1NBY (3 / 3)
Suggestions for clarity.
We thank the reviewer for their detailed read. We have incorporated each suggestion in the revision:
- L154. Let be the task policy, trained on complete-task demonstrations plus the accepted repairs (Eq. 2). It executes every rollout. Each skill is trained on separately collected stage demonstrations , acts only from a predicted failure until its stage is repaired, and then hands control back to . Both use a 0.5-class architecture (L283–285). The skill library stays fixed, and evaluation disables the value model, judge and skills (L314–315).
- L155 “For simplicity.” Each stage has one designated skill, so choosing a skill reduces to identifying the stage that needs repair (L227–229). This keeps the paper's question, whether skill corrections can teach a task policy, separate from choosing among several skills for one stage.
- L155 “For simplicity.” Each stage has one designated skill, so choosing a skill reduces to identifying the stage that needs repair (L227–229). This keeps the paper's question, whether skill corrections can teach a task policy, separate from choosing among several skills for one stage.
- L161. This refers to the sharing experiments in Sec. 4.3 (Pick Sponge and Sponge Wiping, Tables 7–8), and the revised text points there.
- L229. One prompt template serves all five tasks, including Sponge Wiping. Its only task-specific inputs are the task instruction and the skill names (L819–821), so adding a task means supplying these two inputs with no change to the prompt. The revision replaces "for scalability" with "to reuse the procedure across tasks" and aligns L145 and L274 with this input list.
- L252. "The policy's actions" are all actions executed by , including the stages it completes unassisted. "Rejected trials" are skill executions that fail the judge's repair check, including failed retries. Neither enters training. Only the accepted skill execution is kept, from its start to hand-back, and only if the judge confirms that the rollout completed the task (L250–252, App. B.3–B.4).
Response to Reviewer 1NBY (1 / 3)
Thank you very much for your review.
W1: “Justification for using skills only for recovery.” Would executing them throughout the task or invoking them more frequently help?
Our goal is not to create a system that can arbitrarily complete tasks, but rather to develop a framework for autonomously improving policies themselves. Targeted intervention concentrates improvement on the learned policy’s remaining difficulties. Letting the task policy continue outside of interventions preserves its useful execution, and exposes the next error along its own trajectories. In contrast, running skills throughout the task instead only collects states visited by the skill chain.
Using the skills throughout the task at test time is also exactly our CoP baseline, in which a person sequences the skills. The skill-space shooting policy outperforms this baseline by 32.5 and 28.8 progress points on Coffee and Sweeping and scores above it on Stack-3 and Drawer (Table 1). For intervention frequency, we swept the trigger threshold τ on Drawer, each arm running three rounds from the same π0task policy (0/20) with the same collection time:
The threshold trades how fast the policy improves against how good the collected corrections are, and the midpoint τ = 0 that we use on every task gets both. Lowering τ to −0.2 triggers less often (0.6 vs. 0.8 triggers per rollout). The policy then improves more slowly (10/20 vs. 12/20 after two updates) but reaches the same final success of 12/20. Raising τ to +0.4 makes the value model fire about twice as often and doubles the judge calls, yet final success drops to 10/20. We attribute this to the extra triggers firing where the task policy was already doing well, so the loop admits repair segments that replace adequate behavior instead of fixing failures. Running the skills throughout the task (τ = +0.5) is the extreme case: it reaches only 9/20, slightly below the HUMAN baseline (10/20), because a full skill chain is a sub-optimal source of new data. The corrections are therefore most useful when they target predicted failures. A trigger that is too insensitive slows improvement, and one that is too sensitive dilutes the data with segments that were never needed. Final success stays between 10/20 and 12/20 across thresholds from −0.2 to +0.4, so one untuned threshold is enough, and invoking skills more often adds skill time and VLM calls without improving the policy.
In particular, employing a local value model in this manner reserves expensive VLM calls for junctures requiring deeper semantic judgment. For example, but our Drawer and Stack-3 tasks in general response, item 1, where we invoke a VLM harness on top of our skills library throughout both tasks, we observed the following median end-to-end latencies:
The faster value model comparator reduces this overhead, allowing the VLM to be invoked only when necessary (e.g. deciding which correction skill to invoke). This latency issue further emphasizes the value of actually learning the corrections themselves to remove the need to make those calls during unaided deployment. Finally, note that the goal of sharing skills across tasks is NOT to improve robustness. The reviewer is correct in that the skills must be robust to facilitate sharing (as we demonstrate in our response to reviewer EKRP’s W2), but the goal of sharing skills across tasks is to facilitate scalable on-policy collection of corrective data. Moreover, under skill-space shooting, the teaching cost for internalizing any skill within the library is upfront paid once and then amortized over all future corrections, policy updates, and subsequent tasks.