acceptodds
Under review as a conference paper at ICLR 2027

Skill-Space Shooting for Autonomous Robot Policy Improvement

Abstract

Robots deployed in the physical world must be able to improve beyond their initial training as they encounter new situations and failures. For this improvement to scale across tasks, it must make effective use of experience without requiring human demonstration of each correction. Recent agentic systems offer a way to reduce this reliance on human effort by using foundation models to autonomously compose learned behaviors to complete tasks. Yet completing tasks this way does not itself teach a task policy to overcome its own failures; that requires turning these behaviors into learnable corrections for the policy. Our insight is that many such corrections are familiar short behaviors, or skills: they recur across tasks and describe actions that foundation models can reason about from a scene. We introduce skill-space shooting, which uses foundation model guidance to explore corrections through these reusable skills and turn successful trials into policy improvement. Real-world experiments show repeated improvement in policies acting autonomously, while skills can also be shared to reduce the teaching needed to improve on new tasks. By making reusable skills a source of corrective supervision, skill-space shooting enables scalable and generalizable policy improvement within and across tasks. Additional results and videos at https://skill-space-shooting.github.io.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot learning, real-world manipulation, vision-language-action models, failure recovery, value learning, skill reuse

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1NBYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes using reusable skills to teach a policy failure-recovery behaviors. During policy rollouts, a failure detector identifies potential failures, and a VLM selects corrective skills based on the scene, task description, and available skills. Successful repair segments are then added to the training dataset to improve the policy's robustness to failures.

Critical strengths

None

Critical weaknesses

1. Unclear justification for using skills only for recovery.

While I agree that sharing skills across tasks can improve robustness, the paper does not clearly justify restricting skill execution to recovery. Would using skills throughout the entire task, or invoking them more frequently, yield better performance? An ablation on intervention frequency would help clarify this design choice.

2. Concerns about how the task policy improves over time.

  • Frozen failure detector: My understanding is that the failure detector is trained on initial episodes and then frozen. As the task policy evolves, how can the detector identify failures on updated policy? If it continues to trigger at similar failure states, what benefit does updating the task policy provide?
  • Benefit of policy updates: The newly collected recovery trajectories come from skills already available in the library. Why should training the task policy on these trajectories improve performance beyond directly invoking those skills? The paper should clarify what the task policy learns that the existing skill library cannot already provide.
Questions for authors

Suggestions for Clarity

Throughout the paper, I found several concepts and claims that could benefit from clearer definitions or explanations. While I could often infer the intended meaning from the context, explicitly clarifying these points would make the paper easier to follow. Please disregard if some of my suggestions are trivial.

  • L154: Explicitly define the task policy (πtask\pi^{\mathrm{task}}), its dataset (Dtask\mathcal{D}^{\mathrm{task}}), and each skill policy (πskill,i\pi^{\mathrm{skill},i}) and their corresponding dataset. Clarify how these policies and datasets are constructed. This notation would help clarify that the framework maintains two separate sets of policies: a task policy and a library of skill policies. Initially, I found this distinction unclear and assumed the task itself was executed by composing skills.
  • L155 "For simplicity": Explain why is it simplified.
  • L161: Hard to grasp what "our own sharing experiments" refer to. Elaborate what it means and maybe refer to the corresponding section.
  • L229: Clarify why using a fixed prompt makes it scalable.
  • L252: Clarify what is excluded by "the policy's actions and rejected trials." (both task and retrial behaviors are "policy's action")
6 comments
AuthorsOpenReview ↗

Response to Reviewer 1NBY (1 / 3)

Thank you very much for your review.

W1: “Justification for using skills only for recovery.” Would executing them throughout the task or invoking them more frequently help?

Our goal is not to create a system that can arbitrarily complete tasks, but rather to develop a framework for autonomously improving policies themselves. Targeted intervention concentrates improvement on the learned policy’s remaining difficulties. Letting the task policy continue outside of interventions preserves its useful execution, and exposes the next error along its own trajectories. In contrast, running skills throughout the task instead only collects states visited by the skill chain.

Using the skills throughout the task at test time is also exactly our CoP baseline, in which a person sequences the skills. The skill-space shooting policy outperforms this baseline by 32.5 and 28.8 progress points on Coffee and Sweeping and scores above it on Stack-3 and Drawer (Table 1). For intervention frequency, we swept the trigger threshold τ\tau on Drawer, each arm running three rounds from the same π0task\pi^{task}_0 policy (0/20) with the same collection time:

DrawerTriggers per rolloutJudge calls per rolloutπ1task\pi^{task}_1π2task\pi^{task}_2π3task\pi^{task}_3π4task\pi^{task}_4
τ\tau = -0.20.60.67/2010/2011/2012/20
τ\tau = 0 (as in the paper)0.80.87/2012/2011/2012/20
τ\tau = +0.41.61.65/209/2010/2010/20
τ\tau = +0.5 (using skills throughout the entire task)N/A3.36/208/209/209/20

The threshold trades how fast the policy improves against how good the collected corrections are, and the midpoint τ\tau = 0 that we use on every task gets both. Lowering τ\tau to −0.2 triggers less often (0.6 vs. 0.8 triggers per rollout). The policy then improves more slowly (10/20 vs. 12/20 after two updates) but reaches the same final success of 12/20. Raising τ\tau to +0.4 makes the value model fire about twice as often and doubles the judge calls, yet final success drops to 10/20. We attribute this to the extra triggers firing where the task policy was already doing well, so the loop admits repair segments that replace adequate behavior instead of fixing failures. Running the skills throughout the task (τ\tau = +0.5) is the extreme case: it reaches only 9/20, slightly below the HUMAN baseline (10/20), because a full skill chain is a sub-optimal source of new data. The corrections are therefore most useful when they target predicted failures. A trigger that is too insensitive slows improvement, and one that is too sensitive dilutes the data with segments that were never needed. Final success stays between 10/20 and 12/20 across thresholds from −0.2 to +0.4, so one untuned threshold is enough, and invoking skills more often adds skill time and VLM calls without improving the policy.

In particular, employing a local value model in this manner reserves expensive VLM calls for junctures requiring deeper semantic judgment. For example, but our Drawer and Stack-3 tasks in general response, item 1, where we invoke a VLM harness on top of our skills library throughout both tasks, we observed the following median end-to-end latencies:

JudgeMedian latency/call
Gemini 2.5 Pro9.61 s
GPT-6 Astra4.52 s

The faster value model comparator reduces this overhead, allowing the VLM to be invoked only when necessary (e.g. deciding which correction skill to invoke). This latency issue further emphasizes the value of actually learning the corrections themselves to remove the need to make those calls during unaided deployment. Finally, note that the goal of sharing skills across tasks is NOT to improve robustness. The reviewer is correct in that the skills must be robust to facilitate sharing (as we demonstrate in our response to reviewer EKRP’s W2), but the goal of sharing skills across tasks is to facilitate scalable on-policy collection of corrective data. Moreover, under skill-space shooting, the teaching cost for internalizing any skill within the library is upfront paid once and then amortized over all future corrections, policy updates, and subsequent tasks.

AuthorsOpenReview ↗

Response to Reviewer 1NBY (2 / 3)

W2a: “As the task policy evolves, how can the detector identify failures on updated policy? If it continues to trigger at similar failure states, what benefit does updating the task policy provide?”

The frozen value model continues to respond to new behavior because:

  1. The value model is action-conditioned (L206, Eq. 1), and is able to change its prediction when the policy changes its proposed action.

  2. The value model was trained on common failures at every stage before the loop began (General Response, item 2).

A trigger at a familiar state means the updated policy still fails there, which is the intended and desirable behavior. If the subsequent invoked repair succeeds at that failed state, it becomes new supervision (at that state) for the next policy. Over consecutive iterations, these failure modes in the learned policy are gently “ironed out” via this trigger-and-train mechanism. Note that the value model is not used during our reported evaluation (L314–315); the policy executing independently is able to improve on every task: 14/30 to 25/30 on Stack-3, 0/20 to 12/20 on Drawer, 13.8 to 71.3 on Coffee and 18.1 to 83.8 on Sweeping.


W2b: “Why should training a policy on collected recoveries improve performance beyond directly invoking skills?”

Successful skill corrections teach the learned policy new actions in failure states, while skill failures are ignored, thereby preserving good policy behavior despite potential failures. We would like to re-emphasize how the policy imitates the skills. It keeps only accepted repairs from completed rollouts (L248–252), so the 23–30% of skill executions that fail never enter training (Table 2), and at stages it already performs successfully, it keeps its own behavior (L238–240). Thus the task policy learns the skills' successful behavior at its own failure states, without the skills' failures and without hand-offs. The result outperforms both i) invoking skills under human guidance (i.e. the CoP baseline), as well as ii) invoking them with a VLM. The former is Table 1 in the main paper, and the latter is included in our general response item 1. We reproduce both tables here for ease of reference:

TaskInitial policyCoPTrained policy alone
Stack-314/3023/3025/30
Drawer0/208/2012/20
Coffee13.838.871.3
Sweeping18.155.083.8
Test-time systemDrawerStack-3VLM calls per rolloutLatency per call
π0task\pi^{task}_0 alone0/2014/300–
π0task\pi^{task}_0 + value model + skills + Gemini 2.5 Pro5/2016/304.29.61s
π0task\pi^{task}_0 + value model + skills + Astra8/2019/303.84.52s
π4task\pi^{task}_4 alone (ours)12/2025/300–

Note also that the policy’s learnable dataset grows each round while the skills stay fixed (Eq. 2). A natural next step is to add a validated task policy to the library as a longer skill. A reliable drawer-opening-and-placement policy, for example, could supply a correction within a more complex organization task. That higher-level learner would collect successful executions of the new skill at its own failure states and incorporate them through the same update procedure. Repeating this process could turn improvements within individual tasks into a growing library of reusable capabilities. The present experiments establish the first step (learning a stronger task policy from a fixed library) and the sharing results motivate testing this recursive extension, consistent with the library-expansion direction in Section 5, lines 457–462.

AuthorsOpenReview ↗

Response to Reviewer 1NBY (3 / 3)

Suggestions for clarity.

We thank the reviewer for their detailed read. We have incorporated each suggestion in the revision:

  • L154. Let πtask\pi^{task} be the task policy, trained on complete-task demonstrations DtaskD^{task} plus the accepted repairs R0,…,RkR_0, …, R_k (Eq. 2). It executes every rollout. Each skill πiskill\pi^{skill}_i is trained on separately collected stage demonstrations DiskillD^{skill}_i, acts only from a predicted failure until its stage is repaired, and then hands control back to πtask\pi^{task}. Both use a π\pi 0.5-class architecture (L283–285). The skill library stays fixed, and evaluation disables the value model, judge and skills (L314–315).
  • L155 “For simplicity.” Each stage has one designated skill, so choosing a skill reduces to identifying the stage that needs repair (L227–229). This keeps the paper's question, whether skill corrections can teach a task policy, separate from choosing among several skills for one stage.
  • L155 “For simplicity.” Each stage has one designated skill, so choosing a skill reduces to identifying the stage that needs repair (L227–229). This keeps the paper's question, whether skill corrections can teach a task policy, separate from choosing among several skills for one stage.
  • L161. This refers to the sharing experiments in Sec. 4.3 (Pick Sponge and Sponge Wiping, Tables 7–8), and the revised text points there.
  • L229. One prompt template serves all five tasks, including Sponge Wiping. Its only task-specific inputs are the task instruction and the skill names (L819–821), so adding a task means supplying these two inputs with no change to the prompt. The revision replaces "for scalability" with "to reuse the procedure across tasks" and aligns L145 and L274 with this input list.
  • L252. "The policy's actions" are all actions executed by πtask\pi^{task}, including the stages it completes unassisted. "Rejected trials" are skill executions that fail the judge's repair check, including failed retries. Neither enters training. Only the accepted skill execution is kept, from its start to hand-back, and only if the judge confirms that the rollout completed the task (L250–252, App. B.3–B.4).
Reviewer_o2ztOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a method to autonomously correct a policy using skills. The central claim is that these skills can provide autonomous supervision and can be reused across multiple tasks, reducing the human effort to provide demonstrations. A policy is used to roll out an episode. A value model considers the current observation and a sequence of proposed actions to detect whether the continuation will lead to a failure. If so, a foundation model selects a skill from a library of skills defined in advance. The skill is executed as a continuation from the state where a failure is predicted, and the foundation model decides whether the repair is accepted or not. If the repair is accepted, the transitions executed by the skill are used to update the policy using DAgger-style updates. The method is validated on real-world experiments on four tasks (Stack-3, Drawer, Coffee and Sweeping) and compared against alternatives to skill-based corrections (DSRL, human demonstrations, and Chain of primitives). The paper reports that the proposed method reaches the highest performance on all four tasks. The paper also analyses the interventions statistics collected during the experiments reporting the accuracy of failure trigger, skill selection, skill repair success and judge completion. Finally, the paper shows that it is possible to transfer skills across tasks considering an additional task of sponge wiping and sharing skills acquired from Stack-3.

Critical strengths
  • The problem is relevant and well-motivated. Providing human demonstrations is a real bottleneck, so the insight of learning skills that can be reused across multiple tasks can help in this regard. Moreover, using skills as an expert allows the policy to learn in the visited state distribution.
  • The method is demonstrated on a real manipulator, demonstrating the applicability of this method. Moreover, the selected tasks require diverse manipulation skills (grasping, pouring, dynamic contact).
  • The method is validated without judge and skill correction, isolating what the policy has actually learned.
  • The results on Drawer, where the initial policy never succeeds and sparse-reward RL struggles to improve, show the potential of this method and provide insight into when it is most useful.
Critical weaknesses
  • The training of the value model is unclear. From what I understand (from the appendix) the second stage of the value model training appears to require annotating the failure frame. This goes in contrast with the rest of the paper which says that the human is only needed for demonstrations and scene reset (see question 1 for additional details). Moreover, the appendix says that the value model is frozen after a first collection round, but the paper itself argues that as the policy improves it reaches new stages. The failures at these stages are therefore never labeled, so it is not clear how the value model can classify these new cases. For instance, in a pick-then-wipe task such as sponge wiping, wiping failures can only be observed once picking succeeds, which in turn requires the value model to already work for wiping (unless the initial policy already completes picking on its own). This circular dependency makes it hard to understand how the value model is trained in practice. Also, the name "value" is a bit misleading as the function is not a proper value function in the RL sense, but rather a failure classifier.
  • One of the claims is that autonomous skills reduce the human effort needed to improve the policy. Given this claim, I would expect a comparison with corrections provided by a human at the failure states (under a matched human budget). The HUMAN baseline does not answer this question since it adds complete demonstrations from initial states and not failure states.
  • The pick sponge results of section 4.3 are expected. With sharing the pick-primitive receives 150 cube-picking demonstrations + N pick sponge demonstrations, whereas without sharing it only receives N pick sponge demonstrations. Despite the two tasks (pick sponge and pick cube) are not exactly the same, the data still comes from the same setup and a closely related behavior (the only difference between picking a cube and picking a sponge is that the latter is deformable). Thus, it is not surprising that 150 cube + 30 sponge demonstrations outperform 30 sponge demonstrations alone. A zero-shot evaluation of the cube-picking primitive on the sponge would help clarify how much of the gain comes from transferring the skill rather than from the new demonstrations.
  • The VLM should identify the appropriate skill from a faulty state, but the authors state that for simplicity they decomposed the task into stages with designated skills. It appears that the role of the VLM comes down to identifying the current stage, and a heuristic could also do the job. It would be interesting to show settings where the mapping between stage and skill is not one-to-one, so that the VLM has to choose among several plausible skills.
Questions for authors

Q1) how is the value model trained? How are failure frames annotated? The appendix says that the value model is first trained on task data, but by definition the task demonstrations are all successful and only provide positive targets. So how can the value model learn to classify in this stage? Q2) does the frozen value model spot failures in later stages? (see the first weakness for more context) Can you report also the recall? Q3) Why is the flow matching head needed for the value model? and how is the scalar value obtained from it? Q4) Does the judge receive a frame or a sequence of frames? If it does receive only one frame, is it enough to decide the most appropriate skill? Are the skills language-conditioned? For instance, in the Stack-3 example a grasping skill requires to know which cube you are trying to grasp in order to provide a meaningful continuation. Will you release the prompts used? Q5) What is the zero-shot success rate of pick-sponge, using only pick-cube demonstrations and no pick-sponge demonstrations?

6 comments
AuthorsOpenReview ↗

Response to Reviewer o2zt (1 / 4)

Thank you very much for your review.

Q1 and W1. “How is the value model trained?” How are failure frames annotated, and where do the negative examples come from?

We expand on our answer given in the general response (item 2). Critically, the value model’s initial demonstrations contain both successes and failures. In general, many of the tasks we chose to empirically verify our framework had well-anticipated failure modes, such as missed grasps, unstable stacks, and unsuccessful placements. Successful task demonstrations provide +0.5 targets throughout. For failure demonstrations, we manually review each failed recording and mark F, the first frame at which the intended subtask has visibly failed. Negative targets of −0.5 begin H = 16 steps before F. The model therefore learns from proposed actions that lead toward failure, as well as actions from successful executions (App. B.1, lines 796–803).

We use the term “value model” for the action-conditioned score that determines whether the robot should continue its proposed motion or request a repair. This is the source of the intervention signal needed by our improvement loop. However, our contribution is the use of reusable skills to generate corrective supervision and incorporate it into an unaided task policy. This does not depend on introducing a new value modeling or failure detection mechanism. Section 5, lines 468–473, already describes how stronger foundation models could anticipate failure and eventually combine intervention timing with skill selection. Better predictors would make our very same correction-and-learning procedure even easier to apply to new tasks by reducing the aforementioned detector-specific preparation.


Q2. “Does the frozen value model spot failures in later stages?” Can you also report recall?

Later-stage detection is possible because the value model sees stage-specific failures before the autonomous deployment loop starts. To compute the recall for our failure mode, we run the frozen model in shadow mode during 40 additional unaided Drawer rollouts per checkpoint. Without looking at the value model’s scores, we manually identify the first task-ending failure and its stage. A failure is considered caught when the score crosses the threshold within the preceding 16 executed steps. Early-stage failures concern drawer opening; later-stage failures concern object placement or drawer closing. We present the quantitative results below (entries are caught failures divided by total failures.)

PolicyEarly stagesLater stages
Initial78.1% (25/32)75.0% (6/8)
Update 182.4% (14/17)77.8% (7/9)
Update 277.8% (7/9)71.4% (5/7)
Update 371.4% (5/7)75.0% (6/8)
Update 480.0% (8/10)85.7% (6/7)

The updated policies have fewer failures, while the frozen model continues to catch most of the remaining early- and late-stage failures. False alarms on successful rollouts occur at 0.08 per policy-minute, and caught failures have a median warning of 13.2 executed steps. This combination demonstrates the value mode’s utility for intervention: true positive detection remains high through iterations, warnings arrive consistently before failure, and successful behavior rarely prompts an unnecessary takeover.

AuthorsOpenReview ↗

Response to Reviewer o2zt (2 / 4)

Q3. “Why is the flow matching head needed for the value model?” How is the scalar value obtained?

The flow head is a particular instantiation of producing the action-conditioned intervention score. We condition on DINOv2 image tokens and encoded proposed actions to jointly predict the post-chunk robot state and value (App. B.1). The +0.5/−0.5 value targets and their timing relative to the failure frame are already specified in Appendix B.1, lines 799–803; the revised main-text explanation will state them alongside the intervention rule. The decoded value component for that particular chunk supplies the scalar used in thresholding Vϕ(o,a)<τV_\phi(o, a) < \tau. This joint prediction gives the model supervision about the physical consequence of the proposed motion as well as its failure-related score. However, our framework is robust to the parameterization of this head. To demonstrate this, we keep the image encoder, action inputs, training episodes, and training budget fixed, and compare the flow head with a ViT readout and an MLP readout that regress the same scalar target. All heads are tested with the same τ=0\tau = 0 threshold, on the same held-out set of 100 failure-approaching and 100 safe chunks, drawn from episodes excluded from training.

MSE (lower is better)

Taskstackthreedrawercoffeespongesweep
Flow Matching0.001070.0145920.0055720.0259910.018364
ViT0.000880.0157590.0057490.0223240.019768
MLP0.0028390.0192620.0073840.0256160.016185

AUC (higher is better)

Taskstackthreedrawercoffeespongesweep
Flow Matching0.9997070.9011260.9941300.9319600.933460
ViT0.9981950.8826600.9792190.9524830.934124
MLP0.9989610.8419450.9823680.9438770.930254

All three heads identify most approaching failures with similar AUC. The small observed differences support the use of simpler readouts where appropriate. The essential ingredient is supervision that connects proposed actions to likely failure; the improvement loop otherwise remains wholly applicable across predictor architectures.


Q4. “Does the judge receive a frame or a sequence of frames?” Are the skills language-conditioned, and will the prompts be released?

The judge receives one current frame, the task instruction and the names of the available skills (L221–222, App. B.2). One frame suffices for these tasks because the stage that needs repair is visible in the scene: which cubes are stacked, whether the drawer is open and the tape inside, where the blocks and the broom are. For tasks whose state is not visible in one frame, a short history would require minimal adjustment of our same prompt.

The skills are indeed language-conditioned. For example, in Stack-3 each pick/place skill names its corresponding cube (pick red, place red on orange, pick yellow, place yellow on red). Without loss of generality for this illustrative task, the Stack-3 prompt lists the stack order explicitly, so the judge simply selects the next unfinished stage in that order. We will include the full selection, repair-verification, and task-completion prompts in the revised manuscript and release them with the code.

AuthorsOpenReview ↗

Response to Reviewer o2zt (3 / 4)

Q5 and W3. “What is the zero-shot success rate of pick-sponge, using only pick-cube demonstrations and no pick-sponge demonstrations?”

The benefit of sharing is to reduce future teaching on new tasks, given a library that has already been trained on previous tasks. The interpretation of Figure 4 is not that “150 cube + 30 sponge demonstrations outperform 30 sponge demonstrations alone”, Rather, these results show how 30 new sponge demonstrations (combined with 150 existing cube demos), reach 85% success, matching the observed success of 50 fresh sponge demonstrations without sharing. The comparison therefore measures a saving of 20 new sponge demonstrations. We nevertheless add the requested zero-shot comparison. We train on 150 cube demonstrations and zero sponge demonstrations. We also test a 20-cube/30-sponge mixture against 50 sponge demonstrations (with no cube demonstrations). All three checkpoints are evaluated on the same 20 sponge placements.

Cube demosSponge demosSuccess
150050
203065
05085

The zero-shot baseline at 50% success shows useful grasping behavior is contained within the demos already before any subsequent sponge teaching. The mixed-data condition reaches 65% using 20 fewer sponge demonstrations than the sponge-only condition, with equal total data. These results clearly separate transfer from subsequent adaptation: cube experience provides a useful starting point, and a smaller amount of object-specific teaching makes the skill dependable enough to support the new task. To re-emphasize: this is the role sharing plays in the full system. Once a local capability is in the library, preparing it for another task can require less teaching on that new task. The resulting skill can then continue to generate many corrections during policy improvement, further separating the cost of teaching a skill from the number of recoveries it can supply.


Q6 and W2. “Comparison with corrections provided by a human at the failure states (under a matched human budget).”

In autonomous improvement after deployment, human effort/involvement is the most critical bottleneck. With this in mind, we run an additional set of evaluations on Drawer which address the reviewer’s question.

Every arm starts from the same π0task\pi^{task}_0 (0/20) and uses the same initial states, update recipe and evaluation, with the listed collection time per round. In HG-DAgger a person watches every rollout, takes over at each failure and demonstrates the correction before handing control back, and only those corrections are aggregated (Kelly et al., 2019). The first schedule evenly splits this human monitoring + collection of corrective behavior between all policy update rounds. The second schedule moves human time to later rounds, where failures are rarer and harder. Our arm is the τ\tau = 0 run of the threshold sweep in the reply to reviewer 1NBY’s W1. Matching collection time gives HG-DAgger 60 minutes of human corrections that our loop does not use, so this comparison favors HG-DAgger over one at a matched human budget, where it would receive no correction time. Our loop's extra human input is one-time teaching, the skill and failure demonstrations and the round-0 labels (totalling ~60 minutes). As we note in Q5, this cost is actually further amortized via sharing and reuse as the skill library scales up and consumes more tasks over time.

Drawerπ1task\pi^{task}_1π2task\pi^{task}_2π3task\pi^{task}_3RolloutsHuman correction time
HG-DAgger, 20 + 20 + 20 min5/207/209/202760 min
HG-DAgger, 10 + 20 + 30 min2/206/2010/202660 min
Ours--12/20-0 min

We conclude that under equal collection time, skill-space shooting produces the strongest final policy while using less total human effort. Moreover, HG-DAgger under successive iterations becomes increasingly time-consuming. The task policy becomes increasingly performant, thereby making failures rarer, subtler and further downstream. By the third iteration, over 60% of the 30 minute human correction time is spent observing and waiting for an eventual failure.

Reviewer_EKRPOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

Authors introduce skill-space shooting, a method to improve robot policies by having a VLM orchestrate a set of frozen learned skills whenever a learned model predicts failure. Authors show that this method autonomously improves performance of a learned policy on 4 real-world robot manipulation tasks.

Critical strengths
  • Results show that the proposed method can significantly improve the performance of a learned policy with online interaction with the environment, by essentially distilling behaviors coming from frozen skills and judgement (which skills to use, i.e. picking the right mode of behavior) coming from a stronger VLM.

  • The method requires no additional human demonstrations (though does require human involvement for resets) after having given the fixed set of demos for the skills, including for tasks that initially get 0% success rate.

Critical weaknesses
  • The problem setting that the proposed method addresses does not seem very relevant to the current state of the field of robotics.

Authors operate under the following setting: humans collect a fixed set of initial demonstrations, which are decomposed into skill policies, which are frozen. During the autonomous policy improvement loop, all of the executed actions come from these frozen skill policies. If the data used to pretrain the learned policy is the same as used to train the frozen skill policies, then all of the improvement seen during autonomous deployment comes either from the VLM showing the right modes of behavior to use at new circumstances, or from the improvement we would expect from filtered BC. For the former, we could simply query the VLM during inference (which is a missing baseline). And for the latter, filtered BC is a weak online learning algorithm [1, 2, 3] which would be expected to perform worse than other online RL methods, and potentially fail if the skills themselves are not successful.

  • The method is not tested out-of-distribution. The improvement pipeline relies on the strength of both the frozen skills and the value function. It is unclear how these components would perform under distribution shifts, which is very important to actually enable “scalable and generalizable policy improvement”.

  • Limited novelty: the proposed method essentially is filtered BC, with the online policy coming from a VLM with frozen skills. None of the components are novel, and it is unclear what new insight there is that hasn’t appeared before in prior autonomous improvement methods, e.g. SOAR.

Readability / Clarity concerns:

  • Figure 1 is very unclear and hard to understand.
  • Please add a figure showing each of the real-robot tasks you use.

References: 1: Kumar, Aviral, et al. "When should we prefer offline reinforcement learning over behavioral cloning?." arXiv preprint arXiv:2204.05618 (2022). 2: Mirchandani, Suvir, et al. “So you think you can scale up autonomous robot data collection?” In Proceedings of the 8th Conference on Robot Learning (CoRL), volume 270 of Proceedings of Machine Learning Research, pp. 2791–2810, 2024. 3: Giridhar, Varun, et al. “Beyond imitation: Self-improving robot policies via off-policy Q-planning”. arXiv preprint arXiv:2608.21204, 2026.

Questions for authors
  • The COP baseline uses a human to decide which skill to use, so it is a sort of oracle method, right? If so, how is it possible that the presented method significantly outperforms this oracle baseline on multiple tasks?
3 comments
AuthorsOpenReview ↗

Response to Reviewer EKRP (1 / 2)

Thank you very much for your review.

W1 (relevance): “The problem setting that the proposed method addresses does not seem very relevant to the current state of the field of robotics.”

Improving a deployed VLA from corrections to its own mistakes is an active problem in robot learning. RECAP [1] incorporates teleoperated interventions, while PLD [2] trains residual-RL specialists to generate data from learner-visited states. These methods establish the value of corrective experience; our paper addresses how to obtain it repeatedly without a person supplying every correction or an RL specialist being trained for each task. A demonstrated skill is reusable expertise. Once taught, it can generate many successful continuations from the imperfect states a task policy reaches. Our loop uses this property to improve the policy after deployment, with the skill library fixed and human teaching concentrated in initialization. The real-robot results show that this is productive even on Drawer, where the initial policy has zero observed successes, and the sharing experiments show that prior skill data can reduce the new teaching needed for another task. These are concrete steps toward reducing the recurring human cost of policy improvement.

[1] Physical Intelligence et al. π0.6∗\pi^*_{0.6}: a VLA That Learns From Experience.

[2] Xiao et al. Self-Improving Vision-Language Action Models with Data Generation via Residual RL. ICLR 2026.


W1 (source of improvement): “If the data used to pretrain the learned policy is the same as used to train the frozen skill policies, then all of the improvement seen during autonomous deployment comes either from the VLM showing the right modes of behavior to use at new circumstances, or from the improvement we would expect from filtered BC.”

The reviewer is correct that every action the loop adds to training comes from the skills. For clarity, we re-summarize the entire framework:

  1. The task policy executes each rollout and holds control until the value model predicts a failure.
  2. A skill policy (chosen by a VLM judge) then acts from the state the task policy reached, only until that stage is repaired, and control returns to the task policy (L201, L234–240).
  3. Training the next round’s task policy keeps only accepted repairs from completed rollouts (L250–252).

Note that the two policies are also initialized from different data: the task policy starts from complete-task demonstrations D^task, and each skill from separately collected stage demonstrations (L154–158).

Because the skills are equipped with their demonstrations, one reasonable counterargument is that these extra demonstrations alone are sufficient to explain the gain. We show this is not the case below via a simple co-training experiment. We co-train π0task\pi^{task}_0 on DtaskD^{task} plus every skill demonstration for the Drawer task (50 total demos), and compare this against a single iteration of skill-space shooting (which sees only 20 corrective demos.)

Co-training Task and Skill Demonstrations4/20
π1task\pi^{task}_1 of Skill-Space Shooting7/20

Thus, the gain depends not on total data volume, but on where the skills' actions are collected (i.e. at the on-policy failure states).

The reviewer also pointed out a few additional baselines or frameworks, which we now contrast more clearly with skill-space shooting.

Querying the VLM at inference. This is the baseline in General Response item 1. π0task\pi^{task}_0 with the VLM and skills at test time reaches 8/20 on Drawer and 19/30 on Stack-3, against 12/20 and 25/30 for π4task\pi^{task}_4 alone. Furthermore, the VLM-assisted system pays an expensive latency cost per call, which makes it impractical for on-robot control.

Filtered BC. Our loop is much closer to DAgger with an automated expert, rather than filtered BC which imitates data already collected. Our “filter” acts on a different source, namely actions from separately trained skills collected at the states the learner reaches, with no task-policy actions recycled for training.

Online RL. The paper compares explicitly against online RL. DSRL receives the same collection time per round (L307–308). It plateaus within 2.5 progress points of its start on Coffee (13.8 to 16.3) and Sweeping (18.1 to 20.0) and reaches at most 2/20 on Drawer, where one update of ours goes from 0/20 to 7/20 (Tables 1, 5). Sparse-reward RL has to discover corrective behavior from terminal rewards, whereas the skills supply this directly in skill-space shooting(L405–408). Note that the Q-Planning [3] baseline pointed out by the reviewer similarly cannot bootstrap from a policy with no successes (its Sec. 6), which is where our Drawer policy starts. The revised related work now also incorporates the reviewer’s [1–3] through this lens.

AuthorsOpenReview ↗

Response to Reviewer EKRP (2 / 2)

W2: “The method is not tested out-of-distribution… It is unclear how these components would perform under distribution shifts.”

We include two new evaluations on OOD settings for the Drawer task. In particular, 1) we use an alternate roll of tape with a different color and smaller size, and 2) changed environmental illumination using a focused tripod lamp (third-view), instead of the original top-down diffuse environment lighting. Each experiment starts from the previously trained task policy. We freeze the skill library, value model, and judge prompt, and update only the task policy using accepted repairs collected in the changed scene. There are no new demonstrations or failure labels for either shift. Each checkpoint is evaluated without assistance on 20 initial states.

ShiftBeforeUpdate 1Update 2Update 3Update 4
Object color/size040606560
Lighting030455555

Both changes initially impair the task policy, yet the existing corrective components provide enough useful supervision to recover and improve performance. The gains are 60 and 55 percentage points after the same four iterations of updates. This validates our complete improvement process under potentially dramatic distribution shifts.


W3: “It is unclear what new insight there is that hasn't appeared before in prior autonomous improvement methods, e.g. SOAR.”

To the best of our knowledge, skill-space shooting is the first work to demonstrate how a library of reusable skills can serve as the expert in DAgger, with no reward and no per-task RL, to improve end-to-end policies until they outperform any naive composition of those skills.

SOAR improves a policy by re-imitating its own VLM-verified successes, which bounds its data by what the collecting system already does. Our loop imitates a different controller, the VLM-selected skills, at the task policy's failure states and keeps none of the task policy's actions. The policy thus improves without first producing the correct behavior, as on Drawer, which rises from 0/20.

PLD is the closest prior work: it hands control from a VLA to a per-task residual-RL expert at random probe steps and fine-tunes on the result. Our expert needs no reward or RL, its skill data can be shared across tasks (Sec. 4.3), and it acts only where the value model predicts failure. The aggregation rule is that of human-gated DAgger (HG-DAgger), with its human gate and human expert replaced by the value model, the judge and the skills. Tables 1, 7 and 8 support the insight: the policy alone beats COP by 32.5 and 28.8 points on Coffee and Sweeping; corrections at the policy's failure states beat duration-matched complete human demonstrations by 25.0 and 20.6 points, the advantage RaC reports for human corrections, here with no person correcting; shared skill data cut the new demonstrations a new object needs and support improvement on a new task (Table 7)

Together, these results establish our core set of novel contributions: that a reusable library can supply useful supervision at the learner’s failures, improve an unaided policy, and reduce the new teaching needed to extend that library.


W4: Readability and clarity, including Figure 1 and a figure of the real-robot tasks.

We have prepared a new overview diagram that separates initialization, collection, training-data selection, and evaluation. It shows the task policy executing between interventions, a skill temporarily taking control at a predicted failure, and only accepted skill segments from completed rollouts entering the next update. The evaluation panel explicitly removes the value model, judge, and skill library. A companion montage shows real execution frames for Stack-3, Drawer, Coffee, and Sweeping.


Q5. “How is it possible that the presented method significantly outperforms [the COP] baseline on multiple tasks?”

CoP’s human guidance improves selection and recovery decisions, but the selected skill still has to execute the motion successfully. A correct choice of grasping skill can still fail to actually grasp; a successful grasp can still then be followed by a failed placement. Rewinds provide additional attempts, but the complete execution still depends on the skill controllers at every stage (App. D.1). The learned policy combines complete-task demonstrations with successful corrections collected from its own rollout states. It can preserve a grasp it already performs well, acquire a better placement from a skill, and execute the combination without interruption.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer EKRP,

Thank you again for your review. With the discussion period closing soon, we wanted to check whether our response has addressed your concerns. To summarize:

  • Querying the VLM at inference (W1). We added this baseline. π0task\pi^{task}_0 with the value model, skills and a VLM (Gemini 2.5 Pro or GPT-6 Astra) reaches at most 8/20 on Drawer and 19/30 on Stack-3, against 12/20 and 25/30 for π4task\pi^{task}_4 alone. Each VLM call also halts the robot for 4.5–9.6 s.

  • Filtered BC and Other Baselines (W1). Skill-space shooting does not train on any of the task policy's actions, which makes it more akin to DAgger, with the skill library as the expert, rather than filtered BC. We show that beyond data volume, the important lever here is additional intervention data. The revised related work also incorporates your references [1–3].

  • Distribution shift (W2). With the skills, value model and judge prompt frozen and no new demonstrations or labels, Drawer success rises from 0% to 60% after an object change and to 55% after a lighting change.

  • Novelty (W3). To the best of our knowledge, skill-space shooting is the first work to demonstrate how a library of reusable skills can serve as the expert in DAgger, with no reward and no per-task RL, to improve end-to-end policies until they outperform any naive composition of those skills. Empirically, under equal collection time, our loop even stays ahead of human-gated DAgger, in which a person manually corrects each failure for 60 minutes in total, at every iteration (see reply to Reviewer o2zt Q6).

  • Clarity (W4). The revised PDF adds a text-only overview diagram (Fig. R2) and the task montage (Fig. R1).

If these address your concerns, we would be grateful if you would reconsider your score. If a concern remains, please let us know which one, and we will do our best to address it before the end of the discussion period.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to Reviewers (1 / 2)

We thank the reviewers for their detailed feedback. We have made substantial improvements to the body of the paper, complete with several new experiments and evaluation settings, clarifications, and additional details. We invite the reviewers to peruse these improvements in the updated paper, marked in blue. To summarize:

  • VLM at inference: the initial task policy with the value model, the skills and Gemini 2.5 Pro or GPT-6 Astra, against the final skill-space shot policy alone (item 1).
  • Co-training control: the initial task policy trained on the task demonstrations plus every skill demonstration (EKRP, W1).
  • Scene shift: OOD evaluation on the Drawer task with a new object and new lighting, with every other framework component frozen (EKRP, W2).
  • COP failure explanation: where and why COP's rollouts fail (EKRP, Q5).
  • Value model: recall measured in shadow mode, and an ablation of the value head (o2zt, Q2 and Q3).
  • Sharing: zero-shot and equal-total-data skill transfer to Pick Sponge (o2zt, Q5).
  • Human corrections: Human-DAgger at matched collection time (o2zt, W2).
  • Intervention frequency: a sweep of the trigger threshold τ\tau, plus an arm that runs the skills at every stage (1NBY, W1).

In the remainder of our rebuttal, we first address two questions shared across the reviews: why learning from skill corrections improves on invoking them at test time, and how the frozen value model supports continued improvement. On notation, we define πitask\pi^{task}_i to be the task policy after the ith iteration of skill-space shooting (so π0task\pi^{task}_0 is the initial task policy trained on initial demonstrations, before any autonomous improvement).

1. Why not invoke the VLM at inference, or compose skills at inference? Why can the learned policy outperform the CoP baseline?

A VLM composing the skills is bounded above by our CoP baseline, in which a person chooses the skill for each stage and may rewind and retry up to three times per rollout (L953–955). Our policies updated by skill-space shooting already clearly beat CoP by 32.5 progress points on Coffee and 28.8 on Sweeping, and also score above it on Stack-3 and Drawer (Table 1).

This is because CoP executes each stage with a skill, and the skills are imperfect: in our loop they repair only 70–77% of correctly triggered failures (Table 2). Skill-space shooting thus only imitates the skills when their executed corrections are successful (L248–252); failed skill executions never enter training. This results in a DAgger-style setting where the policy learns the skills' successful behavior on its own state distribution, which mitigates the compounding error of imitating fixed demonstrations (Ross et al., 2011).

Finally, to test keeping the VLM in the loop at test time, we ran π0task\pi^{task}_0 with the value model, the skills and an off-the-shelf VLM selecting and checking each repair, either Gemini 2.5 Pro (the model behind our judge) or GPT-6 Astra. We compared each with π4task\pi^{task}_4 acting alone (i.e. the learned policy after 4 iterations of skill-space shooting):

Test-time systemDrawerStack-3VLM calls per rolloutLatency per call
π0task\pi^{task}_0 alone0/2014/300–
π0task\pi^{task}_0 + value model + skills + Gemini 2.5 Pro5/2016/304.29.61s
π0task\pi^{task}_0 + value model + skills + GPT-6 Astra8/2019/303.84.52s
π4task\pi^{task}_4 alone (ours)12/2025/300–

Beyond markedly underperforming skill-space shooting, including the VLM during inference also exposes an extreme latency issue. Latency is the median wall-clock time from request to parsed response, during which the robot holds still. Note that the VLM-assisted system must pay this latency cost for every call (i.e. every time the value model triggers). Without the value model, the latency cost would manifest at every chunk, i.e. the VLM must determine the accuracy/fidelity of every action issued. This is completely prohibitive for real robot control.

AuthorsOpenReview ↗

General Response to Reviewers (2 / 2)

2. How is the value model trained, and why does it remain useful despite freezing?

The value model receives all of its labels by the end of the first collection round and is frozen after that. VϕV_\phi scores the current observation together with the task policy's proposed 16-step action chunk (Eq. 1), and its labels come from three sources:

  • Positives. The already collected successful task demonstrations, with target labels +0.5 throughout.
  • Negatives. Failure demonstrations for every stage (1:1 ratio with successful demos), each with a marked failure frame F. Chunks from one chunk length (H = 16 steps) before F onward receive −0.5, so the model flags a motion while a skill can still repair it (L211–213, App. B.1). Many failure modes of these tasks can be anticipated (a missed or slipping grasp, a toppled cube).
  • Retraining. The model is retrained once, from scratch, on these data plus the labeled rollouts of the first collection round (L805–806, Table 4), then frozen, with trigger threshold τ\tau = 0 on every task.

The failure demonstrations and first-round labels are part of the one-time initial teaching, with the task and skill demonstrations. In the revised draft, we make more explicit that the initial demonstrations include both successful and unsuccessful demonstrations.

Note that despite freezing the value model, it can still respond differently to the different proposed actions. For example, between iterations, an updated policy can propose a safer chunk in the same state and receive a different score.

Finally, our autonomous framework depends on a detector of likely failures, but the results and our core contributions are independent from the instantiation of this detector. Foundation models already detect execution failures and track task progress (Hansen & Xu, 2026), and learned progress and reward models such as ReWiND (Zhang et al., 2025), RoboReward (Lee et al., 2026) and TOPReward (Chen et al., 2026) score robot video. Extending them to proposed motions would let one model decide both when to shoot and which skill to try. Better detectors will replace ours, while the autonomous loop that turns skill corrections into a better policy will continue to carry over unchanged.

Hi everyone, please acknowledge the authors’ rebuttal with a brief reply, even if your assessment is unchanged (if you already did or are about to do so, please ignore this message).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.