acceptodds
Under review as a conference paper at ICLR 2027

Recova: Agent-Guided Failure Recovery for Autonomous Robotic Manipulation

Abstract

Manipulation failures can leave scenes in states from which a task policy cannot recover. Learning corrective behaviors requires scalable failure exploration and physical grounding. We present Recova, an agent-guided framework that jointly develops task execution and recovery in a reconstructed digital twin, then verifies and refines both through real-world experience. In the twin, the agent diagnoses failures, tests corrective programs, and collects successful task and recovery rollouts for separate policies. During deployment, it monitors progress, invokes a learned or programmatic recovery, verifies scene restoration, and resumes execution. When no suitable recovery is available, a human demonstration resolves the failure and enters the learning loop, allowing the system to expand its recovery capabilities. Physical rollouts and human demonstrations are routed to the corresponding policy for DAgger training. Across six LIBERO-Pro settings and four MolmoSpaces categories, Recova achieves 78.8% and 64.9% mean success, compared with 71.7% and 38.0% for the strongest baselines. With parallel collection across four real-robot workstations, DAgger fine-tuning raises mean success from 23.8% to 77.5%, and recovery skills further raise it to 87.5%. Over four collection rounds on one task, observed human intervention falls from 87.5% to 0%. Together, these results show how agent-guided recovery turns failures into reusable capabilities, improving robustness while progressively reducing human intervention.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, Robotic Manipulation, Failure Recovery, Coding Agents, Digital Twins, Human-in-the-Loop Imitation Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gKdROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This manuscript presents Recova, a framework for autonomous robot manipulation failure recovery guided by an agent. The core idea is a real-sim-real closed loop where a coding agent explores failures in a digital twin diagnoses causes and generates corrective programs then validates them on a real robot and feeds the experience back into policy training. The system uses decoupled task and recovery policies plus a reusable skill library and falls back to human demonstrations via DAgger when needed. The reported numbers look strong with human intervention dropping from 87.5% to 0% and success rising from 23.8% to 87.5% and it also beats baselines on LIBERO-Pro and MolmoSpaces.

Critical strengths
  1. The system connects failure detection, recovery execution, outcome verification, and human demonstration collection. Experience from handling current failures is reused in subsequent training, giving the framework clear practical value for deployment.
  2. Successful task trajectories and recovery trajectories update their respective policies. New failure cases are incorporated into subsequent deployment through demonstrations and skill registration. This provides a clear mechanism for continually expanding recovery capabilities and is a main contribution of the system. 3.In addition to two simulation benchmarks, the paper compares performance with and without recovery on four real-world tasks. Enabling recovery improves mean success by 10 percentage points, providing direct evidence that recovery complements the task policy.
Critical weaknesses
  1. Table 3 demonstrates the gains from DAgger and enabling recovery, but does not isolate the effects of digital-twin initialization, ongoing skill development by the agent, or continual recovery-policy updates. This makes the necessity and added value of the real–sim–real pathway difficult to assess.
  2. The paper supports an observed reduction in interventions in one small-scale experiment, but does not establish how stable or generalizable this reduction is, or which component drives it. Fewer interventions could result from the task policy making fewer errors, or from the recovery policy handling errors more effectively.
  3. Autonomous recovery trajectories that pass the VLM check enter the next training round, but the paper presents only judgment examples without quantifying errors. Falsely accepting an unrecovered scene could cause another task failure and introduce incorrect labels into recovery training, undermining the reliability of the learning loop.
Questions for authors
  1. How much does digital-twin initialization contribute? The paper gives no dedicated ablation of the digital twin's role in building the initial policy and treats it as a prerequisite, so its independent contribution isn't established through a controlled comparison. With the same amount of real-world data, how would removing digital-twin initialization affect initial performance and later learning?
  2. Difficulty attributing gains to skill expansion. Skills accumulate across rounds, but the paper reports only the final skill library without identifying which skills were added in each round or what gains they provided. This makes it hard to tell whether every skill is beneficial or whether one particular new skill drives an improvement in a given round, which weakens the attribution of gains to the recovery mechanism.
  3. How do visual recovery-check errors affect execution and training? Using independent human annotations, could the authors report the proportion of unrecovered scenes incorrectly judged as recovered, and the success rate of continued task execution after recovery passes verification? Were the accepted autonomous recovery trajectories given additional human review before being added to the training set?
3 comments
AuthorsOpenReview ↗

Response to Reviewer gKdR (1/2)

We thank the reviewer for their valuable feedback. We address your comments in the following.


Q: Table 3 does not isolate digital-twin initialization, skill development, or recovery-policy updates.

A: We provide complementary evidence for each component's contributions:

  • Digital-twin initialization. We conducted a controlled real-robot ablation using identical failure states. A recovery policy trained only on digital-twin rollouts achieves 24% recovery success without human recovery demonstrations. With the same real-data budget, adding digital-twin rollouts improves recovery success from 72% to 78% (General Response (1)).

  • Skill development. Enabling learned recovery skills improves task success across all evaluated tasks by 5–15 percentage points (skill table below). Fig. 4 provides a concrete example: a fallen-tile recovery skill added in round 2 successfully resolved two subsequent failures, including the only failure in round 4, which required no human takeover.

  • Recovery-policy updates. New recovery skills become executable after the recovery policy is updated with their demonstrations, while existing skills are refined through additional human demonstrations following failed recovery attempts. These updates support both skill acquisition and refinement.

Together, these results show how digital-twin initialization, skill development, and recovery-policy updates contribute to the effectiveness of Recova.


Q: How stable is the reduction in interventions, and which component drives it?

A: The reduction in human intervention persists over a larger deployment set. Across 155 additional episodes collected on four workstations, the human takeover rate remained at 20.6%, substantially below the 87.5% observed in the initial round, with similar rates across tasks (18.9–23.1%).

Both components contribute, but in different ways. Task-policy improvement reduces how often failures occur, with standalone success increasing from 12.5% to 57.1%, 75.0%, and 85.7% across the four rounds in Fig. 4. Recovery learning further reduces human intervention by autonomously handling failures that the task policy cannot avoid. Table 3 further confirms the contribution of recovery: enabling recovery improves mean task success from 77.5% to 87.5%, even after task-policy fine-tuning.

AuthorsOpenReview ↗

Response to Reviewer gKdR (2/2)

Q: Which skills were added in each round, and what gains did they provide?

A: Fig. 4 shows the overall gains from iterative learning, including improved task success and reduced human intervention. To better understand the contribution of each recovery skill, we additionally report when it was introduced and how often it successfully recovered failures during subsequent real-robot deployment.

Recovery skillIntroducedTriggeredSuccessful recoveries
Return drawn tile to wallRound 1105 (50.0%)
Realign displaced hand tileRound 1117 (63.6%)
Stand fallen tile uprightRound 2177 (41.2%)

The first two skills were introduced in round 1, followed by the fallen-tile skill in round 2. In rounds 3–4, the robot continued to reuse and refine the existing skills.

As different failure types arise across deployment episodes, these statistics provide a broader view of how individual skills are used in practice. Across subsequent real-robot deployment, all three skills successfully recovered failures, totaling 19 successful recoveries out of 38 activations. These results show that the learned recovery capabilities extend beyond a single skill, with each addressing a distinct failure mode.

Beyond the tile manipulation example, the table below summarizes seven learned recovery skills across the evaluated tasks. Enabling the recovery library improved task success by 5–15 percentage points, further demonstrating its effectiveness across different manipulation tasks.

Learned real recovery skills after 4 rounds

TaskSuccess without skills (%)Success with skills (%)
Pencil box70.085.0
Stack rings75.090.0
Draw tile and discard tile82.587.5

Recovery skills by task

  • Pencil box: Reinsert the pencil; slide a detached tray back into its sleeve
  • Stack rings: Push down a stuck ring; remove a ring stacked out of order
  • Draw tile and discard tile: Stand a fallen tile upright; return a drawn tile to the wall; realign a displaced hand tile

Q: How do errors of the visual recovery check affect execution and training?

A: We evaluate restoration-check reliability using independent human annotations (General Response (2)). The monitor has a 6.5% false-accept rate. Among accepted restorations, the task policy subsequently completed the original task in 88% of cases, while the remaining 12% required human intervention.

At execution time, the task policy restarts under continued monitor supervision. If the restored scene remains invalid and another failure is detected, the system requests further intervention; two consecutive interventions trigger a full human demonstration.

For training, accepted autonomous recovery trajectories are added to the training set without additional human review, relying on the VLM restoration check for filtering. While false accepts may introduce noisy supervision, the measured low false-accept rate suggests that such errors are relatively infrequent.


Please do not hesitate to let us know if you have any additional comments.

Dear Reviewer gKdR,

Thank you again for your valuable feedback and the time you have dedicated to reviewing our paper.

We wanted to kindly follow up and see if you have any remaining questions or concerns regarding our rebuttal. Please feel free to let us know if any part of our response would benefit from further clarification.

Thank you again for your time and thoughtful consideration!

Reviewer_VEbgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Recova addresses failures in robotic manipulation through a coding agent, digital twins, and policy updates from deployment. The coding agent develops recovery programs and generates training trajectories in reconstructed simulation environments. During deployment, a vision-language monitor assesses task progress, selects recovery instructions, and verifies restoration, while unresolved failures elicit human demonstrations. The real-robot task and recovery policies are both initialized from π0.5 and undergo separate rounds of supervised fine-tuning on their respective data. Simulation experiments use MolmoAct2 with programmatic recovery, reporting mean success rates of 78.8% on LIBERO-Pro and 64.9% on MolmoSpaces. Across four real-robot tasks, mean success increases from 23.8% for the base task policy to 77.5% after DAgger fine-tuning and 87.5% with recovery enabled. The paper also reports changes in human takeover rates across four collection rounds on one task.

Critical strengths
  1. The paper integrates digital-twin construction, recovery-program development, and training-trajectory generation through a coding agent. The twin supports exploration of failure states in simulation, while recovery programs and successful trajectories support skill execution and policy training, respectively. This provides a complete system design connecting failure exploration to recovery-policy learning.
  2. The task and recovery policies are updated iteratively using separately collected data, allowing both task execution and scene restoration to incorporate new demonstrations from deployment. The four-workstation experiments demonstrate the operation of this process. In Table 3, enabling recovery increases mean task success from 77.5% to 87.5%, showing additional benefit after the task policy has undergone DAgger fine-tuning.
Critical weaknesses
  1. Recovery-instruction selection, conditional action execution, and restoration verification are not evaluated separately.

    • The recovery policy is conditioned on observations and a recovery instruction, while appropriate corrections for similar failure configurations may vary with the original task and execution stage.
    • The monitor has access to task context, but the experiments do not separately report instruction-selection accuracy, execution success given a correct instruction, or original-task completion following an accepted restoration. Aggregate success rates therefore do not locate the source of recovery failures or establish whether recovery instructions adequately convey the task context required for execution.
    • The applicability of learned skills to held-out failure configurations or task stages also lacks quantitative evaluation.
  2. The acceptance procedure for simulated recovery data and its contribution to real-world recovery remain unclear.

    • Section 3.2 describes successful recovery rollouts, but does not sufficiently specify how the initial simulated recovery trajectories are accepted, how many are generated and retained, or their distribution across skills.
    • Table 3 shows the deployment benefit of the final recovery module, but the pipeline combines simulation initialization, real-data training, and human demonstrations, preventing attribution of that benefit to agent-generated data.
    • The paper does not report real-robot recovery success when training uses only simulated recovery demonstrations, before adding real human recovery demonstrations, or provide a comparison without simulation initialization under the same real-data budget.
  3. Although Table 5 specifies the optimizer and learning-rate settings, the trainable parameter subsets and per-round data composition remain unspecified, limiting reproducibility.

Questions for authors
  1. At identical failure states, what are the recovery success rate and subsequent original-task completion rate under human-specified correct instructions versus agent-generated instructions? Can this comparison be conducted on held-out failure configurations or task stages?
  2. Are the initial simulated recovery trajectories accepted using simulator-state predicates, a VLM, or human review, and how many trajectories are generated and retained per skill? What real-robot recovery success is achieved using only these simulated demonstrations, before adding real human recovery demonstrations?
  3. Which base model and tool interfaces does the coding agent use? Do the task and recovery policies use full fine-tuning, LoRA, or frozen modules, and what are the per-round data counts and sampling proportions for simulation, autonomous real execution, and human demonstrations?
6 comments
AuthorsOpenReview ↗

Response to Reviewer VEbg (1/2)

We thank the reviewer for their valuable feedback. We address your comments in the following.


Q: Recovery stages and human-specified versus monitor-selected instructions.

A: We first clarify that the monitor does not freely generate arbitrary instructions to steer the recovery policy. Instead, it selects from a registry of previously learned recovery skills, each associated with a specific recovery instruction. When a new failure cannot be handled by the existing library, the system requests a human demonstration, and the corresponding skill can be incorporated through subsequent training.

Thus, the proposed comparison between human-specified and agent-generated instructions is more precisely a comparison between human-oracle and monitor-based skill selection, with the recovery policy held fixed.

To evaluate this component, we manually verified the monitor's instruction selections and found 92.2% accuracy (General Response (2)). In real-robot deployment, 79% of recovery attempts passed the restoration check, and the original task was subsequently completed in 88% of accepted recoveries. These results demonstrate reliable instruction selection and quantify the downstream effectiveness of the autonomous recovery pipeline.


Q: Does the recovery policy receive sufficient task context?

A: Recovery instructions are selected by the monitor using the task context and recent observations. For example, the monitor may select "Return the loose extra tile to the tile wall" rather than a generic instruction such as "Move the tile." Furthermore, the recovery policies are fine-tuned per task. Recovery aims to restore a valid scene from which the original task policy can restart from its home pose, rather than resume the interrupted trajectory. This allows recovery to focus on restoring a task-valid configuration, without requiring the full history of the interrupted execution.


Q: Held-out failure configurations.

A: All reported recovery results are on failure states not seen in training. In simulation, programs are developed and evaluated on disjoint seeds with held-out initial object configurations (Sec. 4.1). On the robot, failures arise naturally from the task policy during evaluation rollouts, after the recovery policy was trained, across four stations with different backgrounds and distractors; the 38 recovery activations in the 155 additional episodes are all such unseen states. What we do not evaluate is transfer of a learned skill to a different task or to a failure type outside the registry. In that case the system does not attempt autonomous recovery and instead requests a demonstration, which becomes a new skill. We will state cross-task skill transfer as a limitation and future work.

AuthorsOpenReview ↗

Response to Reviewer VEbg (2/2)

Q: Simulated recovery trajectory acceptance and contribution to real-world recovery.

A: Simulated recovery trajectories are retained when they satisfy the simulator's success predicate for restoring the scene. To isolate the contribution of the digital twin, we additionally evaluated three recovery policies on the same real-robot failure states (General Response (1)):

  • Digital-twin rollouts only: 24% recovery success.

  • Real data only: 72% recovery success.

  • Digital-twin rollouts + real data: 78% recovery success.

Importantly, the latter two settings use the same real-data budget. Thus, digital-twin demonstrations provide a useful initialization even without human recovery data, and further improve the final recovery policy when combined with real-world demonstrations.

We also evaluate recovery learning in simulation on LIBERO-Pro. A policy trained on simulated recovery demonstrations recovers 32.8% of failed states, compared with 21.5% for task-policy retry, further supporting the utility of generated recovery data.

LIBERO-Pro Goal-Pos: MolmoAct2 followed by each form of recovery

Recovery after MolmoAct2Failed states recovered (%)End-to-end success (%)
None–21.0
Learned recovery policy32.851.8
Recovery program63.685.0

Q: Coding agent, trainable parameters, and training data composition.

A: The coding agent uses Fable 5.1 at maximum reasoning effort. It develops recovery programs using an explicit tool library (Sec. 3.1), including GroundingDINO and SAM2 for object grounding and segmentation, GraspGenX for grasp generation, and collision-aware motion planning with RRT-Connect and damped-least-squares IK in simulation. We will include these details in the final version of the paper.

Both task and recovery policies are initialized from π0.5 and fully fine-tuned, with all 3.35B parameters trainable (no LoRA or frozen modules), following Table 5. As described in Sec. 3.4 (Eq. 1), autonomous successes (S) and human task demonstrations (E) train the task policy, while human recovery demonstrations (R) and verified autonomous recoveries (A) train the recovery policy. Both policies are iteratively updated from their respective previous checkpoints (Appendix C.3). General Response (3) provides the per-task trajectory counts and human control time.


Please do not hesitate to let us know if you have any additional comments.

Dear Reviewer VEbg,

Thank you again for your thoughtful feedback and for taking the time to review our work.

We wanted to kindly follow up and see if you have any remaining questions or concerns regarding our rebuttal. We would be happy to provide any additional information or clarification before the discussion period ends.

Thank you for your time and consideration!

Reviewer_qwepOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes recova, it tries to solve manipulation failures that leave a scene in a state the task policy cannot continue from. Recova's method has 3 stages: 1) Digital twin: A coding agent rebuilds the workstation as a MuJoCo twin and attempts the task there. For each failure it names a recovery instruction z. It then produces two things: a code-as-policy recovery program, and recovery rollouts. 2) Real-robot deployment: A VLM monitor (Gemini 3.8 Flash) runs four kinds of queries: requirement, completion, intervention and restoration. 3) Data routing: Data go into 4 sets: autonomous successes S, human task demos E, human recovery demos R, and VLM-verified autonomous recoveries A.

Critical strengths
  1. The problem is real and important. VLA policies stall in off-distribution states they caused themselves. The motivation is concrete.
  2. The real-robot system is well-established and well-documented.
  3. Data design is handled carefully.
Critical weaknesses
  1. From my understanding of the paper (you may correct me during the rebuttal), the simulation and real-robot experiments evaluate different versions of the method. In simulation, recova is a VLA policy followed by recovery programs, with no learned recovery policy / DAgger, while on the robot only the learned recovery policy is used, so I think this design does not fully show the full loop with both learned and programmatic recovery.
  2. Some controlled experiments can be added as well for the simulation results, since recova gives the VLA its full budget and then a second chance through programs developed on the same benchmark tasks, while other compared baselines are not this kind of "cascade baselines". Probably one / some cascade baselines can be provided, like the same VLA followed by ASPIRE [1]?

[1] Lu, R., Wu, Y., Kou, E., Fu, L., Xiao, W., Mandlekar, A., Xu, Y., Shi, G., Goldberg, K., Chen, A., Chowdhury, M., Zhu, Y., Fan, L., & Wang, G. (2026). ASPIRE: Agentic /skills discovery for robotics (arXiv:2607.00272). arXiv. https://doi.org/10.48550/arXiv.2607.00272

Questions for authors
  1. How does digital twin contribute to the real-robot results? Is the base policy trained on the twin data? How was the gap between MuJoCo renders and real images handled?
  2. In simulation, how many successes are completed entirely by the program? What does MolmoAct2 alone score under your exact protocol?
  3. How many trajectories are in S, E, R and A for each task? How many rounds and human-hours produced Tab. 3?
3 comments
AuthorsOpenReview ↗

Response to Reviewer qwep (1/2)

We thank the reviewer for their valuable feedback. We address your comments in the following.


Q: Full-loop evaluation with both learned and programmatic recovery.

A: We evaluate both forms of recovery in the same simulation setting: the recovery program behind the submission’s simulation results and the learned recovery policy deployed on the robot. Here, the learned policy is trained only on the program’s successful rollouts and takes the program’s place after MolmoAct2. The learned policy recovers a third of the failed states and lifts end-to-end success from 21.0% to 51.8%. The recovery program remains the stronger of the two, and on the robot the learned policy is refined further by DAgger rounds with human demonstrations (Table 3, Fig. 4).

LIBERO-Pro Goal-Pos: MolmoAct2 followed by each form of recovery

Recovery after MolmoAct2Failed states recovered (%)End-to-end success (%)
None–21.0
Learned recovery policy32.851.8
Recovery program63.685.0

Q: Cascade baselines, such as the same VLA followed by ASPIRE, should be provided.

A: We added two cascade baselines: the same VLA as its own second stage, and the same VLA followed by ASPIRE. Every second stage starts from the identical failed scene, with the same remaining budget, the same success rule and no reset.

  1. Same VLA as second stage. MolmoAct2 either keeps acting extra steps or re-homes and retries. From the same failed states and with the same budget, the Recova program rescues nearly twice as many episodes as a retry on LIBERO-Pro and nearly three times as many on MolmoSpaces.

    LIBERO-Pro: rescue rate (%) from identical failed states

    SettingExtra stepsHome and retryOurs
    Obj-Pos16.132.374.2
    Obj-Task11.532.751.9
    Goal-Pos17.421.563.6
    Goal-Task15.412.810.3
    Spa-Pos31.256.266.1
    Spa-Task28.628.628.6
    All20.331.857.8

    Bold marks the best rescue rate in each row, including ties.

    MolmoSpaces: rescue rate (%) from identical failed states

    TaskExtra stepsHome and retryOurs
    Pick14.711.360.7
    Pick-and-place11.412.022.8
    Open5.617.236.1
    Close20.321.956.2
    All10.914.740.1

    Bold marks the best rescue rate in each row, including ties.

  2. Same VLA followed by ASPIRE. On the task where MolmoAct2 attempts first and fails. Recova rescues the episode from the failed state while ASPIRE cannot. ASPIRE is written to solve the task from the initial state and assumes the gripper points down when it starts, which no longer holds after a failed attempt, while Recova are built for recovery and take over from whatever state the policy leaves.

    Starting stateASPIRERecova
    Initial state (no failure)0.921.0
    Failed state left by MolmoAct200.98
AuthorsOpenReview ↗

Response to Reviewer qwep (2/2)

Q: How does the digital twin contribute to the real-robot results?

A: Please refer to General Response (1) for our controlled real-robot ablation. Digital-twin rollouts alone achieve 24% recovery success, and adding them to the same real-data budget improves recovery success from 72% to 78%.

Both the task and recovery policies are initialized from π0.5. We use digital-twin trajectories to train the corresponding policies before real-world deployment. To reduce the sim-to-real gap between MuJoCo renders and real observations, we apply data augmentation and domain randomization. The policies are further adapted to real-world observations through DAgger fine-tuning.


Q: How many successes are completed entirely by the program, and what does MolmoAct2 alone score?

A: MolmoAct2 alone reproduces its published results under our exact protocol: 34.3% on LIBERO-Pro and 36.3% on MolmoSpaces. The table below splits Recova's successful episodes by who completes them. Recova's advantage comes from using both: MolmoAct2 completes the episodes it can, the recovery program recovers a large part of those it fails, and together they outperform either one alone.

Share of Recova's successful episodes

BenchmarkMolmoAct2 completesMolmoAct2 fails; program recovers
LIBERO-Pro35.5%64.5%
MolmoSpaces57.9%42.1%

Q: How many trajectories (S, E, R, A), rounds and human-hours produced Table 3?

A: Please refer to our General Response (3).


Please do not hesitate to let us know if you have any additional comments.

Dear Reviewer qwep,

Thank you again for your thoughtful feedback and for taking the time to review our work.

We wanted to kindly follow up and see if you have any remaining questions or concerns. We would be happy to provide any additional clarification before the discussion period ends.

Thank you again for your time and consideration!

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (1/2)

We thank the reviewers for their constructive feedback and recognition of Recova's contributions to robust robotic manipulation. The reviewers highlighted the importance of failure recovery (qwep), the end-to-end integration of digital-twin exploration and recovery-policy learning (VEbg), and the practical value of continually learning from deployment failures (gKdR).

To address the reviewers' questions, we provide additional controlled experiments and quantitative analyses that clarify the contribution of digital-twin initialization, evaluate the reliability of recovery selection and verification, and quantify real-world data collection and human effort. We summarize these results in three common responses below, followed by reviewer-specific answers.

AuthorsOpenReview ↗

General Response (2/2)

(1) Contribution of the Digital Twin. [Reviewer qwep, VEbg, gKdR]

Our controlled real-robot ablation demonstrates two complementary benefits of digital-twin training: it enables recovery before any human recovery demonstrations and improves performance when combined with real-world data. We evaluate three recovery policies trained with different data sources on identical failure states. A policy trained solely on digital-twin rollouts already recovers 24% of failures without human demonstrations. Under the same real-data budget, adding digital-twin rollouts further improves recovery success from 72% to 78% and subsequent task completion from 74% to 88%.

Training dataRecovery success (%)Task completed afterwards (%)
Digital-twin rollouts only2418
Real data only7274
Digital-twin rollouts and real data (ours)7888

(2) Stage-wise Evaluation and Monitor Reliability. [Reviewer VEbg, gKdR]

Following the reviewer’s request, we measured each stage in the real-robot deployment. We manually verified the accuracy of instruction selection and the restoration check.

  • Failure detection. The monitor rarely misses a failure: only 4% of human takeovers were initiated by the operator without a monitor request.

  • Instruction selection. The monitor selected the correct recovery instruction in 92.2% of attempts.

  • Recovery execution and restoration check. The check accepted 79% of attempts and returned the other 21% to the operator, with only 6.5% false accepts. Recovery failures therefore arise mainly in skill execution, not in instruction selection or verification.

  • Task completion after an accepted restoration. The task policy then completed the original task in 88% of cases; the remaining 12% ended in a human takeover.

(3) Data and Human Effort per Task. [Reviewer qwep, VEbg]

We provide the trajectories in S, E, R and A for each task below, together with the human control time. Each task was learned with 0.5-1.5 hours of human control, and a large part of its data are autonomous successes, which cost no human time. Most human demonstrations are recovery demonstrations RR, not task demonstrations (E), so the human time goes to the failures the policy cannot yet handle.

TaskSERAHuman control (min)
Pencil box6726321277.5
Stack rings1188112343.6
Draw tiles8724341889.5
Discard tiles9019281562.6

(S: autonomous successes. E: human task demonstrations. R: human recovery demonstrations. A: recoveries by the recovery policy that passed the restoration check.)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.