- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This manuscript presents Recova, a framework for autonomous robot manipulation failure recovery guided by an agent. The core idea is a real-sim-real closed loop where a coding agent explores failures in a digital twin diagnoses causes and generates corrective programs then validates them on a real robot and feeds the experience back into policy training. The system uses decoupled task and recovery policies plus a reusable skill library and falls back to human demonstrations via DAgger when needed. The reported numbers look strong with human intervention dropping from 87.5% to 0% and success rising from 23.8% to 87.5% and it also beats baselines on LIBERO-Pro and MolmoSpaces.
Critical strengths
- The system connects failure detection, recovery execution, outcome verification, and human demonstration collection. Experience from handling current failures is reused in subsequent training, giving the framework clear practical value for deployment.
- Successful task trajectories and recovery trajectories update their respective policies. New failure cases are incorporated into subsequent deployment through demonstrations and skill registration. This provides a clear mechanism for continually expanding recovery capabilities and is a main contribution of the system. 3.In addition to two simulation benchmarks, the paper compares performance with and without recovery on four real-world tasks. Enabling recovery improves mean success by 10 percentage points, providing direct evidence that recovery complements the task policy.
Critical weaknesses
- Table 3 demonstrates the gains from DAgger and enabling recovery, but does not isolate the effects of digital-twin initialization, ongoing skill development by the agent, or continual recovery-policy updates. This makes the necessity and added value of the real–sim–real pathway difficult to assess.
- The paper supports an observed reduction in interventions in one small-scale experiment, but does not establish how stable or generalizable this reduction is, or which component drives it. Fewer interventions could result from the task policy making fewer errors, or from the recovery policy handling errors more effectively.
- Autonomous recovery trajectories that pass the VLM check enter the next training round, but the paper presents only judgment examples without quantifying errors. Falsely accepting an unrecovered scene could cause another task failure and introduce incorrect labels into recovery training, undermining the reliability of the learning loop.
Questions for authors
- How much does digital-twin initialization contribute? The paper gives no dedicated ablation of the digital twin's role in building the initial policy and treats it as a prerequisite, so its independent contribution isn't established through a controlled comparison. With the same amount of real-world data, how would removing digital-twin initialization affect initial performance and later learning?
- Difficulty attributing gains to skill expansion. Skills accumulate across rounds, but the paper reports only the final skill library without identifying which skills were added in each round or what gains they provided. This makes it hard to tell whether every skill is beneficial or whether one particular new skill drives an improvement in a given round, which weakens the attribution of gains to the recovery mechanism.
- How do visual recovery-check errors affect execution and training? Using independent human annotations, could the authors report the proportion of unrecovered scenes incorrectly judged as recovered, and the success rate of continued task execution after recovery passes verification? Were the accepted autonomous recovery trajectories given additional human review before being added to the training set?
3 comments
Response to Reviewer gKdR (2/2)
Q: Which skills were added in each round, and what gains did they provide?
A: Fig. 4 shows the overall gains from iterative learning, including improved task success and reduced human intervention. To better understand the contribution of each recovery skill, we additionally report when it was introduced and how often it successfully recovered failures during subsequent real-robot deployment.
| Recovery skill | Introduced | Triggered | Successful recoveries |
|---|---|---|---|
| Return drawn tile to wall | Round 1 | 10 | 5 (50.0%) |
| Realign displaced hand tile | Round 1 | 11 | 7 (63.6%) |
| Stand fallen tile upright | Round 2 | 17 | 7 (41.2%) |
The first two skills were introduced in round 1, followed by the fallen-tile skill in round 2. In rounds 3–4, the robot continued to reuse and refine the existing skills.
As different failure types arise across deployment episodes, these statistics provide a broader view of how individual skills are used in practice. Across subsequent real-robot deployment, all three skills successfully recovered failures, totaling 19 successful recoveries out of 38 activations. These results show that the learned recovery capabilities extend beyond a single skill, with each addressing a distinct failure mode.
Beyond the tile manipulation example, the table below summarizes seven learned recovery skills across the evaluated tasks. Enabling the recovery library improved task success by 5–15 percentage points, further demonstrating its effectiveness across different manipulation tasks.
Learned real recovery skills after 4 rounds
| Task | Success without skills (%) | Success with skills (%) |
|---|---|---|
| Pencil box | 70.0 | 85.0 |
| Stack rings | 75.0 | 90.0 |
| Draw tile and discard tile | 82.5 | 87.5 |
Recovery skills by task
- Pencil box: Reinsert the pencil; slide a detached tray back into its sleeve
- Stack rings: Push down a stuck ring; remove a ring stacked out of order
- Draw tile and discard tile: Stand a fallen tile upright; return a drawn tile to the wall; realign a displaced hand tile
Q: How do errors of the visual recovery check affect execution and training?
A: We evaluate restoration-check reliability using independent human annotations (General Response (2)). The monitor has a 6.5% false-accept rate. Among accepted restorations, the task policy subsequently completed the original task in 88% of cases, while the remaining 12% required human intervention.
At execution time, the task policy restarts under continued monitor supervision. If the restored scene remains invalid and another failure is detected, the system requests further intervention; two consecutive interventions trigger a full human demonstration.
For training, accepted autonomous recovery trajectories are added to the training set without additional human review, relying on the VLM restoration check for filtering. While false accepts may introduce noisy supervision, the measured low false-accept rate suggests that such errors are relatively infrequent.
Please do not hesitate to let us know if you have any additional comments.
Dear Reviewer gKdR,
Thank you again for your valuable feedback and the time you have dedicated to reviewing our paper.
We wanted to kindly follow up and see if you have any remaining questions or concerns regarding our rebuttal. Please feel free to let us know if any part of our response would benefit from further clarification.
Thank you again for your time and thoughtful consideration!
Response to Reviewer gKdR (1/2)
We thank the reviewer for their valuable feedback. We address your comments in the following.
Q: Table 3 does not isolate digital-twin initialization, skill development, or recovery-policy updates.
A: We provide complementary evidence for each component's contributions:
Digital-twin initialization. We conducted a controlled real-robot ablation using identical failure states. A recovery policy trained only on digital-twin rollouts achieves 24% recovery success without human recovery demonstrations. With the same real-data budget, adding digital-twin rollouts improves recovery success from 72% to 78% (General Response (1)).
Skill development. Enabling learned recovery skills improves task success across all evaluated tasks by 5–15 percentage points (skill table below). Fig. 4 provides a concrete example: a fallen-tile recovery skill added in round 2 successfully resolved two subsequent failures, including the only failure in round 4, which required no human takeover.
Recovery-policy updates. New recovery skills become executable after the recovery policy is updated with their demonstrations, while existing skills are refined through additional human demonstrations following failed recovery attempts. These updates support both skill acquisition and refinement.
Together, these results show how digital-twin initialization, skill development, and recovery-policy updates contribute to the effectiveness of Recova.
Q: How stable is the reduction in interventions, and which component drives it?
A: The reduction in human intervention persists over a larger deployment set. Across 155 additional episodes collected on four workstations, the human takeover rate remained at 20.6%, substantially below the 87.5% observed in the initial round, with similar rates across tasks (18.9–23.1%).
Both components contribute, but in different ways. Task-policy improvement reduces how often failures occur, with standalone success increasing from 12.5% to 57.1%, 75.0%, and 85.7% across the four rounds in Fig. 4. Recovery learning further reduces human intervention by autonomously handling failures that the task policy cannot avoid. Table 3 further confirms the contribution of recovery: enabling recovery improves mean task success from 77.5% to 87.5%, even after task-policy fine-tuning.