- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents REPAIRSPACE, a failure representation that captures robot failures based on their responses to interventions. The authors obtain recovery outcomes for a set of simulator checkpoints and learn both a failure representation and a response predictor, using an ordinal loss to bring failures with similar recovery responses closer together. During deployment, the approach uses a cost budget to choose a correction before running a fixed policy. The paper evaluates recovery for unseen tasks, failures caused by policies not used during training, held-out intervention parameters, and a small real-robot experiment.
Critical strengths
- The paper raises a good question: should two failures be considered similar if the same correction can fix both? Defining similarity based on repair results offers a concrete approach to categorize failures based on recovery, apart from perceptual or semantic category labels.
- The comparison with the response-only baseline is a key factor in this evaluation. When equal supervision, architecture, and search budget are used, the proposed objective increases unseen-task success from 79.8% to 82.1%. The ablations also quantify the benefit of confidence filtering and variance correction.
Critical weaknesses
- Implementation details are unclear, especially the architecture and training procedure of the continuation policy and how REFLECT and FLARE were adapted to the shared intervention library and budget. These details would be useful for reproducing and understanding the differences in the comparisons.
- The cross-policy experiments alter the policy that produces failures but hold the continuation policy and intervention library constant. Because recovery responses are reliant on both of these, these experiments demonstrate transfer within a specified recovery condition. While the authors note this as a limitation, it is not clear if the representation would remain useful with a changing continuation policy.
- The bound is about transferring an intervention from a nearby measured failure, whereas the deployed algorithm uses the learned response head and CEM search. The ordinal objective learns relative similarity over sampled interventions, whereas the bound assumes uniform control over response differences. The paper separates these settings, but the analysis doesn't lend much direct support to the deployed procedure.
- The real-robot evaluation tests eight planned failure configurations, each with ten trials, where REPAIRSPACE gets 68 successes while the response-only baseline gets 62. This is promising, but the small number of configurations and trials makes it hard to be sure how reliably this improvement would be observed. Providing an uncertainty estimate for the difference and testing more failures would be better.
Questions for authors
- What continuation policy is used after each correction, and how is it trained? Would the learned representation still transfer if we changed the continuation policy, rather than only the policy that generates the failures?
- Could the authors indicate the uncertainty in the real-robot improvement over the baseline, and whether trials were paired across methods under similar initial conditions? Demonstrating typical failures, attempted corrections, and resulting recoveries would aid interpretation.