- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces RoboRSI, a robot self-improvement framework designed for complex, real-world environments. RoboRSI aims to transform execution feedback into skills that can be sustainably accumulated and reused. RoboRSI achieves higher success rates than the baselines across four simulation benchmarks, and ablation studies validate the effectiveness of composite skills in enhancing success rates and reducing execution overhead.
Critical strengths
The paper features a clear overall design and offers valuable solutions to practical challenges encountered during long-term autonomous robot execution.
RoboRSI outperforms the baseline methods across all four simulation benchmarks, achieving an 11.0 percentage point increase in success rate on LIBERO-PRO.
Furthermore, The 104-round physical development study demonstrate the system's potential for practical deployment.
Critical weaknesses
Since RoboRSI simultaneously introduces post-condition verification, failure recovery, multi-agent collaboration, and online skill updates, it is difficult to determine whether the performance gains stem from TSR or from these other mechanisms.
Skill repair relies primarily on offline replay-based verification, lacking repeated testing and regression rate analysis after the skill library has been frozen.
Questions for authors
Could you include more rigorous ablation studies to clearly demonstrate the independent contribution of TSR? Specifically, please compare the complete RoboRSI system against versions that retain only the post-condition verification and recovery mechanisms, as well as a version that employs Flat Revision for online skill updates.
How should versioning, conflicts, and shared dependencies be managed as the skill library grows? How should the impact scope of a single revision be estimated and how does the Manager’s review cost change with library size?
2 comments
Thank you for your rebuttal. I maintain my previous assessment.
Thank you for the positive assessment of the structure, the practical motivation and the physical experiments, and for the concrete ablation design.
W1/Q1: Separating TSR from the other mechanisms.
We ran exactly the requested ablation on LIBERO-PRO under the protocol of Table 1. Variant (b) keeps postcondition verification and recovery but never revises skills; variant (c) adds online skill updates with flat revision; (a) is the full system with TSR. Success (%) (new Appendix K):
Postcondition checking and recovery alone add 0.5 pp over OpenETA, and online updates with flat revision do not improve on them. With TSR, the same online updates raise success to 49.5%, +10.5 pp over (b) and +13.7 pp over (c). A revision-level comparison on 24 failed tasks, with each library evaluated frozen on a held-out seed, gives the same ordering: TSR 11/24, flat revision 8/24, starting library 7/24 (our response to mGzr, W2).
W2: Frozen re-testing and regression.
The LIBERO library is evaluated frozen on the 840 perturbed LIBERO-Plus instances (Table 1: 42.1% vs. 36.4% for OpenETA), and each arm of the 24-task TSR comparison is evaluated frozen on a held-out seed (TSR 11/24 vs. 7/24 for the starting library). Regression is guarded at release time: a revision is released only after it passes replay on every recorded task whose call graph reaches the revised scope (the replay set Ht), so a change that breaks a previously solved task is not released. On the physical robot, validation replays the most recent runs (Appendix F.5).
Q2: Versioning, conflicts, dependencies and review cost as the library grows.
Statistics. Paired over the 120 LIBERO-PRO tasks (5 seeds each), RoboRSI improves over OpenETA by +11.0 pp (95% bootstrap CI [+5.7, +16.3]; better on 54 tasks, worse on 21; Wilcoxon p = 2.2×10⁻⁴), over Maestro by +16.5 pp ([+11.7, +21.7]; p = 3.5×10⁻⁸) and over CaP-X by +31.0 pp ([+25.5, +36.5]; p = 1.7×10⁻¹⁵) (Appendix P).
Thank you again for the concrete suggestions.