- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper investigates silent erosion in embodied recursive self-improvement (RSI), where nominal task success improves while tail and error-recovery capabilities deteriorate. The authors identify two contributing mechanisms, selection bias and visitation drift, and propose RESCUE, a coverage-aware data selection and retention method designed to preserve previously acquired capabilities. Experiments across multiple VLA architectures and benchmarks, together with qualitative real-robot demonstrations, show that RESCUE can substantially mitigate recovery degradation while maintaining nominal performance.
Critical strengths
-
Meaningful research problem. The paper identifies an important failure mode in RSI, where improvements in nominal success can mask degradation in tail and recovery capabilities. Controlled experiments provide useful evidence for the roles of selection bias and visitation drift.
-
Comprehensive experimental evaluation. The authors evaluate multiple VLA architectures and benchmarks, with extensive baseline comparisons and ablation studies. The inclusion of real-robot experiments further supports the practical relevance of the findings.
Critical weaknesses
- Limited validation of the generality of the findings. The experiments primarily focus on success-filtered RSI. It remains unclear whether similar degradation would occur when recovery tasks are sufficiently represented during training, or whether RESCUE would provide substantial additional benefits when the initial policy already has strong recovery capabilities.
Questions for authors
-
If each RSI generation explicitly includes recovery tasks initialized from failure states, with recovery success incorporated into the training objective, would the observed divergence between nominal and recovery success still occur under a matched training budget? How much additional benefit would RESCUE provide in this setting?
-
Table 9 shows that increasing initial recovery demonstrations (R30 to R130) can prevent recovery degradation even without RESCUE. If the external data budget used by RESCUE across generations were instead allocated to initial training, followed by standard replay, could this achieve comparable final performance?
3 comments
Q2: Table 9 shows ...
We tested this at by matching a total budget of 1,300 recovery demonstrations between upfront seed training and streaming. Over 10 generations at , RESCUE draws from unique external demonstrations, which equals the difference between the R130 seed (1,300) and the default R30 seed (300) in Table 9. The R130 seed matches the budget but not the trajectories, since its additional 1,000 demonstrations come from the scripted controller on scenarios disjoint from (Appendix E). From the R130 seed, we ran RESCUE and uniform replay of the seed's own recovery demonstrations (as " recovery-demo replay" in Table 11).
| Setting | Rec. demos (seed + streamed) | (Δ) | (Δ) | RESCUE − uniform, | RESCUE − uniform, | |
|---|---|---|---|---|---|---|
| R130 seed, arm D, (Table 9) | 1,300 + 0 | 88.1 | 55.9 (−7.5) | 73.9 (+2.6) | – | – |
| R130 seed, uniform replay (new) | 1,300 + 0 | 87.8 | 56.8 (−6.6) | 75.2 (+4.1) | – | – |
| R130 seed, RESCUE (new) | 1,300 + 0 | 88.0 | 61.8 (−1.6) | 76.1 (+5.0) | +0.9 ± 4.8 | +5.0 ± 3.6 |
| R30 seed, streamed uniform replay (Table 2) | 300 + 1,000 | 89.5 | 55.1 (−6.9) | 33.9 (−10.7) | – | – |
| R30 seed, streamed RESCUE (Table 2) | 300 + 1,000 | 89.4 | 59.8 (−2.2) | 43.1 (−1.5) | +9.2 ± 2.3 | +4.7 ± 2.2 |
| R30 seed, arm D, (Table 1) | 300 + 0 | 90.2 | 37.1 (−24.9) | 11.2 (−33.4) | – | – |
| R30 seed, uniform replay of seed demos (Table 11) | 300 + 0 | 89.6 | 53.8 (−8.2) | 32.6 (−12.0) | – | – |
| R30 seed, RESCUE on seed demos (new) | 300 + 0 | 89.4 | 58.6 (−3.4) | 38.9 (−5.4) | +6.0 ± 4.2 | +4.6 ± 3.6 |
On , upfront allocation is clearly preferable: including the demonstrations in seed training gives far higher than streaming them into the R30 seed with either method. A seed above the erosion threshold of Theorem 1(i) does not erode even at , and RESCUE adds little there. RESCUE still helps in two measured cases: on the tail tier of the R130 seed, and in the regime where only the 300 teleoperated demonstrations exist, in line with the small-pool result of Table 13e.
To address the same question on exactly the trajectories that RESCUE streams, we trained a new seed on and followed it with uniform replay.
| Setting (all at ) | Δ vs. streamed RESCUE | ||||
|---|---|---|---|---|---|
| Seed R30 ∪ + uniform replay (new) | 68.9 | 86.8 | 59.3 | 71.2 | +27.4 ± 3.4 |
| Seed R130 + uniform replay (run above) | 71.1 | 87.1 | 59.8 | 74.1 | +30.3 ± 3.0 |
| Seed R30 + streamed uniform replay (Table 2 run) | 44.3 | 88.1 | 57.6 | 37.8 | −6.0 ± 3.0 |
| Seed R30 + streamed RESCUE (Table 2 run) | 44.3 | 88.0 | 60.4 | 43.8 | – |
On the identical trajectory set, upfront allocation remains far ahead (+27.4 ± 3.4 at ), so the advantage stems from higher initial recovery competence rather than from differences in the trajectories. It costs 1–2 points of nominal success and requires all recovery data to be available before . We therefore recommend including recovery data in seed training whenever they are available before deployment, and using RESCUE for data that arrive during the loop and for protecting the tail tier.
To test this last regime, we split into 10 batches of 100, with one batch becoming available per generation (, Chains 1–3, external budget as in Table 2). Batches are drawn either uniformly at random or grouped by task (one task per generation, as when a fleet logs failures on one skill at a time). Both methods draw only from batches that have already arrived. Cells give / / .
| Arrival | Uniform replay | RESCUE | RESCUE − uniform, |
|---|---|---|---|
| Pool at (Table 2) | 89.5 / 55.1 / 33.9 | 89.4 / 59.8 / 43.1 | +9.2 ± 2.3 |
| Random batches | 89.3 / 54.6 / 33.4 | 89.2 / 59.1 / 42.2 | +8.8 ± 3.4 |
| Task-grouped batches | 89.0 / 51.4 / 31.6 | 88.9 / 57.2 / 40.6 | +9.0 ± 3.9 |
With random arrival, both methods lose at most one point relative to the pool available at . With task-grouped arrival, both lose more, since a task whose batch arrives late goes six or more generations without external coverage, but the gap between them remains unchanged, on the tail tier as well. The selection advantage therefore holds when data arrive during the loop, where upfront allocation is not possible.
Thank you for the detailed responses and additional experiments. The rebuttal has improved the empirical soundness of the paper. I maintain my Weak Accept recommendation and have increased my Soundness score.
Thank you for the careful review!
Regarding generality beyond success-filtered BC, PG with an outcome reward and advantage-conditioned cloning also erode under identical recursion (Table 10), and Table 15 replicates the divergence on other architectures and on RoboCasa.
The paper covers recovery-aware training only through arm 6c, which starts rollouts from failure states at the seed's visitation share and recovers about half of arm D's loss (Table 10). We therefore ran Rec-RSI (G10, Chains 1–3) with the same interaction budget (150 rollouts per task per generation) and gradient budget (5,000 updates per generation) as all other arms. It starts 20% of rollouts (30 per task, stratified over L1–L4) from failure states constructed with the LIBERO-Recover pipeline, disjoint from Erec, Drecext and both banks, and trains either with success-filtered BC or with on-policy PG using a binary task-success reward (Table 18).
The divergence narrows but persists. Under Rec-RSI, nominal success still rises while recovery and tail success end well below the seed. L1–L2 are largely retained, but L3–L4 lose more than half of their success, because resets at these levels rarely succeed and are thus filtered out or receive zero reward. This pattern follows from Theorem 1: fixed resets remove visitation drift but not selection, since the policy is supervised only by the recoveries it completes. The recovery weight, which is proportional to 3νˉq3 at a reset share νˉ, therefore still shrinks as recovery competence falls.
RESCUE remains beneficial in this setting. RESCUE alone clearly exceeds Rec-RSI, although the two require different resources: Rec-RSI needs resets but no demonstrations, and RESCUE the reverse. Without external data, RESCUE-0 and Rec-RSI perform on par. Combined, the two give the best result and reach the seed's level. The two approaches are complementary, since Rec-RSI acquires recovery behaviors the seed lacked.