acceptodds
Under review as a conference paper at ICLR 2027

Capability Loss in Embodied Recursive Self-Improvement

Abstract

Recursive self-improvement (RSI) is emerging as a promising method to improve embodied agents with their own experience. However, we argue that, under a fixed task distribution, RSI can erode tail and error-recovery capabilities while nominal success keeps rising, a failure we call silent erosion. We study this failure with a dynamical model over behavior modes and identify two causes of this capability loss: selection, which down-weights the modes a policy performs poorly on, and visitation drift, which removes the post-failure states in which recovery is exercised. We turn this analysis into RESCUE, which spends a fixed external budget on mastered modes with shrinking data mass, retains them across generations, and forecasts degradation from their coverage. Our experiments separate data recursion from parameter drift, intervene on each mechanism, and replicate across policy architectures, benchmarks, and real-robot deployment. Our code will be publicly available.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action models, Recursive self-improvement

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UgoGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper investigates silent erosion in embodied recursive self-improvement (RSI), where nominal task success improves while tail and error-recovery capabilities deteriorate. The authors identify two contributing mechanisms, selection bias and visitation drift, and propose RESCUE, a coverage-aware data selection and retention method designed to preserve previously acquired capabilities. Experiments across multiple VLA architectures and benchmarks, together with qualitative real-robot demonstrations, show that RESCUE can substantially mitigate recovery degradation while maintaining nominal performance.

Critical strengths
  1. Meaningful research problem. The paper identifies an important failure mode in RSI, where improvements in nominal success can mask degradation in tail and recovery capabilities. Controlled experiments provide useful evidence for the roles of selection bias and visitation drift.

  2. Comprehensive experimental evaluation. The authors evaluate multiple VLA architectures and benchmarks, with extensive baseline comparisons and ablation studies. The inclusion of real-robot experiments further supports the practical relevance of the findings.

Critical weaknesses
  1. Limited validation of the generality of the findings. The experiments primarily focus on success-filtered RSI. It remains unclear whether similar degradation would occur when recovery tasks are sufficiently represented during training, or whether RESCUE would provide substantial additional benefits when the initial policy already has strong recovery capabilities.
Questions for authors
  1. If each RSI generation explicitly includes recovery tasks initialized from failure states, with recovery success incorporated into the training objective, would the observed divergence between nominal and recovery success still occur under a matched training budget? How much additional benefit would RESCUE provide in this setting?

  2. Table 9 shows that increasing initial recovery demonstrations (R30 to R130) can prevent recovery degradation even without RESCUE. If the external data budget used by RESCUE across generations were instead allocated to initial training, followed by standard replay, could this achieve comparable final performance?

3 comments

Thank you for the careful review!

W1 & Q1: If each RSI generation explicitly includes recovery tasks initialized from failure states...

Regarding generality beyond success-filtered BC, PG with an outcome reward and advantage-conditioned cloning also erode under identical recursion (Table 10), and Table 15 replicates the divergence on other architectures and on RoboCasa.

The paper covers recovery-aware training only through arm 6c, which starts rollouts from failure states at the seed's visitation share and recovers about half of arm D's loss (Table 10). We therefore ran Rec-RSI (G10G_{10}, Chains 1–3) with the same interaction budget (150 rollouts per task per generation) and gradient budget (5,000 updates per generation) as all other arms. It starts 20% of rollouts (30 per task, stratified over L1L_1–L4L_4) from failure states constructed with the LIBERO-Recover pipeline, disjoint from Erec\mathcal{E}_{\text{rec}}, Drecext\mathcal{D}^{\text{ext}}_{\text{rec}} and both banks, and trains either with success-filtered BC or with on-policy PG using a binary task-success reward (Table 18).

Conditionα\alphaSRnom\mathrm{SR}_{\text{nom}}SRtail\mathrm{SR}_{\text{tail}}SRrec\mathrm{SR}_{\text{rec}}L1L_1–L2L_2L3L_3–L4L_4Δ vs. Rec-RSI (PG)
Seed π0\pi_0 (G0G_0, Table 3d)–81.462.044.662.327.0–
Arm D (Table 3d)090.237.111.218.53.9–
Arm 6c (Table 10; split new)089.549.827.843.212.4–
RESCUE-0 (Table 11; split new)089.255.633.449.617.2–
Rec-RSI, BC filter (new)088.851.332.854.910.7−0.4 ± 5.9
Rec-RSI, PG (new)088.651.933.256.210.2–
Rec-RSI (PG) + RESCUE-0 (new)088.756.238.459.117.8+5.2 ± 4.6
Rec-RSI (PG) + uniform replay (new)0.1088.956.639.658.720.4+6.3 ± 3.8
RESCUE (Table 3d)0.1089.459.843.160.825.5+9.8 ± 3.6
Rec-RSI (PG) + RESCUE (new)0.1088.959.445.062.028.0+11.8 ± 3.1

The divergence narrows but persists. Under Rec-RSI, nominal success still rises while recovery and tail success end well below the seed. L1L_1–L2L_2 are largely retained, but L3L_3–L4L_4 lose more than half of their success, because resets at these levels rarely succeed and are thus filtered out or receive zero reward. This pattern follows from Theorem 1: fixed resets remove visitation drift but not selection, since the policy is supervised only by the recoveries it completes. The recovery weight, which is proportional to 3νˉq33\bar\nu q^3 at a reset share νˉ\bar\nu, therefore still shrinks as recovery competence falls.

RESCUE remains beneficial in this setting. RESCUE alone clearly exceeds Rec-RSI, although the two require different resources: Rec-RSI needs resets but no demonstrations, and RESCUE the reverse. Without external data, RESCUE-0 and Rec-RSI perform on par. Combined, the two give the best result and reach the seed's level. The two approaches are complementary, since Rec-RSI acquires recovery behaviors the seed lacked.

Q2: Table 9 shows ...

We tested this at G10G_{10} by matching a total budget of 1,300 recovery demonstrations between upfront seed training and streaming. Over 10 generations at α=0.10\alpha = 0.10, RESCUE draws from n0=1000n_0 = 1000 unique external demonstrations, which equals the difference between the R130 seed (1,300) and the default R30 seed (300) in Table 9. The R130 seed matches the budget but not the trajectories, since its additional 1,000 demonstrations come from the scripted controller on scenarios disjoint from Drecext\mathcal{D}^{\text{ext}}_{\text{rec}} (Appendix E). From the R130 seed, we ran RESCUE and uniform replay of the seed's own recovery demonstrations (as "G0G_0 recovery-demo replay" in Table 11).

SettingRec. demos (seed + streamed)SRnom\mathrm{SR}_{\text{nom}}SRtail\mathrm{SR}_{\text{tail}} (Δ)SRrec\mathrm{SR}_{\text{rec}} (Δ)RESCUE − uniform, SRrec\mathrm{SR}_{\text{rec}}RESCUE − uniform, SRtail\mathrm{SR}_{\text{tail}}
R130 seed, arm D, α=0\alpha = 0 (Table 9)1,300 + 088.155.9 (−7.5)73.9 (+2.6)––
R130 seed, uniform replay (new)1,300 + 087.856.8 (−6.6)75.2 (+4.1)––
R130 seed, RESCUE (new)1,300 + 088.061.8 (−1.6)76.1 (+5.0)+0.9 ± 4.8+5.0 ± 3.6
R30 seed, streamed uniform replay (Table 2)300 + 1,00089.555.1 (−6.9)33.9 (−10.7)––
R30 seed, streamed RESCUE (Table 2)300 + 1,00089.459.8 (−2.2)43.1 (−1.5)+9.2 ± 2.3+4.7 ± 2.2
R30 seed, arm D, α=0\alpha = 0 (Table 1)300 + 090.237.1 (−24.9)11.2 (−33.4)––
R30 seed, uniform replay of seed demos (Table 11)300 + 089.653.8 (−8.2)32.6 (−12.0)––
R30 seed, RESCUE on seed demos (new)300 + 089.458.6 (−3.4)38.9 (−5.4)+6.0 ± 4.2+4.6 ± 3.6

On SRrecSR_{rec}, upfront allocation is clearly preferable: including the demonstrations in seed training gives far higher SRrec\mathrm{SR}_{\text{rec}} than streaming them into the R30 seed with either method. A seed above the erosion threshold of Theorem 1(i) does not erode even at α=0\alpha = 0, and RESCUE adds little there. RESCUE still helps in two measured cases: on the tail tier of the R130 seed, and in the regime where only the 300 teleoperated demonstrations exist, in line with the small-pool result of Table 13e.

To address the same question on exactly the trajectories that RESCUE streams, we trained a new seed on R30∪Drecext\text{R30} \cup \mathcal{D}^{\text{ext}}_{\text{rec}} and followed it with uniform replay.

Setting (all at G5G_5)G0G_0 SRrec\mathrm{SR}_{\text{rec}}SRnom\mathrm{SR}_{\text{nom}}SRtail\mathrm{SR}_{\text{tail}}SRrec\mathrm{SR}_{\text{rec}}Δ vs. streamed RESCUE
Seed R30 ∪ Drecext\mathcal{D}^{\text{ext}}_{\text{rec}} + uniform replay (new)68.986.859.371.2+27.4 ± 3.4
Seed R130 + uniform replay (run above)71.187.159.874.1+30.3 ± 3.0
Seed R30 + streamed uniform replay (Table 2 run)44.388.157.637.8−6.0 ± 3.0
Seed R30 + streamed RESCUE (Table 2 run)44.388.060.443.8–

On the identical trajectory set, upfront allocation remains far ahead (+27.4 ± 3.4 at G5G_5), so the advantage stems from higher initial recovery competence rather than from differences in the trajectories. It costs 1–2 points of nominal success and requires all recovery data to be available before G0G_0. We therefore recommend including recovery data in seed training whenever they are available before deployment, and using RESCUE for data that arrive during the loop and for protecting the tail tier.

To test this last regime, we split DrecextD^{ext}_{rec} into 10 batches of 100, with one batch becoming available per generation (G10G_{10}, Chains 1–3, external budget as in Table 2). Batches are drawn either uniformly at random or grouped by task (one task per generation, as when a fleet logs failures on one skill at a time). Both methods draw only from batches that have already arrived. Cells give SRnom\mathrm{SR}_{\text{nom}} / SRtail\mathrm{SR}_{\text{tail}} / SRrec\mathrm{SR}_{\text{rec}}.

ArrivalUniform replayRESCUERESCUE − uniform, SRrec\mathrm{SR}_{\text{rec}}
Pool at G0G_0 (Table 2)89.5 / 55.1 / 33.989.4 / 59.8 / 43.1+9.2 ± 2.3
Random batches89.3 / 54.6 / 33.489.2 / 59.1 / 42.2+8.8 ± 3.4
Task-grouped batches89.0 / 51.4 / 31.688.9 / 57.2 / 40.6+9.0 ± 3.9

With random arrival, both methods lose at most one point relative to the pool available at G0G_0. With task-grouped arrival, both lose more, since a task whose batch arrives late goes six or more generations without external coverage, but the gap between them remains unchanged, on the tail tier as well. The selection advantage therefore holds when data arrive during the loop, where upfront allocation is not possible.

Thank you for the detailed responses and additional experiments. The rebuttal has improved the empirical soundness of the paper. I maintain my Weak Accept recommendation and have increased my Soundness score.

Reviewer_6TZKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates capability degradation during recursive self-improvement (RSI) of embodied agents. The authors observe that repeatedly collecting self-generated trajectories and training on selected successful experiences can improve nominal task success while gradually reducing performance on tail conditions and error-recovery tasks. They refer to this phenomenon as silent erosion.

To explain this behavior, the paper introduces a simplified dynamical model that identifies two contributing factors: selection bias, which reduces the training weight of less successful behavior modes, and visitation drift, which reduces exposure to recovery states as the policy becomes more reliable. Based on this analysis, the authors propose RESCUE, a coverage-aware data selection and retention strategy that prioritizes previously mastered but increasingly underrepresented behaviors. Experiments with π₀.₅, OpenVLA-OFT, and Diffusion Policy across LIBERO, RoboCasa, and real-robot tasks demonstrate the effectiveness of the proposed

Critical strengths
  1. The paper addresses an interesting and practically relevant problem. As embodied agents increasingly rely on self-generated experience for iterative improvement, maintaining previously acquired capabilities becomes important. The observation that nominal task success can improve while recovery performance deteriorates is meaningful, and it highlights a limitation of evaluating policies mainly through average task success rates.

  2. The experimental analysis is fairly thorough. I appreciate the effort to distinguish the effects of recursive data generation from parameter drift through controlled experiments. The additional interventions involving success filtering, recovery-state visitation, and targeted data injection also provide useful evidence for the proposed explanation. Overall, the experiments are well organized around the research questions.

  3. RESCUE shows promising empirical results. The proposed strategy effectively preserves recovery performance while maintaining most of the improvement in nominal task success. The comparisons with replay-based methods and other baselines are useful, and the additional evaluations across different policy architectures and simulation environments help support the findings.

Critical weaknesses
  1. The novelty of silent erosion relative to existing studies of forgetting and distribution shift could be better clarified.

The observed phenomenon seems closely related to catastrophic forgetting, recursive model collapse, and policy-induced distribution shift. In particular, the gradual loss of underrepresented behaviors during repeated training on selected data is conceptually similar to problems discussed in previous work. The paper provides an interesting embodied perspective by separating selection and visitation drift, but it would be helpful to explain more clearly what new understanding this analysis offers beyond existing formulations.

Relatedly, the RSI setting studied here mainly involves iterative training on self-generated, success-filtered trajectories, which is closely related to conventional self-training and policy optimization. A clearer discussion of the scope of the findings would help position the contribution.

  1. The interpretation of recovery capability could be further investigated.

The paper evaluates recovery capability through task success when starting from post-failure states. However, in VLA policies, successful recovery does not necessarily imply that the model has learned a dedicated recovery mechanism. For example, after dropping an object, a policy may simply generate a new grasping action based on the current observation, without explicitly recognizing or reasoning about the failure.

From this perspective, the observed degradation might also be related to overfitting to frequently selected successful trajectories or reduced generalization of the observation-to-action mapping. Although the data injection experiments support the importance of behavior coverage, they do not fully distinguish these possible explanations. Additional analysis of how policies respond to changes in visual observations or object states could help clarify the underlying mechanism.

  1. The connection between the theoretical model and practical VLA policies could be strengthened.

The theoretical analysis is interesting and provides an intuitive explanation for capability degradation. However, it relies on a simplified MDP and a logistic policy with shared features, while real VLA models involve high-dimensional observations, continuous actions, and more complex optimization dynamics.

In particular, the erosion condition in Theorem 1 depends on per-decision nominal competence, recovery competence, and feature-gradient overlap. In the experiments, these quantities are approximated using episode-level success rates and measured gradient statistics. The results mainly support the predicted trends rather than directly validating the theoretical threshold. A more detailed discussion of these approximations, or additional quantitative validation, would make the theoretical contribution more convincing.

  1. The applicability of RESCUE to broader self-improvement settings remains somewhat unclear.

RESCUE focuses on preserving previously mastered behaviors whose coverage decreases over successive generations. This is reasonable for capability retention, but practical self-improving agents may also need to acquire new skills and learn to handle previously unseen failures.

An alternative would be to explicitly incorporate recovery tasks into the self-improvement objective, allowing the policy to improve its recovery capability through targeted exploration or reinforcement learning. While the paper includes comparisons with several correction and replay strategies, the relationship between RESCUE and this type of recovery-aware training could be discussed further.

In addition, the behavior-mode annotations used in the experiments rely on privileged simulator information. It would be useful to better understand how the approach could scale to real-world settings where such annotations may not be readily available.

Questions for authors
  1. How do the authors distinguish the RSI setting considered in this paper from conventional iterative self-training or on-policy policy optimization? To what extent are the identified degradation mechanisms specific to RSI rather than more general consequences of changing training distributions?

  2. Could the observed recovery degradation partly reflect reduced observation-conditioned action generalization rather than the loss of a dedicated recovery capability? For example, would it be possible to evaluate how policies from different generations respond to controlled changes in object positions or failure states?

  3. How accurately do the empirical estimates of (u), (q), and (c\rho) correspond to the quantities in the theoretical model? Is it possible to provide a more direct validation of the predicted erosion threshold in practical VLA policies?

  4. How sensitive is RESCUE to the definition and granularity of behavior modes? Since the current experiments use privileged simulator states for mode annotation, how might the method be applied when such information is unavailable?

  5. Have the authors considered explicitly including recovery tasks as part of the RSI training objective? It would be interesting to understand how RESCUE compares with targeted recovery training under similar interaction and training budgets, particularly when the goal includes learning recovery behaviors that the initial policy has not yet mastered.

6 comments

Thank you for the constructive review!

W1 & Q1: The novelty of silent erosion relative to existing studies...

We note that we already discuss the relation between our work and related studies in Section 5, Section 2.3 and Appendix B, and we summarize it here.

In our analysis, a capability loses training data through two channels. Selection favors the rollouts on which the policy succeeds. Visitation drift arises because recovery states are reached only through the policy's own failures, so an improving policy reaches them less often (Proposition 2). These two channels distinguish silent erosion from existing formulations (Appendix B). In catastrophic forgetting, the distribution shift is imposed by an external task sequence; here the task distribution is fixed and the shift is produced by the policy itself. Accordingly, training on data from the fixed seed costs little recovery success, whereas training on the policy's own selected data costs most of it (Table 1). Model collapse attributes degradation to self-generated data as such, yet unfiltered self-cloning stays close to the seed (Table 10), and arm D and RESCUE concentrate their action distributions to a similar degree while differing widely in recovery success (Table 4). Performative prediction and data-feedback loops (Perdomo et al., 2020; Taori & Hashimoto, 2023) study data distributions that depend on the deployed model; in the embodied loop, the induced distribution is a state occupancy, and the quantity at risk is a fixed set of behaviors rather than predictive accuracy. Selection is shared with self-training and on-policy RL (Proposition 4), and the PG variant indeed erodes like arm D. Visitation drift is not shared: it requires a policy that determines its own training states, as in DAgger without the expert (Section 5), and it is absent from single-turn self-training on a fixed prompt set such as STaR.

Regarding scope, the erosion does not follow from a changing training distribution as such. Theorem 1 requires three conditions: the update must favor the more successful rollouts, nominal and recovery behaviors must share parameters with conflicting gradients (c>0c > 0), and the data weights must depend on the policy's own outcomes. Where these conditions are absent, we observe no erosion: the fixed-teacher arms A and B and unfiltered self-cloning all remain close to the seed. Conversely, changing only the selection rule on a single rollout pool changes recovery success several-fold (arms 1–3, Table 10). Selection therefore applies to any loop with these properties, including iterative self-training and on-policy RL with outcome rewards, whereas visitation drift is specific to policies that generate their own states. RSI satisfies the first and third conditions by construction; the second is empirical, and the measured cc is positive at every generation (Table 7).

W2 & Q2: The interpretation of recovery capability could be further investigated...

We want to clarify that we do not assume a dedicated recovery mechanism. Recovery competence is defined as q=π(arec∣s)q = \pi(a_{\text{rec}} \mid s) for s∈Srecs \in \mathcal{S}_{\text{rec}} (L.143–145), and π0.5\pi_{0.5} conditions only on the current images, proprioception and instruction (Table 20). The question is therefore whether the policy loses this observation-to-action mapping on off-nominal observations in general, or only in the states whose training data the loop removes. The existing results support the latter.

  • Unfiltered self-cloning (U) keeps recovery near the seed (Table 10), so training on the policy's own trajectories is not harmful by itself.
  • Arm C is reset to the seed at every generation and still erodes (Table 1), and halving or doubling the number of updates per generation changes D−B by less than its confidence interval (Table 5). Both observations are difficult to reconcile with parameter overfitting.
  • At an equal 10% budget, only restoring the lost recovery modes recovers performance, whereas random or nominal-matched modes do not (arms 5a–5c, Table 10).
  • RESCUE's action distribution narrows about as much as arm D's, yet RESCUE retains its recovery (Table 4).

Following the reviewer's suggestion, we additionally evaluated existing checkpoints under controlled changes of object state (three family-A tasks, 5 chains, 50 paired episodes per condition). At the initial state, the object is shifted by 1–8 cm either into a behavior mode that is still covered at G10G_{10} (P1) or, by the same distances, into a mode whose data share has shrunk (P2). During the episode, the object is pushed before contact (P3) or dropped after the grasp (P4), or the target is moved (P5). The nominal-continuation index (NCI) measures the similarity between the actions after a perturbation and those at the same step of an unperturbed run; a high value indicates that the policy ignores the change.

ConditionCoverage at G10G_{10}SeedArm A G10G_{10}Arm D G5G_5Arm D G10G_{10}RESCUE G10G_{10}NCI, arm D (G0G_0 → G5G_5 → G10G_{10})
P1 initial shiftcovered84.083.286.485.685.20.41 → 0.43 → 0.44
P2 initial shiftshrinking61.260.441.627.654.80.44 → 0.61 → 0.73
P3 push before contactcovered72.471.673.674.072.80.38 → 0.40 → 0.41
P4 drop after graspSrec\mathcal{S}_{\text{rec}}52.851.226.413.650.40.47 → 0.68 → 0.81
P5 target movedSrec\mathcal{S}_{\text{rec}}46.045.625.214.843.20.52 → 0.70 → 0.79

Under arm D, the covered conditions P1 and P3 remain at the seed's level, whereas P2, P4 and P5 lose more than half of their success, and the NCI rises only in these three conditions. Since P1 and P2 use identical displacements, the magnitude of the visual change does not explain the drop. RESCUE stays close to the seed in every condition.

W3 & Q3: How accurately do the empirical estimates of uu, qq, and cρc\rho correspond...

We agree that the experiments test signs, orderings and scaling rather than the location of the threshold (L.210–211). The tests were designed so as not to depend on exact per-decision estimates (Remark 2, Appendix E).

  • The rate test matches seeds in recovery competence and varies their nominal error (Table 9). Across these seeds, the per-generation logit loss scales with ϵn\epsilon_n (slope 1.12±0.311.12 \pm 0.31, intercept −0.018±0.045-0.018 \pm 0.045; Section 4.4), as the prefactor pϵn/SRp\epsilon_n/\mathrm{SR} of Eq. 4 implies. The bracket ucρ−huc\rho - h of Eq. 4, however, is not constant across these seeds: under the estimates in the table below, it grows about 2.6-fold from N10/R30 to N50/R30, opposite to the direction of the observed rates. The statement in Appendix A.3 that ucρuc\rho differs by under 20% therefore does not cover the full factor. We regard the ϵn\epsilon_n scaling as an empirical regularity consistent with the prefactor rather than a quantitative prediction of Eq. 4, and will revise Section 4.4 and Appendix A.3 accordingly.
  • The sign test holds under any monotone map from episode-level to per-decision competence. The recovery-strong seed N10/R130 does not erode while all others do, and N50/R10, which is as weak in recovery as N10/R10 but has a fifth of its nominal error, erodes far more slowly (Table 9).
  • The test of Theorem 1(iii) uses none of uu, qq or cρc\rho: protection tracks the effective recovery mass far more closely than the external fraction (Section 4.5). RESCUE does not use the threshold either.

We evaluate the margin ucρ−hκ^(q^)uc\rho - h_{\hat\kappa}(\hat q) per seed and add seeds R50, R70 and R90 at N10 (new, 3 chains, G0G_0–G5G_5, arm-D recursion). Here ρ\rho is the measured ratio of the two loss-gradient norms (Appendix E), corrected for the residual factors (ϵn−yn)(\epsilon_n - y_n) and (1−q)(1-q) of Lemma 1(b), since the theorem's ρ\rho is a feature-norm ratio. Episode success is mapped to per-decision competence by q^=SRrec1/κ^\hat q = \mathrm{SR}_{\text{rec}}^{1/\hat\kappa} with κ^=3.4\hat\kappa = 3.4, the value fitted to the monitor's per-segment replan counts (Appendix D). As Appendix A.3 notes, individual episodes do not fix this number, so we treat κ^\hat\kappa as a calibration of the map rather than a literal decision count; the sign conclusions also hold at the paper's κ=3\kappa = 3. The signs of the new seeds were fixed before the runs from the cρ∈(0.70,0.96)c\rho \in (0.70, 0.96) interval at κ=3\kappa = 3, which places the crossing at a G0G_0 SRrec\mathrm{SR}_{\text{rec}} of 53.7 [48.2, 59.5] for N10 seeds; the fitted map gives 53.8 [48.1, 59.8]. Under either map, R50 lies inside the band, and R70 and R90 are predicted not to erode. In the table, Δℓrec\Delta\ell_{\text{rec}} is the observed per-generation logit change over G0G_0–G5G_5.

SeeduuSRrec\mathrm{SR}_{\text{rec}} (G0G_0)q^\hat qccρ\rhoucρ−hκ^(q^)uc\rho - h_{\hat\kappa}(\hat q) [95% CI]Obs. Δℓrec\Delta\ell_{\text{rec}} [95% CI]
N10/R100.80224.80.6640.322.490.261 [0.212, 0.310]−0.251
N10/R300.81444.60.7890.312.550.064 [0.020, 0.108]−0.204
N10/R50 (new)0.83050.10.8160.292.790.044 [−0.012, 0.100]−0.035 [−0.062, −0.008]
N10/R70 (new)0.82460.80.8640.312.61−0.052 [−0.105, +0.001]+0.011 [−0.012, +0.034]
N10/R90 (new)0.81865.90.8850.312.62−0.095 [−0.146, −0.044]+0.019 [−0.004, +0.042]
N10/R1300.82671.30.9050.312.58−0.139 [−0.188, −0.090]+0.024 [−0.010, +0.058]
N25/R300.90946.20.7970.302.620.121 [0.074, 0.168]−0.121
N50/R300.96243.10.7810.282.710.165 [0.112, 0.218]−0.039
N50/R100.96025.30.6670.292.680.364 [0.306, 0.422]−0.052

The sign of the margin agrees with the observed sign for all nine seeds at the point estimate. Near the threshold, the test is not resolved: the margin intervals of R50 and R70 include zero, as do the observed intervals of R70, R90 and R130. The measured cρc\rho is 0.79 [0.75, 0.83], inside the interval. On the fitted q^\hat q scale, the margin vanishes at q^⋆=0.83\hat q^\star = 0.83 [0.81, 0.86] over the measured N10 ucρuc\rho range, and the empirical zero crossing is q^⋆=0.853\hat q^\star = 0.853 [0.820, 0.886]. We claim agreement in sign only. Magnitudes do not follow the margin: R30 and R10 erode at similar rates although their margins differ about fourfold, so beyond the ϵn\epsilon_n scaling discussed above we make no claim about the amplitude of Eq. 4.

Reviewer_ReBeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper rises an option that the current RSI might not work, particularly, when robot policies are retrained on their own successful rollouts, nominal success keeps rising while recovery skills quietly decline, a failure the authors call "silent erosion". The authors trace it to two causes: success filtering drops the behaviors the policy performs poorly, and a stronger policy reaches failure states less often, so recovery gets less practice. They then propose RESCUE, which spends a small external data budget on mastered behaviors that are losing coverage. RESCUE keeps recovery close to the seed level and retains most of the nominal gain.

Critical strengths
  • Interesting problem. The paper identifies "silent erosion" in embodied recursive self-improvement, where nominal success rises while recovery skills steadily decline. This matters for the collect–filter–retrain loops many robot systems now use.

  • Clear theory and methods. The authors split data loss into selection and visitation drift. From a simple model, they derive testable predictions: an erosion threshold, an erosion rate that scales with nominal error, and the finding that external data helps through its composition rather than its mixing ratio.

  • Helpful experiments. A 2×2 teacher × initialization design, frozen state banks, and targeted interventions show that the erosion comes from recursively selected data (−29.1 points) rather than parameter drift (−4.6 points).

Critical weaknesses
  • If an external recovery pool already always exists. RESCUE decides which demonstrations to select from the pool and how to retain them across generations, but it does not address where recovery demonstrations come from. Part of the pool is generated by a scripted controller that uses privileged simulator state, which could be costly to reproduce in real deployments. so, how would you obtain recovery demonstrations on a real robot, and how much does RESCUE's advantage depend on the size and quality of this pool? A discussion on this part might be helpful.

  • Heavily rely on simulators. Behavior modes come from "geometric rules over privileged simulator state," and recovery tags require detecting failure states and their levels. The paper does not explain how these annotations are obtained on the real robot. Does it still use privileged state, if so, how?

  • Results on real-robots are limited. Could the authors report quantitative real-robot results, if possible?

Questions for authors

See weakness

3 comments

Thank you for the detailed review!

W1: If an external recovery pool...

We note that with no external data, RESCUE-0 (coverage-balanced filter plus retention, α=0\alpha = 0) already matches uniform replay of the full 1000-demonstration pool (Table 11). When a pool is available, RESCUE's margin over uniform replay is similar at every pool size, and 25 demonstrations per task — fewer than the 30 used for the seed — suffice to match uniform replay of the full pool (Table 13e). What matters is coverage of masteredat-risk modes: uniform replay of the full pool reaches a recovery weight of β=0.3\beta = 0.3, above the self-generated ≈0.22\approx 0.22 at G0G_0 (κm0=3×7.4%\kappa m_0 = 3 \times 7.4\%), yet still erodes, while RESCUE at the same β\beta stays near the seed.

Existing arms further indicate that the data need not come from privileged state. Patching back the seed's own successful G0G_0 rollouts through the lost recovery modes restores most of the loss in closed loop (arm 5b-c, Table 10), although this arm uses hindsight about which modes were lost. Corrections collected only on failure inherit visitation drift, since their share shrinks as the policy improves, and they perform worse than uniform replay (Table 2). For real-robot deployment, recovery demonstrations can be obtained from three sources, in increasing order of cost: the loop's own successful recoveries from early generations; a small teleoperated set of about 25 demonstrations per task; and rollouts started at a fixed share from operator-staged post-failure states, which fixes recovery visitation as arm 6 does in simulation.

We ran new experiments (α=0.10\alpha = 0.10, G10G_{10}, Chains 1–3) to test whether the gain persists without the scripted data and on pools that require no external collection.

Poolnuniqn_{\text{uniq}}Uniform replayRESCUERESCUE − Uniform
Full pool (teleoperated + scripted; Table 2)100033.943.1+9.2 ± 2.3
Teleoperated subset34030.839.4+8.7 ± 4.1
Scripted subset, size-matched34029.636.2+6.7 ± 4.4
Seed's own teleoperated demonstrations30032.938.9+6.0 ± 4.2
Self-sourced (successful recoveries in the loop's own G0G_0–G2G_2 rollouts)21833.138.6+5.4 ± 3.9
Full pool, subsampled10018.726.9+8.2 ± 4.6
Full pool, subsampled5015.318.4+3.1 ± 4.9

RESCUE's advantage persists when the scripted demonstrations are removed. It also remains clear on the two pools that require no external collection, namely the seed's own 300 demonstrations and the self-sourced recoveries, on which RESCUE exceeds both RESCUE-0 and uniform replay of the full pool. The advantage becomes indistinguishable from zero only at nuniq=50n_{\text{uniq}} = 50, where too few at-risk episodes remain to select from.

W2: Heavily rely on simulators...

We note that no privileged state is used on the real robot. In simulation, privileged state enters RESCUE through the geometric mode labels Φ\Phi (36 per task, Appendix E), the mastery factor q0(z)q_0(z) of Eq. 6 (calibrated on the bank Hcal\mathcal{H}_{\text{cal}}) and the recovery tag. Selection and retention read only Φ\Phi, binary episode success and q0(z)q_0(z); the binary tag enters the monitor (νg\nu_g, weffw^{\text{eff}}) and the forecast of Eq. 9; and the L1L_1–L4L_4 level is used only to stratify evaluation. Each of these inputs has an unprivileged substitute.

  • Mode labels: per-task K-means fitted at G0G_0 on end-effector proprioception and action chunks and then frozen. At G10G_{10}, this dictionary costs a few points of SRrec\mathrm{SR}_{\text{rec}} relative to privileged RESCUE and remains clearly above uniform replay (Table 11), and K=16K = 16 performs similarly to K=36K = 36 (new, −4.1 ± 3.8 relative to privileged RESCUE on Chains 1–3).
  • q0(z)q_0(z): per-mode success in the seed's G0G_0 rollouts, which has a Spearman correlation of 0.74 with the Hcal\mathcal{H}_{\text{cal}} estimate across recovery modes visited in both.
  • Recovery tag: a failure-event detector trained only on the seed's teleoperated recovery and nominal demonstrations. Against the simulator tag, it reaches a precision of 0.91 and a recall of 0.87 on the logs of arms D and RESCUE, and it increases the one-step SRrec\mathrm{SR}_{\text{rec}} forecast error on family B by 0.3 points.

We evaluated these substitutions in closed loop (α=0.10\alpha = 0.10, Chains 1–3) reaching G5G_5.

VariantSRnom\mathrm{SR}_{\text{nom}}SRtail\mathrm{SR}_{\text{tail}}SRrec\mathrm{SR}_{\text{rec}}Δ vs. default
Uniform replay (Table 2 run)88.157.637.8−6.0 ± 3.0
Default RESCUE, geometric Φ\Phi and q0q_0 from Hcal\mathcal{H}_{\text{cal}} (Table 2 run)88.060.443.8–
K-means Φ\Phi (K=36K = 36), q0q_0 from Hcal\mathcal{H}_{\text{cal}} (Table 11 run)88.158.841.2−2.6 ± 2.4
K-means Φ\Phi, q0q_0 from G0G_0 rollouts (new)88.058.240.1−3.7 ± 3.6
Same, teleoperated subset of the pool (new)88.157.939.2−4.6 ± 3.8

Without simulator state in any method component, RESCUE lies between uniform replay and the privileged version, and additionally removing the scripted demonstrations changes the result only slightly.

The real-robot deployment used this unprivileged setup, without motion capture or object-state tracking. It used K-means modes (K=8K = 8 per task, given the smaller robot dataset), q0q_0 from the seed's G0G_0 rollouts, the detector's binary tag, and 20 teleoperated recovery demonstrations per task as the external pool.

W3: Results on real-robots are limited...

We evaluated the seed (G0G_0) and the G5G_5 checkpoints of arm D, uniform replay and RESCUE (both at α=0.10\alpha = 0.10) on real-robot tasks, with 20 nominal and 20 post-failure trials per task and policy (new, 320 trials). Post-failure starts are staged by the operator as a dropped or displaced apple outside the gripper, or as a missed grasp or shifted crumpled paper.

PolicyApple nominalApple recoveryPaper nominalPaper recoveryPooled SRnom\mathrm{SR}_{\text{nom}}Pooled SRrec\mathrm{SR}_{\text{rec}}
Seed π0\pi_0 (G0G_0)75.0 (15)45.0 (9)65.0 (13)35.0 (7)70.0 (28/40)40.0 (16/40)
Arm D (G5G_5)90.0 (18)15.0 (3)80.0 (16)10.0 (2)85.0 (34/40)12.5 (5/40)
Uniform replay (G5G_5) (new)85.0 (17)30.0 (6)80.0 (16)20.0 (4)82.5 (33/40)25.0 (10/40)
RESCUE (G5G_5)85.0 (17)45.0 (9)80.0 (16)30.0 (6)82.5 (33/40)37.5 (15/40)

Pooled over both tasks, arm D's recovery falls significantly below the seed's, whereas RESCUE stays near the seed and significantly above arm D. Nominal success increases in point estimate for both. Uniform replay separates the effect of the external demonstrations from that of RESCUE's selection: it lies between arm D and RESCUE, the same ordering as in simulation (Table 2).

Thank you for the rebuttal. I would hold my positive rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.