acceptodds
Under review as a conference paper at ICLR 2027

RECALL: Recovery Experience Collection for Active Lifelong Learning in Vision-Language-Action Models

Abstract

Vision-Language-Action (VLA) models are typically trained offline on large demonstration datasets and updated post-deployment through additional fine-tuning. A more capable system should instead improve continually, actively identifying states in which it should collect new demonstrations and without forgetting previously learned behaviors. We present RECALL, the first empirical study of active, lifelong learning in VLAs, investigating two coupled questions: where to collect new supervision and how to incorporate it into a pretrained policy. We evaluate a VLA's ability to retain prior behaviors while adapting to new data, comparing start-state demonstrations against policy-induced recovery states, uncertainty-guided versus non-uncertainty-based state selection, sparse versus dense recovery collection, and continual adaptation strategies including reduced learning rates, elastic weight consolidation, and replay. Key findings include: recovery-state supervision is often more useful than repeated start-state demonstrations, though this advantage does not require uncertainty-based selection; more recovery data is not always better; targeted recovery data alone can cause severe forgetting; and preserving prior behavior requires maintaining appropriate behavioral coverage. We further demonstrate uncertainty-triggered recovery collection as a pre-execution intervention on a physical robot. Together, these findings establish an empirical foundation for active lifelong learning in VLAs and offer practical guidance for building robot policies that improve continuously from deployment experience.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Active Learning, Continual Learning, Uncertainty Quantification, Robot Learning, Imitation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oFK6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RECALL studies how a pretrained VLA can learn from human recovery demonstrations. Using pi0-FAST on LIBERO, the authors compare where demonstrations begin, how many recoveries to collect, and how to update the policy while retaining prior skills. Intermediate-state recoveries give higher average success than start-state demonstrations, with no clear advantage for uncertainty-based selection or denser collection. Training only on recoveries can hurt both collected and retained tasks; the runs with replay perform better. On an xArm-7, recovery and start-state collection with 30 demonstrations each yield 71.7% and 53.3% success over 60 matched scene pairs.

Critical strengths
  • I found the random intermediate-state controls particularly useful. They obtain similar observed success rates to uncertainty-triggered collection, showing why the recovery gains should not be credited to the uncertainty predictor alone. Reporting how many candidate states were needed to obtain the demonstrations also gives a more useful picture of collection effort.

  • The real-robot comparison is a strong evidence in the paper. Both policies use 30 demonstrations and matched training settings, with fixed checkpoints and randomized policy order across 60 paired scenes. The 18.3-percentage-point gain supports the value of collecting recoveries on hardware.

Critical weaknesses
  • Each simulation condition has one training run and one update cycle, evaluated on five collected and five retained tasks. Checkpoint averages do not show whether the findings repeat across seeds, and no RQ1/RQ2 comparison is significant after the stated Holm correction. The authors acknowledge these limits. The results are useful observations, but retention through repeated updates remains untested.

  • Replay and regularization are not always compared under the same normalization. Changing replay scope also changes dataset size and the balance of old and new samples. This makes it hard to tell how much of the retention difference comes from behavioral coverage. The normalization checks only partly address this concern.

Questions for authors
  1. What are the replay dataset sizes and effective old/new sampling ratios for the three replay scopes? These would help assess the coverage comparison.
  2. If additional experiments are available, do the recovery-versus-start and first-versus-all trends repeat across independent collection/training seeds under the same checkpoint protocol?
2 comments

We thank the reviewer for the constructive assessment and for highlighting the usefulness of the random intermediate-state controls and matched physical-robot evaluation. We have completed additional fine-tuning runs addressing Q2 and further examined the replay-data accounting raised in Q1.

Q2: Do the RQ1/RQ2 trends repeat across independent seeds?

Yes, the principal trends repeat across training runs. We added two runs per condition, yielding three runs for each of the five RQ1/RQ2 configurations. All runs use the same demonstration datasets, base checkpoint, training schedule, and ten-checkpoint evaluation protocol.

For RQ1, each run contributes one ten-checkpoint average. The table reports the mean and sample SD across the three runs.

MethodOverall success (mean ± SD)Collected-task success (mean ± SD)Retained-task success (mean ± SD)
Start57.03 ± 0.69%26.83 ± 0.59%87.23 ± 1.00%
INSIGHT-First66.93 ± 1.09%45.63 ± 2.05%88.23 ± 0.62%
Random-All67.63 ± 0.55%48.20 ± 0.70%87.07 ± 1.01%
Random-Success67.78 ± 0.21%47.61 ± 0.46%87.95 ± 0.30%

All three intermediate-state methods outperformed Start in every run. INSIGHT-First exceeded Start by +8.92, +10.42, and +10.36 percentage points overall (mean +9.90). The gains were concentrated on collected tasks, and retained-task success stayed around 87–88% across the four conditions.

For RQ2, INSIGHT-First averaged 66.93 ± 1.09% overall versus 63.62 ± 2.56% for INSIGHT-All. First led in all three runs by +3.58, +6.22, and +0.12 percentage points. The collected-task means were similar (45.63% versus 44.71%), and First had higher retained-task success (88.23% versus 82.53%). Collecting 3.87× as many demonstrations yielded a lower observed overall average in each run.

With three paired training runs, the minimum attainable two-sided exact sign-flip p-value is 0.25. No contrast was significant after Holm correction. These results describe training-run repeatability with the demonstration datasets held fixed.

Q1: Replay dataset sizes and sampling ratios

Full, LIBERO-10, and Targeted replay draw from 1,693, 379, and 180 prior demonstrations across 40, 10, and five tasks, respectively. All three use the same 149 recovery demonstrations. LIBERO-10 replay performed similarly to Full replay with less than a quarter as many prior demonstrations. This suggests that which tasks are represented may matter more than total demonstration count in this setting.

Task coverage and dataset size vary together in RQ5. Demonstration counts alone do not determine how often prior and recovery examples are sampled during training. We are running a follow-up with 180 prior demonstrations per condition, preserving representation of all 40, 10, or five tasks and holding the recovery data and old/new sampling ratio fixed. We will report the results when the runs are complete.

Thanks for the additional results. The new runs help address my concern about repeatability, and the replay breakdown clarifies the setup. The role of replay and retention over repeated updates remain less clear. I’m keeping my score.

Reviewer_6VYPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an empirical study of active lifelong learning for VLA models. Using FAST as the base policy, the authors investigate six research questions concerning recovery-state data collection, uncertainty-guided intervention, demonstration density, catastrophic forgetting, conservative fine-tuning, and replay strategies.

The experiments suggest that collecting demonstrations from policy-induced intermediate states is more effective than collecting additional demonstrations from standard task starts. However, random intermediate-state selection achieves comparable performance to uncertainty-guided selection. The study also shows that recovery-only fine-tuning can cause severe forgetting, while replay helps preserve previously learned behaviors. Finally, real-robot experiments demonstrate an 18.3 percentage-point improvement over a standard-start demonstration baseline.

Critical strengths
  1. Active data collection and continual adaptation are important challenges for deploying general-purpose robotic policies. Studying these two aspects jointly is well motivated.

  2. The paper examines six well-structured research questions and includes meaningful controls, particularly random intermediate-state selection and different replay configurations. The separation between adaptation and retention performance is also valuable.

  3. The physical-robot experiments demonstrate a substantial improvement over standard-start demonstrations (71.7% vs. 53.3%), supported by a paired statistical test. This provides practical evidence for the value of recovery-state supervision.

Critical weaknesses
  1. The main findings largely reiterate well-established principles from interactive imitation learning and continual learning: collecting supervision from real deployments mitigates distribution shift, narrowly distributed fine-tuning data causes forgetting, and replay preserves previously learned behaviors. These observations are hardly surprising given the extensive literature on DAgger and experience replay.

  2. Despite the framing of "active lifelong learning", the experiments involve only a single collection-and-adaptation cycle. Moreover, uncertainty-guided selection does not outperform random intermediate-state sampling, leaving the specific benefit of the proposed active selection mechanism unclear. The paper therefore provides limited evidence for either the "active" or "lifelong" aspect of its claimed contribution.

  3. The simulation experiments are restricted to one VLA architecture and LIBERO-10, while the real-robot experiments cover only three tasks.

  4. The 37-page submission provides an extraordinary amount of procedural detail, repetitive discussion, and incremental observations, yet the underlying technical content is remarkably limited. The manuscript reads more like an exhaustively expanded experimental report than a focused research paper. Extensive documentation is appreciated, but excessive elaboration should not be mistaken for scientific depth. The paper would benefit substantially from a more concise presentation that clearly distinguishes genuinely new findings from expected outcomes.

Questions for authors
  1. What is the most important scientific insight of this work that could not reasonably have been anticipated from existing DAgger and continual-learning literature? Can the authors articulate a concrete conceptual or methodological contribution beyond empirical confirmation of known principles?

  2. Given that random intermediate-state selection performs comparably to or better than INSIGHT, what justifies uncertainty-based selection as a central component? Would simpler intervention heuristics provide the same practical benefits?

  3. Why is the work framed as lifelong learning when only a single adaptation cycle is evaluated? Can the authors demonstrate sequential adaptation across multiple rounds, including retention of previously acquired capabilities?

2 comments

We thank the reviewer for the detailed assessment and clarify the contribution below.

Q1: What is the scientific insight beyond DAgger and conventional continual learning?

DAgger motivates collecting expert demonstrations on states visited by the policy, and continual learning motivates preserving prior competence during updates. To our knowledge, RECALL is the first empirical study to jointly examine uncertainty-guided recovery collection after task-specific fine-tuning and the continual-learning trade-offs of using those recoveries to update a token-based VLA. We vary both how recovery demonstrations are selected from the policy’s own rollouts and how they are used in the next update.

First, imitation learning motivates supervision on states the learner visits. We added two fine-tuning runs per RQ1 condition. Across three runs, overall success was 66.93 ± 1.09% for INSIGHT-First, 67.63 ± 0.55% for Random-All, 67.78 ± 0.21% for Random-Success, and 57.03 ± 0.69% for Start. All three intermediate-state methods outperformed Start in each run. Recoveries that begin partway through a task-tuned VLA’s own rollouts can improve full-task execution under both uncertainty-guided and random collection.

Second, collecting at every trigger covers more intervention opportunities. INSIGHT-First averaged 66.93 ± 1.09% overall with 149 demonstrations, compared with 63.62 ± 2.56% for INSIGHT-All with 576. First had a higher overall ten-checkpoint average in all three runs, and collected-task means were similar (45.63% versus 44.71%). Under this protocol, collecting at the first trigger yielded comparable collected-task performance with about one quarter as many demonstrations. The overall difference is descriptive.

Third, continual learning motivates checking whether an update preserves prior competence. In our recovery-only experiment (Sec. 7), success on tasks receiving new data fell to 0.4% at one checkpoint, compared with 32.0% for the original policy. Fine-tuning solely on expert recoveries can erode the ability to complete the very tasks those demonstrations target.

Q2: Why is uncertainty selection central if random selection is competitive?

We expected recoveries from uncertain states to provide more useful supervision. The comparable post-fine-tuning success on LIBERO-10 makes random sampling a useful baseline for testing which tasks or uncertainty measures benefit from guided collection.

On hardware, INSIGHT scores action chunks before execution, giving the operator a chance to intervene before a flagged motion. A live random trigger fires independently of the proposed action and can arrive after an error. This motivates our choice of INSIGHT for live collection. Measuring how often each trigger prevents failure requires a direct hardware comparison.

Q3: Why frame the work as lifelong learning with one adaptation cycle?

RECALL examines a first step toward lifelong learning. We fine-tune a pretrained VLA on task demonstrations, then collect expert recoveries from its rollouts for a second update. Repeating this process requires each updated policy to retain enough competence to execute tasks and generate useful experience for the next round.

We test that requirement for one update on both tasks receiving recoveries and previously learned tasks receiving no new data. Even this single cycle exposes an unresolved challenge. Recovery-only fine-tuning can degrade full-task success on the tasks being updated. How to incorporate recovery supervision while preserving full-task competence across repeated updates remains open.

Scope and presentation

We acknowledge the single-architecture and benchmark limitations. In our physical-robot study, with 30 demonstrations per condition and 60 matched scene pairs, recovery-trained adaptation improved success from 53.3% to 71.7% (two-sided exact McNemar p = 0.0347). We will revise the introduction and main results to foreground the findings on intermediate-state collection, collection density, and retention of full-task competence. We will also state explicitly that the experiments cover one post-deployment update.

Thank you for the detailed response and the additional experiments. While the multi-seed results improve the reliability of the findings, my primary concern regarding the limited novelty and significance remains.

Among the three highlighted contributions, the second finding—comparable performance with substantially fewer recovery demonstrations—is arguably the only somewhat non-obvious observation. However, this comparison is conducted exclusively in simulation and relies on restoring intermediate simulator states, making its practical relevance to real-world interactive learning unclear. The other two findings largely confirm well-established principles from DAgger and continual learning.

More broadly, the experimental scope remains quite limited. The simulation study is restricted to libero-long, a benchmark on which recent methods have already reported near-saturated performance. The evaluation uses only one VLA architecture and, most importantly, only a single adaptation cycle, despite positioning lifelong learning as the central research problem. The small-scale real-robot experiments do not resolve these limitations or validate the distinctive First-vs-All finding.

Taken together, the limited novelty, saturated simulation benchmark, and single-cycle evaluation make the work feel closer to a small-scale proof of concept than a convincing study of active lifelong learning. Despite its extensive presentation, I remain unconvinced that the paper provides sufficient new scientific insight or experimental depth for ICLR.

I appreciate the authors' efforts but will maintain my original rating.

Reviewer_rngAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents RECALL, an empirical study of active lifelong learning for a pretrained π₀-FAST VLA policy. It investigates six questions: whether recovery demonstrations should begin at task starts or policy-induced intermediate states; whether uncertainty-guided selection helps choose those states; how densely to collect recovery demonstrations; whether recovery data alone can support fine-tuning; whether reduced learning rates or EWC can limit forgetting; what prior data replay should cover; and whether recovery collection can be used on a physical robot.

In simulation, the authors compare start-state demonstrations with uncertainty-guided and random intermediate-state recovery demonstrations, then study collection density, recovery-only fine-tuning, conservative updates, and replay buffers with different coverage. They evaluate overall success as well as performance on tasks receiving recovery data and tasks without new demonstrations. In a separate xArm-7 study, they compare a policy fine-tuned on uncertainty-triggered recovery demonstrations with one fine-tuned on matched start-state demonstrations.

The authors report higher observed performance for intermediate-state demonstrations than for start-state demonstrations, similar results for uncertainty-guided and random intermediate-state selection, no higher observed mean from denser recovery collection, degradation under recovery-only fine-tuning, limited adaptation under conservative updates, and a role for replay coverage in retaining prior behaviors. In the hardware study, the recovery-trained policy performs better in aggregate than the start-state-trained policy on the three evaluated tasks.

Critical strengths

S1) The paper provides a broad empirical study of recovery collection and VLA adaptation. It compares recovery-state selection and collection density, then examines recovery-only fine-tuning, update constraints, and replay coverage. These experiments offer useful evidence about the tradeoffs involved in adapting a policy while retaining prior behavior.

S2) The paper evaluates recovery collection on a physical robot. On an xArm7, the authors compare uncertainty-triggered recovery demonstrations with matched start-state demonstrations and report higher aggregate success for the recovery-trained policy across three tasks. This provides a practical feasibility result, while leaving open whether uncertainty is the best way to select recovery states.

Critical weaknesses

W1) The authors appropriately acknowledge that each simulation condition uses one fine-tuning run and that the RQ1/RQ2 comparisons are exploratory. These limitations nevertheless weaken the evidence for the findings. The corrected RQ1 comparisons do not establish a reliable advantage for intermediate-state collection, and the nonsignificant RQ2 comparison does not establish equivalence. The single-run design also provides no estimate of variability across training runs. Normalization differences further complicate attribution in some adaptation comparisons.

W2) The practical benefit of adding uncertainty-guided selection is not established. In simulation, random intermediate-state controls perform similarly to INSIGHT, and the hardware study does not compare INSIGHT with a simpler online recovery-selection strategy. Although the authors report low added inference latency and demonstrate the intervention loop on an xArm7, they do not show that the full system reduces expert time or collection cost. Thus, the paper demonstrates feasibility and target-task improvement over start-state collection, but leaves unclear whether uncertainty-guided selection justifies its added system complexity.

W3) The paper does not introduce a novel active-learning criterion, and its experiments do not establish a distinct benefit from the existing uncertainty-based selector. The simulation controls show similar performance for random intermediate-state selection, while the hardware experiment compares uncertainty-triggered recovery with start-state demonstrations, not with another recovery-state selection strategy. The paper therefore provides an empirical study of recovery supervision, but the evidence for its specific active-selection contribution is limited (please see Q2).

Questions for authors

Q1) What evidence supports a benefit from uncertainty-guided state selection beyond its operational feasibility? In simulation, random intermediate-state controls perform similarly to INSIGHT, and the hardware study compares INSIGHT-triggered recovery demonstrations with start-state demonstrations rather than another recovery-state selector. Should the paper’s contribution therefore be understood as studying recovery-state supervision generally, rather than validating uncertainty as an especially useful selection criterion?

Q2) The paper describes itself as the first empirical study of active lifelong learning in VLAs. Could the authors clarify which aspects they consider novel relative to prior work on continual VLA adaptation, including the works discussed in Section 2? In particular, is the contribution the recovery-state collection setting and its empirical analysis, rather than a novel state-selection criterion or adaptation method?

2 comments

We thank the reviewer for the constructive feedback and for recognizing the value of our empirical analysis and hardware experiments.

W1: Independent training runs and statistical robustness

We completed two additional fine-tuning runs for each of the five RQ1/RQ2 configurations, yielding three runs per method. All runs use the same collected datasets, base checkpoint, training procedure, and ten-checkpoint evaluation protocol (50 rollouts per LIBERO-10 task at each checkpoint).

The additional runs extend the RQ1 comparison in Figure 2. We average success across the same ten evaluation checkpoints within each run, then report the mean and sample standard deviation across the three runs.

MethodOverall success (mean ± SD)Collected-task success (mean ± SD)Retained-task success (mean ± SD)
Start57.03 ± 0.69%26.83 ± 0.59%87.23 ± 1.00%
INSIGHT-First66.93 ± 1.09%45.63 ± 2.05%88.23 ± 0.62%
Random-All67.63 ± 0.55%48.20 ± 0.70%87.07 ± 1.01%
Random-Success67.78 ± 0.21%47.61 ± 0.46%87.95 ± 0.30%

All three intermediate-state methods outperformed Start in every training run, with average overall gains of 9.90 percentage points for INSIGHT, 10.61 for Random-All, and 10.75 for Random-Success. The gains were concentrated on collected tasks, where mean success ranged from 45.63% to 48.20% across the three methods versus 26.83% for Start. Retained-task success remained similar across all four conditions (87.07%–88.23%).

For RQ2, INSIGHT-First achieves 66.93 ± 1.09% overall success versus 63.62 ± 2.56% for INSIGHT-All, using 149 demonstrations versus 576. INSIGHT-First has a higher overall ten-checkpoint average in all three runs, although the advantage ranges from +0.12 to +6.22 percentage points. INSIGHT-First led on collected tasks in two of the three runs, with differences of +1.40, +3.56, and −2.20 percentage points relative to INSIGHT-All. The retained-task advantage was larger on average and positive in all three runs.

These experiments demonstrate that our main observed trends are consistent across independent training runs. Nevertheless, three runs provide limited inferential power: the exact two-sided seed-level sign-flip test has a minimum attainable p-value of 0.25. We applied Holm correction separately to the 18 RQ1 contrasts and the three RQ2 contrasts. For each method pair, overall, collected-task, and retained-task success count as separate contrasts. No contrast was significant after correction.

We therefore interpret these results as strengthened evidence of repeatability, not definitive statistical proof or equivalence. Because the demonstrations are held fixed, these experiments assess training variability rather than independent collection variability.

W2 and Q1: Value of uncertainty-guided selection

Across all three LIBERO-10 training runs, both random intermediate-state baselines beat Start and performed comparably to INSIGHT. Random sampling is therefore a useful baseline for testing other uncertainty measures and tasks. We expected uncertainty-guided collection to outperform random sampling. The comparable performance on LIBERO-10 motivates testing whether uncertainty-guided collection offers an advantage on other tasks.

On hardware, INSIGHT can flag a high-uncertainty action before execution, giving the operator a chance to intervene before a potential error. A live random trigger would fire independently of the proposed action and could fire after an error. The hardware study we show in the paper establishes the feasibility of this loop. Thus the value of the uncertainty–guided selection is in its practical deployment even if it does not result in learning-based performance gains.

W3 and Q2: Scientific contribution

To our knowledge, RECALL is the first empirical study to jointly examine (1) uncertainty-guided recovery collection after task-specific fine-tuning and (2) the continual-learning trade-offs of using those recoveries to update a token-based VLA. We build on the continual learning, VLA, and active learning work in Section 2 by treating how new supervision is collected during deployment as an experimental variable alongside how the policy is updated. The novelty of the paper is in this setting and its systematic evaluation using established selection and adaptation methods.

Intermediate-state recovery data improved performance, while the tested uncertainty criterion showed no advantage over random selection in simulation. This motivates future work to address the question: When do uncertainty measures identify states that yield more useful supervision? Furthermore, we found that recovery-only fine-tuning could also degrade success on the very tasks receiving new demonstrations. This prompts the question: What prior behaviors must a replay buffer preserve for the policy to benefit from those recoveries?

Reviewer_rngAOpenReview ↗

reply to rebuttal

Thank you for rebuttal. The multi-seed results now provide more evidence about the components of the method. However, my central concern in Q1 remains. INSIGHT enables pre-execution intervention, which is an operational feasibility, but it has not yet been shown an advantage over other online intervention strategies, such as, random or periodic pre-execution triggers. Therefore, the specific contribution of uncertainty-guided selection is still unclear.

Reply supplied to Q2 is rather more critical. The authors clarify that the contribution lies in the empirical setting. But, this does not constitute a sufficiently distinct scientific contribution. More importantly, the experiments evaluate only one cycle, without demonstrating how the selection mechanism and adaptation interact during learning updates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.