- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
RECALL studies how a pretrained VLA can learn from human recovery demonstrations. Using pi0-FAST on LIBERO, the authors compare where demonstrations begin, how many recoveries to collect, and how to update the policy while retaining prior skills. Intermediate-state recoveries give higher average success than start-state demonstrations, with no clear advantage for uncertainty-based selection or denser collection. Training only on recoveries can hurt both collected and retained tasks; the runs with replay perform better. On an xArm-7, recovery and start-state collection with 30 demonstrations each yield 71.7% and 53.3% success over 60 matched scene pairs.
Critical strengths
-
I found the random intermediate-state controls particularly useful. They obtain similar observed success rates to uncertainty-triggered collection, showing why the recovery gains should not be credited to the uncertainty predictor alone. Reporting how many candidate states were needed to obtain the demonstrations also gives a more useful picture of collection effort.
-
The real-robot comparison is a strong evidence in the paper. Both policies use 30 demonstrations and matched training settings, with fixed checkpoints and randomized policy order across 60 paired scenes. The 18.3-percentage-point gain supports the value of collecting recoveries on hardware.
Critical weaknesses
-
Each simulation condition has one training run and one update cycle, evaluated on five collected and five retained tasks. Checkpoint averages do not show whether the findings repeat across seeds, and no RQ1/RQ2 comparison is significant after the stated Holm correction. The authors acknowledge these limits. The results are useful observations, but retention through repeated updates remains untested.
-
Replay and regularization are not always compared under the same normalization. Changing replay scope also changes dataset size and the balance of old and new samples. This makes it hard to tell how much of the retention difference comes from behavioral coverage. The normalization checks only partly address this concern.
Questions for authors
- What are the replay dataset sizes and effective old/new sampling ratios for the three replay scopes? These would help assess the coverage comparison.
- If additional experiments are available, do the recovery-versus-start and first-versus-all trends repeat across independent collection/training seeds under the same checkpoint protocol?
2 comments
Thanks for the additional results. The new runs help address my concern about repeatability, and the replay breakdown clarifies the setup. The role of replay and retention over repeated updates remain less clear. I’m keeping my score.
We thank the reviewer for the constructive assessment and for highlighting the usefulness of the random intermediate-state controls and matched physical-robot evaluation. We have completed additional fine-tuning runs addressing Q2 and further examined the replay-data accounting raised in Q1.
Q2: Do the RQ1/RQ2 trends repeat across independent seeds?
Yes, the principal trends repeat across training runs. We added two runs per condition, yielding three runs for each of the five RQ1/RQ2 configurations. All runs use the same demonstration datasets, base checkpoint, training schedule, and ten-checkpoint evaluation protocol.
For RQ1, each run contributes one ten-checkpoint average. The table reports the mean and sample SD across the three runs.
All three intermediate-state methods outperformed Start in every run. INSIGHT-First exceeded Start by +8.92, +10.42, and +10.36 percentage points overall (mean +9.90). The gains were concentrated on collected tasks, and retained-task success stayed around 87–88% across the four conditions.
For RQ2, INSIGHT-First averaged 66.93 ± 1.09% overall versus 63.62 ± 2.56% for INSIGHT-All. First led in all three runs by +3.58, +6.22, and +0.12 percentage points. The collected-task means were similar (45.63% versus 44.71%), and First had higher retained-task success (88.23% versus 82.53%). Collecting 3.87× as many demonstrations yielded a lower observed overall average in each run.
With three paired training runs, the minimum attainable two-sided exact sign-flip p-value is 0.25. No contrast was significant after Holm correction. These results describe training-run repeatability with the demonstration datasets held fixed.
Q1: Replay dataset sizes and sampling ratios
Full, LIBERO-10, and Targeted replay draw from 1,693, 379, and 180 prior demonstrations across 40, 10, and five tasks, respectively. All three use the same 149 recovery demonstrations. LIBERO-10 replay performed similarly to Full replay with less than a quarter as many prior demonstrations. This suggests that which tasks are represented may matter more than total demonstration count in this setting.
Task coverage and dataset size vary together in RQ5. Demonstration counts alone do not determine how often prior and recovery examples are sampled during training. We are running a follow-up with 180 prior demonstrations per condition, preserving representation of all 40, 10, or five tasks and holding the recovery data and old/new sampling ratio fixed. We will report the results when the runs are complete.