- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes "RVSR", a retrieval framework for few-shot robot imitation learning. It segments demonstrations using frozen video features and constructs separate representations for target relevance and redundancy. A sequential selector combines these signals with language consistency to retrieve training data. Experiments on LIBERO-10 and three real-robot tasks report improvements over retrieval baselines, with 66.4% average success on LIBERO-10.
Critical strengths
- The paper is well written and easy to follow. The motivation, method, and experimental results are presented in a logical order.
- The authors make a careful effort to control the comparisons. They report both original and matched baseline settings, aligning per-query retrieval budgets and target/prior sampling.
- The problem is well motivated. Considering both relevance and redundancy addresses a practical concern when selecting prior demonstrations for few-shot learning.
- The detailed appendices document implementation choices, evaluation protocols, and additional analyses. The supplied code supports reproduction of the proposed simulation pipeline.
Critical weaknesses
-
The claim that video representations better capture behavioral relevance needs more direct evidence. In Table 22, V-JEPA 2 with Top-K achieves 56.5±8.5%, compared with 57.5±5.3% for DINOv2. This does not show an advantage for video features under Top-K, although downstream success does not directly measure retrieval relevance. The encoder comparison also changes segmentation boundaries. The full selector improves performance with V-JEPA 2, but this does not separate better relevance estimation from better redundancy estimation or their interaction. Figure 1 provides qualitative examples, while Action-DTW measures redundancy among retrieved segments. An independent assessment of behavioral matches between queries and candidates would better support this claim.
-
The evaluation provides limited evidence under stronger distribution shifts. The simulation setting retrieves from LIBERO-90 for long-horizon LIBERO-10 tasks, with the same robot and observation/action spaces. The Mug-Mic analysis illustrates reuse of related manipulation behaviors from prior tasks. This is a useful transfer setting, but it leaves open how retrieval works when target demonstrations contain unfamiliar object categories or backgrounds. The paper studies cross-scene retrieval and real-world scene variation, but does not systematically evaluate these shifts. Such tests would better support the motivation of matching behavior across visual differences.
-
The policy evaluation is limited to BC Transformer. Using the same architecture across methods is appropriate for comparing retrieval procedures. However, it remains unclear whether the gains transfer to pretrained vision-language-action policies such as π0 or GR00T. These models have different visual and behavioral priors, which may change the benefit of retrieved data. Evaluating retrieval methods with a shared pretrained VLA under the same few-shot budget would make the practical value more convincing.
-
The improvement over the strongest baseline is modest, and broader benchmark evidence would strengthen the result. On LIBERO-10, "RVSR" achieves 66.4%, compared with 63.5% for COLLAGE*, a gain of 2.9 percentage points. The real-robot results are encouraging, but cover three pick-and-place tasks with ten trials per task. Although the high LIBERO-10 success reported by OpenVLA-OFT uses the full LIBERO training data (up to 50 demonstrations per task) and is not directly comparable, the absolute success rates here (44.9–66.4%) leave open whether the gains persist with stronger policies (see Weakness 3). Broader task coverage and stronger visual variation would help establish the method's value beyond the current setting. Tasks from RoboLab or RoboTwin could provide useful additional evaluation settings.
Questions for authors
Please see weaknesses.
4 comments
Response to Reviewer MPuY (2/2)
W2. Action retrieval under background changes and added distractors
Using LIBERO-Plus (Fei et al., 2025), we changed the background materials or added three surrounding objects in target observations. We kept the manipulation targets, trajectories, actions, and LIBERO-90 prior unchanged.
KTS boundaries are recomputed for each observation condition. Within each condition, the no-penalty control and RVSR use the same queries, V-JEPA 2 features, KTS boundaries, top-30 candidate pool, final selection size of five, and relevance/language weights. Retrieval metrics follow the definitions in W1.
| Target observations | Selection rule | Action match ↑ | Redundancy among action-matched pairs ↓ |
|---|---|---|---|
| Baseline observations | No redundancy penalty (fixed weights) | 86.7% | 46.8% |
| Baseline observations | RVSR | 83.3% | 24.8% |
| Changed background materials | No redundancy penalty (fixed weights) | 85.2% | 43.5% |
| Changed background materials | RVSR | 80.8% | 23.1% |
| Three added distractors | No redundancy penalty (fixed weights) | 86.6% | 45.5% |
| Three added distractors | RVSR | 82.7% | 23.2% |
The baseline and perturbed observations use the same video-generation setup.
RVSR's action match is 83.3% under the baseline observations, 80.8% after the background change, and 82.7% with added distractors. Under both perturbations, redundancy among action-matched pairs remains lower than in the no-penalty control.
These experiments complement the existing cross-scene retrieval examples by varying observations while preserving the manipulation behavior and prior dataset. They provide retrieval-level evidence under the tested visual changes. Action match is object-agnostic and does not directly measure object-specific retrieval accuracy. Downstream policy robustness under these perturbations was not evaluated here.
The revised manuscript reports these experiments in Appendix F.6 (Table 36), with example observation conditions in Figure 5.
W3. Few-shot adaptation of the same pretrained VLA
To examine whether retrieved data remains useful when adapting a policy with pretrained visual and action priors, we started all conditions from the same base checkpoint (Black et al., 2025). We compared target-only adaptation with adding STRAP* or RVSR retrieval data.
All three conditions use five target demonstrations per task and train a shared model jointly on the 10 tasks, with a batch size of 48 and 4,000 updates. Retrieval-augmented conditions use a 50:50 target/retrieved-data sampling ratio.
For each training-data condition, we train three policies with different random seeds. We evaluate each policy checkpoint on 50 trials per task and select the best evaluated checkpoint separately for each task and policy. For each policy, we average these task-wise best-checkpoint success rates across the 10 tasks, then report the mean ± sample standard deviation across the three policies.
| training data | Task-wise best-checkpoint SR ↑ |
|---|---|
| Target only (5 demos/task) | 65.9 ± 5.0% |
| Target + STRAP* | 77.2 ± 1.6% |
| Target + RVSR | 79.3 ± 1.4% |
The asterisk denotes the manuscript's matched per-query retrieval budget and target/retrieved-data sampling ratio.
Starting from the same pretrained initialization and using the same update budget, adding RVSR-selected data increases mean SR from 65.9% with target-only adaptation to 79.3%. This extends the evaluation beyond BC-Transformer to few-shot adaptation of a pretrained VLA and supports the utility of RVSR-selected data in the tested setting.
The fine-tuning and evaluation protocol and results are reported in Appendix E.5 (Table 28) of the revised manuscript.
Thank you for the detailed response and the new experiments. They address most of my concerns. I will raise my score from 2 to 3.
Official Comment by Authors
We are pleased that our response and additional experiments have helped address your concerns. We sincerely appreciate your constructive feedback and your decision to raise the score.
Response to Reviewer MPuY (1/2)
Thank you for suggesting a behavioral relevance evaluation independent of policy success, together with tests using a pretrained policy and visual perturbations. We have added fixed-boundary encoder comparisons, state-based retrieval analyses, visual perturbation experiments, and an adaptation experiment using π0.5. For each policy seed, we first average success rate (SR) across the 10 LIBERO-10 tasks, then report the mean ± sample standard deviation across three policy seeds.
W1. Behavioral relevance of video representations and contributions of the selection rule
Fixed-boundary encoder-by-selector comparison. We agree that Top-K policy SR alone does not establish the behavioral relevance of video representations. We fixed all target and prior segments to the boundaries generated by V-JEPA 2 with KTS, then crossed DINOv2/V-JEPA 2 with Top-K/RVSR. Every condition uses the same query segments and prior segments, and selects five segments per query. The revised manuscript reports this fixed-boundary comparison in Appendix E.3.1 (Table 26), whereas Table 25 uses encoder-specific KTS boundaries.
We derived action labels from robot and object states independently of retrieval features and scores. Action match is the fraction of the five retrieved segments containing all evaluated action types required by the query, with opening and closing treated separately. The metric is object-agnostic: a grasping query can match a grasping segment involving a different object category.
Action match is evaluated on the 334 of the 336 shared queries that contain at least one evaluated action. Retrieval metrics are computed per query, averaged over eligible queries within each task, then averaged across the 10 tasks.
The state-based metric definitions are provided in Appendix F.1 of the revised manuscript.
Redundancy among all selected pairs is the fraction of all 10 pairs formed by the five selected segments satisfying the manuscript's Action-DTW near-duplicate criterion.
Under this common segmentation, V-JEPA 2 yields higher action match with both selection rules, while RVSR reduces redundancy with either encoder. V-JEPA 2 with Top-K has higher action match than DINOv2 with Top-K but does not achieve higher mean policy SR. Among the four evaluated combinations, V-JEPA 2 with RVSR achieves the highest mean SR. These results distinguish action-type matching from downstream policy performance.
Isolating the redundancy term. We held the V-JEPA 2 features, KTS boundaries, top-30 relevance candidates, final selection size of five, relevance/language weights, and first selected candidate fixed. Starting from the full score
0.5 rel - 0.5 red + 0.3 lang, we compared:0.5 rel + 0.3 lang.The no-penalty control preserves the relative weighting of relevance and language. Top-K in the preceding table ranks candidates by mean-feature relevance and does not include the language term.
Redundancy among action-matched pairs is the fraction of pairs satisfying the manuscript's Action-DTW near-duplicate criterion among pairs in which both retrieved segments match the query's evaluated action types. Queries with fewer than two matching segments are excluded. The direction of the reduction is unchanged when restricting the comparison to queries eligible under all three conditions.
With the representation and relevance/language weights fixed, RVSR increases mean SR by 6.4 percentage points over the no-penalty control and by 3.4 percentage points over semantic-only redundancy. Action match decreases from 86.9% to 83.7%, while redundancy among action-matched pairs decreases from 46.5% to 25.7%. These comparisons separately evaluate the representation's action matching and the policy-learning effect of reducing repetition among matching action types.
This controlled ablation is reported in Appendix E.1 (Table 23) of the revised manuscript.