acceptodds
Under review as a conference paper at ICLR 2027

Redundancy-Aware Video Segment Retrieval for Few-Shot Robot Imitation Learning

Abstract

Retrieval-augmented imitation learning improves few-shot robot policy learning by augmenting limited target demonstrations with relevant segments from a large prior dataset. Existing methods, however, often rely on framewise image similarity, which can retrieve visually similar segments despite differences in the underlying manipulation behavior. Moreover, independently ranking candidates with a Top-K rule can fill the retrieval budget with redundant segments that exhibit highly similar spatiotemporal manipulation patterns, limiting the benefit of data augmentation. We propose RVSR, a redundancy-aware video segment retrieval framework that jointly considers target relevance and redundancy among selected segments. RVSR partitions robot demonstrations into temporally coherent segments using a frozen pretrained video representation and derives two role-specific features: a semantic feature for target relevance and a temporal-pattern feature for redundancy estimation. It then sequentially selects relevant segments while suppressing manipulation patterns already represented in the retrieved data. RVSR achieves 66.4% success on LIBERO-10, outperforming the strongest retrieval baseline at 63.5%, and obtains the highest average performance across three real-robot tasks. Ablations further show that temporally structured video representations and redundancy-aware selection are most effective when combined to improve few-shot policy learning. The source code is included in the supplementary materials.

Primary area:
applications to robotics, autonomy, planning
Keywords:
retrieval, few-shot imitation learning, data selection, robot manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MPuYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes "RVSR", a retrieval framework for few-shot robot imitation learning. It segments demonstrations using frozen video features and constructs separate representations for target relevance and redundancy. A sequential selector combines these signals with language consistency to retrieve training data. Experiments on LIBERO-10 and three real-robot tasks report improvements over retrieval baselines, with 66.4% average success on LIBERO-10.

Critical strengths
  1. The paper is well written and easy to follow. The motivation, method, and experimental results are presented in a logical order.
  2. The authors make a careful effort to control the comparisons. They report both original and matched baseline settings, aligning per-query retrieval budgets and target/prior sampling.
  3. The problem is well motivated. Considering both relevance and redundancy addresses a practical concern when selecting prior demonstrations for few-shot learning.
  4. The detailed appendices document implementation choices, evaluation protocols, and additional analyses. The supplied code supports reproduction of the proposed simulation pipeline.
Critical weaknesses
  1. The claim that video representations better capture behavioral relevance needs more direct evidence. In Table 22, V-JEPA 2 with Top-K achieves 56.5±8.5%, compared with 57.5±5.3% for DINOv2. This does not show an advantage for video features under Top-K, although downstream success does not directly measure retrieval relevance. The encoder comparison also changes segmentation boundaries. The full selector improves performance with V-JEPA 2, but this does not separate better relevance estimation from better redundancy estimation or their interaction. Figure 1 provides qualitative examples, while Action-DTW measures redundancy among retrieved segments. An independent assessment of behavioral matches between queries and candidates would better support this claim.

  2. The evaluation provides limited evidence under stronger distribution shifts. The simulation setting retrieves from LIBERO-90 for long-horizon LIBERO-10 tasks, with the same robot and observation/action spaces. The Mug-Mic analysis illustrates reuse of related manipulation behaviors from prior tasks. This is a useful transfer setting, but it leaves open how retrieval works when target demonstrations contain unfamiliar object categories or backgrounds. The paper studies cross-scene retrieval and real-world scene variation, but does not systematically evaluate these shifts. Such tests would better support the motivation of matching behavior across visual differences.

  3. The policy evaluation is limited to BC Transformer. Using the same architecture across methods is appropriate for comparing retrieval procedures. However, it remains unclear whether the gains transfer to pretrained vision-language-action policies such as π0 or GR00T. These models have different visual and behavioral priors, which may change the benefit of retrieved data. Evaluating retrieval methods with a shared pretrained VLA under the same few-shot budget would make the practical value more convincing.

  4. The improvement over the strongest baseline is modest, and broader benchmark evidence would strengthen the result. On LIBERO-10, "RVSR" achieves 66.4%, compared with 63.5% for COLLAGE*, a gain of 2.9 percentage points. The real-robot results are encouraging, but cover three pick-and-place tasks with ten trials per task. Although the high LIBERO-10 success reported by OpenVLA-OFT uses the full LIBERO training data (up to 50 demonstrations per task) and is not directly comparable, the absolute success rates here (44.9–66.4%) leave open whether the gains persist with stronger policies (see Weakness 3). Broader task coverage and stronger visual variation would help establish the method's value beyond the current setting. Tasks from RoboLab or RoboTwin could provide useful additional evaluation settings.

Questions for authors

Please see weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer MPuY (1/2)

Thank you for suggesting a behavioral relevance evaluation independent of policy success, together with tests using a pretrained policy and visual perturbations. We have added fixed-boundary encoder comparisons, state-based retrieval analyses, visual perturbation experiments, and an adaptation experiment using π0.5\pi_{0.5}. For each policy seed, we first average success rate (SR) across the 10 LIBERO-10 tasks, then report the mean ± sample standard deviation across three policy seeds.

W1. Behavioral relevance of video representations and contributions of the selection rule

Fixed-boundary encoder-by-selector comparison. We agree that Top-K policy SR alone does not establish the behavioral relevance of video representations. We fixed all target and prior segments to the boundaries generated by V-JEPA 2 with KTS, then crossed DINOv2/V-JEPA 2 with Top-K/RVSR. Every condition uses the same query segments and prior segments, and selects five segments per query. The revised manuscript reports this fixed-boundary comparison in Appendix E.3.1 (Table 26), whereas Table 25 uses encoder-specific KTS boundaries.

We derived action labels from robot and object states independently of retrieval features and scores. Action match is the fraction of the five retrieved segments containing all evaluated action types required by the query, with opening and closing treated separately. The metric is object-agnostic: a grasping query can match a grasping segment involving a different object category.

Action match is evaluated on the 334 of the 336 shared queries that contain at least one evaluated action. Retrieval metrics are computed per query, averaged over eligible queries within each task, then averaged across the 10 tasks.

The state-based metric definitions are provided in Appendix F.1 of the revised manuscript.

EncoderSelection ruleAction match ↑Redundancy among all selected pairs ↓Policy SR ↑
DINOv2Top-K78.9%35.5%57.9 ± 2.0%
DINOv2RVSR74.7%14.5%58.3 ± 6.8%
V-JEPA 2Top-K88.4%43.1%56.5 ± 8.5%
V-JEPA 2RVSR83.7%23.9%66.4 ± 1.3%

Redundancy among all selected pairs is the fraction of all 10 pairs formed by the five selected segments satisfying the manuscript's Action-DTW near-duplicate criterion.

Under this common segmentation, V-JEPA 2 yields higher action match with both selection rules, while RVSR reduces redundancy with either encoder. V-JEPA 2 with Top-K has higher action match than DINOv2 with Top-K but does not achieve higher mean policy SR. Among the four evaluated combinations, V-JEPA 2 with RVSR achieves the highest mean SR. These results distinguish action-type matching from downstream policy performance.

Isolating the redundancy term. We held the V-JEPA 2 features, KTS boundaries, top-30 relevance candidates, final selection size of five, relevance/language weights, and first selected candidate fixed. Starting from the full score 0.5 rel - 0.5 red + 0.3 lang, we compared:

  • No redundancy penalty (fixed weights): remove only the redundancy term, retaining 0.5 rel + 0.3 lang.
  • Semantic-only redundancy (β=0\beta = 0): retain the redundancy penalty using semantic features alone.
  • RVSR: use semantic and DCT features in the redundancy score.

The no-penalty control preserves the relative weighting of relevance and language. Top-K in the preceding table ranks candidates by mean-feature relevance and does not include the language term.

Selection ruleAction match ↑Redundancy among action-matched pairs ↓Policy SR ↑
No redundancy penalty (fixed weights)86.9%46.5%60.0 ± 1.8%
Semantic-only redundancy (β=0\beta = 0)84.5%35.3%63.0 ± 2.1%
RVSR83.7%25.7%66.4 ± 1.3%

Redundancy among action-matched pairs is the fraction of pairs satisfying the manuscript's Action-DTW near-duplicate criterion among pairs in which both retrieved segments match the query's evaluated action types. Queries with fewer than two matching segments are excluded. The direction of the reduction is unchanged when restricting the comparison to queries eligible under all three conditions.

With the representation and relevance/language weights fixed, RVSR increases mean SR by 6.4 percentage points over the no-penalty control and by 3.4 percentage points over semantic-only redundancy. Action match decreases from 86.9% to 83.7%, while redundancy among action-matched pairs decreases from 46.5% to 25.7%. These comparisons separately evaluate the representation's action matching and the policy-learning effect of reducing repetition among matching action types.

This controlled ablation is reported in Appendix E.1 (Table 23) of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer MPuY (2/2)

W2. Action retrieval under background changes and added distractors

Using LIBERO-Plus (Fei et al., 2025), we changed the background materials or added three surrounding objects in target observations. We kept the manipulation targets, trajectories, actions, and LIBERO-90 prior unchanged.

KTS boundaries are recomputed for each observation condition. Within each condition, the no-penalty control and RVSR use the same queries, V-JEPA 2 features, KTS boundaries, top-30 candidate pool, final selection size of five, and relevance/language weights. Retrieval metrics follow the definitions in W1.

Target observationsSelection ruleAction match ↑Redundancy among action-matched pairs ↓
Baseline observationsNo redundancy penalty (fixed weights)86.7%46.8%
Baseline observationsRVSR83.3%24.8%
Changed background materialsNo redundancy penalty (fixed weights)85.2%43.5%
Changed background materialsRVSR80.8%23.1%
Three added distractorsNo redundancy penalty (fixed weights)86.6%45.5%
Three added distractorsRVSR82.7%23.2%

The baseline and perturbed observations use the same video-generation setup.

RVSR's action match is 83.3% under the baseline observations, 80.8% after the background change, and 82.7% with added distractors. Under both perturbations, redundancy among action-matched pairs remains lower than in the no-penalty control.

These experiments complement the existing cross-scene retrieval examples by varying observations while preserving the manipulation behavior and prior dataset. They provide retrieval-level evidence under the tested visual changes. Action match is object-agnostic and does not directly measure object-specific retrieval accuracy. Downstream policy robustness under these perturbations was not evaluated here.

The revised manuscript reports these experiments in Appendix F.6 (Table 36), with example observation conditions in Figure 5.

W3. Few-shot adaptation of the same pretrained VLA

To examine whether retrieved data remains useful when adapting a policy with pretrained visual and action priors, we started all conditions from the same π0.5\pi_{0.5} base checkpoint (Black et al., 2025). We compared target-only adaptation with adding STRAP* or RVSR retrieval data.

All three conditions use five target demonstrations per task and train a shared model jointly on the 10 tasks, with a batch size of 48 and 4,000 updates. Retrieval-augmented conditions use a 50:50 target/retrieved-data sampling ratio.

For each training-data condition, we train three policies with different random seeds. We evaluate each policy checkpoint on 50 trials per task and select the best evaluated checkpoint separately for each task and policy. For each policy, we average these task-wise best-checkpoint success rates across the 10 tasks, then report the mean ± sample standard deviation across the three policies.

π0.5\pi_{0.5} training dataTask-wise best-checkpoint SR ↑
Target only (5 demos/task)65.9 ± 5.0%
Target + STRAP*77.2 ± 1.6%
Target + RVSR79.3 ± 1.4%

The asterisk denotes the manuscript's matched per-query retrieval budget and target/retrieved-data sampling ratio.

Starting from the same pretrained initialization and using the same update budget, adding RVSR-selected data increases mean SR from 65.9% with target-only adaptation to 79.3%. This extends the evaluation beyond BC-Transformer to few-shot adaptation of a pretrained VLA and supports the utility of RVSR-selected data in the tested setting.

The fine-tuning and evaluation protocol and results are reported in Appendix E.5 (Table 28) of the revised manuscript.

Thank you for the detailed response and the new experiments. They address most of my concerns. I will raise my score from 2 to 3.

AuthorsOpenReview ↗

Official Comment by Authors

We are pleased that our response and additional experiments have helped address your concerns. We sincerely appreciate your constructive feedback and your decision to raise the score.

Reviewer_uhcJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RVSR, a redundancy-aware video segment retrieval framework for few-shot robot imitation learning. Building on methods such as STRAP, RVSR uses temporally contextualized features from a frozen V-JEPA 2 encoder for video segmentation and retrieval. It further combines low-frequency discrete cosine transform (DCT) features and maximal marginal relevance (MMR) to balance relevance and redundancy among retrieved segments, which are used to augment policy training.

RVSR outperforms the strongest matched retrieval baseline on LIBERO-10 and reports promising results on three real-robot tasks.

Critical strengths
  1. Well-motivated problem. The paper explicitly addresses redundant demonstration retrieval under a limited budget, where data diversity matters for imitation learning.

  2. Simple and practical framework. The method combines pretrained video representations with lightweight retrieval, without additional representation learning. The released simulation code supports reproducibility.

  3. Encouraging empirical results. The paper provides useful ablations and promising real-robot results. In particular, the encoder–selector interaction and independent segment re-encoding experiments offer valuable insights into the pipeline.

Critical weaknesses

1. Limited technical novelty and insufficient isolation of the main contribution.

The main components, including pretrained video representations, temporal segmentation, DCT features, and MMR-based selection, are largely established techniques. While their combination for redundancy-aware retrieval is interesting, the distinctive contribution remains unclear.

Despite several useful ablations, it is difficult to determine how much improvement comes specifically from redundancy-aware selection rather than other components, such as video representations and language consistency.

2. Insufficient direct evidence for the proposed retrieval mechanism.

A central motivation is to retrieve behaviorally relevant yet non-redundant demonstrations. However, direct evidence that the proposed representation improves behavior-level retrieval quality remains limited.

Moreover, redundancy suppression operates independently for each target query, allowing the same prior segment to be retrieved repeatedly across queries.

3. Limited experimental generality.

On LIBERO-10, improvements over the strongest matched baseline are concentrated in two tasks. Using only one fixed set of five target demonstrations per task also leaves robustness to different few-shot selections unclear.

The real-robot results are promising, but all three tasks involve similar pick-and-place behaviors, with only ten trials each. Two tasks also have exact object–destination matches in the prior dataset, limiting evidence of broader generality.

Questions for authors

Q1. Can the authors better isolate the contribution of redundancy-aware selection?

Despite the existing ablations, could the authors compare full RVSR against a baseline using identical V-JEPA 2 features, KTS segments, candidate pool, retrieval budget, and relevance/language coefficients, but removing only the redundancy penalty?

Q2. Can the authors provide more direct evidence for retrieval quality and redundancy reduction?

Could the authors demonstrate that RVSR retrieves behaviorally relevant segments while improving coverage of the final training dataset?

For example, a small-scale qualitative or quantitative comparison of behavior-level retrieval quality, together with dataset-level redundancy statistics, would help substantiate the central motivation. Analyses of existing retrieval results would already be valuable.

Minor Questions

  1. Robustness to distracting objects. Since RVSR uses spatially pooled video features without explicit target-object grounding, could similar-looking distractors cause retrieval of wrong-object trajectories and affect the learned policy? A brief discussion or qualitative analysis would be helpful.

  2. Retrieval budget. Although the paper studies different values of K, it reports only RVSR's performance. Does redundancy-aware selection consistently outperform relevance-based Top-K retrieval across budgets? Comparisons at one or two additional values would be informative.

  3. Broader applicability to generalist policies. Given the strong performance of recent generalist robot policies on LIBERO, could RVSR complement pretrained generalist policies or retrieval-augmented in-context learning methods such as REGENT [1], rather than solely augmenting task-specific policy training? While additional experiments may be beyond scope, discussing potential applicability and limitations would clarify the broader significance.

[1] Sridhar, Kaustubh, et al. "REGENT: A retrieval-augmented generalist agent that can act in-context in new environments." International Conference on Learning Representations, 2025.

4 comments
AuthorsOpenReview ↗

Response to Reviewer uhcJ (1/3)

Thank you for suggesting controlled experiments to isolate redundancy reduction and direct analyses of retrieval quality and the resulting training data. We have added a fixed-weight redundancy ablation, state-based analyses, retrieval-budget comparisons, and a pretrained-policy experiment. For each policy seed, we first average success rate (SR) across the 10 LIBERO-10 tasks, then report the mean ± sample standard deviation across three policy seeds.

W1. Technical contribution and the effect of redundancy reduction

RVSR separates two decisions under a limited retrieval budget: how relevant a candidate is to the query, and how much it repeats executions already represented by selected candidates. Mean video features score query-candidate relevance, while semantic and low-frequency temporal features jointly assess candidate-to-candidate redundancy. The technical contribution is this role-specific representation and selection design for few-shot imitation learning, addressing candidate pools that contain relevant segments but are dominated by similar executions.

We held the V-JEPA 2 features, KTS boundaries, top-30 relevance candidates, final selection size of five, relevance/language weights, and first selected candidate fixed. Starting from the full score 0.5 rel - 0.5 red + 0.3 lang, we compared:

  • No redundancy penalty (fixed weights): remove only the redundancy term, retaining 0.5 rel + 0.3 lang.
  • Semantic-only redundancy (β=0\beta = 0): retain the redundancy penalty using semantic features alone.
  • RVSR: use semantic and DCT features in the redundancy score.

The no-penalty control preserves the relative weighting of relevance and language. All conditions use the manuscript's BC-Transformer and a 50:50 target/retrieved-data sampling ratio.

We derived action labels from robot and object states independently of retrieval features and scores. We evaluated:

  • Action match: the fraction of the five retrieved segments containing all evaluated action types required by the query, with opening and closing treated separately. The metric is object-agnostic: a grasping query can match a grasping segment involving a different object category.
  • Redundancy among action-matched pairs: the fraction of pairs satisfying the manuscript's Action-DTW near-duplicate criterion among pairs in which both retrieved segments match the query's evaluated action types. Queries with fewer than two matching segments are excluded.

Controlled redundancy ablation: retrieval, pooled data, and policy learning

MetricNo redundancy penalty (fixed weights)Semantic-only redundancy (β=0\beta = 0)RVSR
Action match ↑86.9%84.5%83.7%
Redundancy among action-matched pairs ↓46.5%35.3%25.7%
Source-frame repetition in the pooled retrieved dataset35.3%33.1%32.3%
Number of distinct prior tasks represented6.47.37.9
Policy SR ↑60.0 ± 1.8%63.0 ± 2.1%66.4 ± 1.3%

Action match is evaluated on the 334 of the 336 shared queries that contain at least one evaluated action. Retrieval metrics are computed per query, averaged over eligible queries within each task, then averaged across the 10 tasks. Pooled-data metrics are computed after combining all retrieved segments within each target task, then averaged across the 10 tasks.

With the representation and relevance/language weights fixed, RVSR increases mean SR by 6.4 percentage points over the no-penalty control and by 3.4 percentage points over semantic-only redundancy. Action match decreases from 86.9% to 83.7%, while redundancy among action-matched pairs decreases from 46.5% to 25.7%. The higher mean SR therefore accompanies reduced repetition among matching action types, despite the decrease in action match.

The revised manuscript reports the controlled ablation in Appendix E.1 (Table 23), the pooled-data results in Appendix F.4 (Table 32), and the state-based metric definitions in Appendix F.1.

AuthorsOpenReview ↗

Response to Reviewer uhcJ (2/3)

W2. Behavioral retrieval quality and repetition in the pooled dataset

To evaluate action matching independently of policy success, we fixed all target and prior segments to the boundaries generated by V-JEPA 2 with KTS, used mean-feature Top-K selection with five retrieved segments per query, and varied the encoder.

Encoder with common boundaries and Top-K selectionAction match ↑
DINOv278.9%
V-JEPA 288.4%

Under this common segmentation, V-JEPA 2 yields higher action match. This provides a direct assessment of matching at the level of evaluated action types.

The full fixed-boundary encoder comparison is reported in Appendix E.3.1 (Table 26).

The direction of W1's reduction in redundancy among action-matched pairs is unchanged when restricting the comparison to queries eligible under all three conditions. We also pooled the retrieved segments from all queries within each target task to examine source reuse and composition.

Source-frame repetition is 1−U/N1-U/N, where NN counts all frame occurrences in the retrieved segments and UU counts unique source frames, identified by source task, demonstration, and time point. We compute this metric within each target task and average across the 10 tasks. Action-DTW measures similarity between action trajectories, including those from different demonstrations.

Compared with the no-penalty control, RVSR reduces source-frame repetition from 35.3% to 32.3% and increases the mean number of represented prior tasks from 6.4 to 7.9. The latter measures source-task diversity.

As you noted, selection operates independently for each query and permits reuse of the same source segment across queries. We therefore tested a variant that includes each identical source segment only once per target task. Segment identity requires the same source demonstration and the same start and end points.

Use of identical source segments across queriesPolicy SR ↑
Include each exact source segment once per target task62.5 ± 3.0%
Allow cross-query reuse (RVSR)66.4 ± 1.3%

Allowing cross-query reuse yields a higher mean SR in this comparison. This result distinguishes redundancy within a query's selections from reuse across queries, which may require shared partial behaviors. RVSR does not explicitly optimize redundancy across the entire pooled dataset.

The source-segment reuse experiment is reported in Appendix F.4 (Table 33) of the revised manuscript.

Q1. Removing only the redundancy penalty under matched conditions

W1 provides the requested control: removing only the redundancy term yields 60.0% mean SR, compared with 66.4% for RVSR, while retaining the representation, segmentation, candidate pool, retrieval budget, relevance/language weights, and first selected candidate.

Q2. Direct evidence on retrieval quality and dataset coverage

W1 and W2 report action matching, within-query redundancy, source-frame repetition, and source-task diversity. The exact-segment deduplication experiment separately tests source-segment reuse across queries.

Minor question: Distractors and wrong-object retrieval

Using LIBERO-Plus (Fei et al., 2025), we added three surrounding objects to observations from the same target demonstrations. We kept the manipulation targets, trajectories, actions, and LIBERO-90 prior unchanged.

KTS boundaries are recomputed for each observation condition. Within each condition, the no-penalty control and RVSR use the same queries, V-JEPA 2 features, KTS boundaries, top-30 candidate pool, final selection size of five, and relevance/language weights. Retrieval metrics follow the definitions in W1.

Target observationsSelection ruleAction match ↑Redundancy among action-matched pairs ↓
Baseline observationsNo redundancy penalty (fixed weights)86.7%46.8%
Baseline observationsRVSR83.3%24.8%
Three added distractorsNo redundancy penalty (fixed weights)86.6%45.5%
Three added distractorsRVSR82.7%23.2%

The baseline and perturbed observations use the same video-generation setup.

RVSR's action match is 83.3% under the baseline observations and 82.7% with distractors. In both conditions, redundancy among action-matched pairs remains lower than in the no-penalty control.

These results provide retrieval-level evidence under added distractors. Keeping the prior fixed does not rule out wrong-object retrieval, because changes in query observations can change retrieval rankings. Action match is object-agnostic and does not directly measure object-specific retrieval accuracy. Downstream policy robustness under these perturbations was not evaluated here. Explicit object grounding is a possible direction for addressing object confusion.

These experiments are reported in Appendix F.6 (Table 36) of the revised manuscript, with example observation conditions in Figure 5.

AuthorsOpenReview ↗

Response to Reviewer uhcJ (3/3)

Minor question: Comparison with relevance-based Top-K across retrieval budgets

We report mean-feature Top-K and RVSR results at K=5K = 5 and K=10K = 10, using V-JEPA 2 and KTS (Appendix D.6, Table 22 of the revised manuscript).

Selection rulePolicy SR at K=5K = 5 ↑Policy SR at K=10K = 10 ↑
Top-K56.5 ± 8.5%59.0 ± 6.7%
RVSR66.4 ± 1.3%62.8 ± 3.7%

RVSR has higher reported mean SR at both budgets, by 9.9 and 3.8 percentage points, respectively. The absolute SRs reported here come from different runs than the within-environment sensitivity analysis in Table 21 of the revised manuscript. This comparison evaluates the complete selection rules; W1 separately isolates the redundancy penalty while retaining the relevance/language weights.

Minor question: Extension to pretrained generalist policies and in-context methods

To evaluate retrieved data for adapting a pretrained generalist policy, we started all conditions from the same π0.5\pi_{0.5} base checkpoint (Black et al., 2025). We compared target-only adaptation with adding STRAP* or RVSR retrieval data.

All three conditions use five target demonstrations per task and train a shared model jointly on the 10 tasks, with a batch size of 48 and 4,000 updates. Retrieval-augmented conditions use a 50:50 target/retrieved-data sampling ratio.

For each training-data condition, we train three policies with different random seeds. We evaluate each policy checkpoint on 50 trials per task and select the best evaluated checkpoint separately for each task and policy. For each policy, we average these task-wise best-checkpoint success rates across the 10 tasks, then report the mean ± sample standard deviation across the three policies.

π0.5\pi_{0.5} training dataTask-wise best-checkpoint SR ↑
Target only (5 demos/task)65.9 ± 5.0%
Target + STRAP*77.2 ± 1.6%
Target + RVSR79.3 ± 1.4%

The asterisk denotes the manuscript's matched per-query retrieval budget and target/retrieved-data sampling ratio.

Starting from the same pretrained initialization and using the same update budget, adding RVSR-selected data increases mean SR from 65.9% with target-only adaptation to 79.3%. This supports the utility of RVSR-selected data for few-shot adaptation of the tested pretrained policy.

The fine-tuning and evaluation protocol and results are reported in Appendix E.5 (Table 28) of the revised manuscript.

REGENT (Sridhar et al., 2025) uses retrieved state-action examples as context at execution time to adapt to new environments without additional fine-tuning. Extending relevance- and redundancy-aware selection to a limited inference context is a possible direction beyond the training-data selection evaluated here. Such an integration would require queries constructed from observation history and explicit consideration of context length and retrieval latency.

We have added REGENT to the related-work discussion in Section 2.1 and identify applying RVSR's selection rule to its execution-time context as a future direction in Section 6 of the revised manuscript.

Thank you again for your constructive feedback. We would appreciate your thoughts on whether the additional experiments and clarifications in our rebuttal address your main concerns. Please let us know if any concerns remain or if further clarification would be helpful.

Reviewer_zMeMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RVSR uses a frozen video representation to segment and encode the demonstrations: 1) It uses the mean representation to find relevant segments. 2) it then measures redundancy by combining semantics with low-frequency temporal changes. 3) Using the task language description, this measure drives a greedy selection within each query. 4) Finally, it trains a behavior-cloning policy with a 50% mixture of the retrieved data. The experiments provide empirical results indicating this method's advantage.

Critical strengths
  • This work identifies an important limitation of relevance-based retrieval: highly relevant segments can still produce redundant training data for few-shot imitation learning.

  • The components align well with this motivation, using KTS for segmentation, semantic features for relevance, DCT for temporal patterns, and language-guided MMR to balance relevance and redundancy.

Critical weaknesses

W1. The selector may pick segments where it does not match with the task semantics

  • Suppose "grasp" and "place" segments have similar semantic means but different DCT temporal patterns. Could MMR then select the mismatched segment because it looks less redundant? The DCT feature enters only the redundancy term (Eq. 6), not the relevance term (Eq. 5).
  • The language term is task-level. One instruction covers both grasp and place, so all segments from the same demonstration get the same language score. Is this enough to tell the two phases apart?
  • It would be nice to see whether such mismatches happen, and whether they affect downstream learning.

W2. The quality of segmentation and retrieval is not measured directly.

  • The paper reports two diagnostics: the low-activity segment rate (App. C.5) and the Action-DTW near-duplicate rate (Table 25). Neither one verifies whether the behavior boundaries or the query–candidate matches are correct. A mismatched action can also increase action distance and lower the near-duplicate rate.
  • Downstream success depends on the encoder, segmentation, retrieval, and policy training together, so it cannot isolate any single step.
  • Could the authors use robot states, object states, or a small set of manual labels to evaluate:
    • segmentation quality,
    • retrieval accuracy,
    • whether de-duplication sacrifices relevance?

W4. The novelty is limited and somewhat incremental.

  • The core design combines existing pieces on top of the existing sub-trajectory retrieval pipeline (STRAP): a pretrained video encoder (V-JEPA 2), KTS segmentation, DCT features, and MMR selection.
  • The paper should state more clearly what its new technical insight is compared with prior methods. The experiments show that the combination works, but they do not attribute the gain to each new design.
Questions for authors

Besides the points listed in the weakness section, here are additional questions:

Q1. Does RVSR work with multi-view RGB?

  • STRAP’s real-robot experiments already fuse multiple views: they average per-timestep embeddings over the left, right, and in-hand cameras. RVSR encodes only the agent-view RGB for retrieval (App. C.3) and has not tested multi-view retrieval.
  • Exterior and wrist views differ in occlusion and camera motion. Can the fused features still give reliable segmentation and DCT temporal patterns? I see this mainly as a limitation of scope.

Q2. Missing ablations on the encoder and the selector.

  • Table 22 crosses DINOv2/V-JEPA 2 with Top-K/RVSR selector. However, this Top-K is not STRAP: it ranks mean-pooled features instead of using S-DTW. Changing the encoder also changes the KTS segment boundaries.
  • The paper does not report STRAP + V-JEPA 2 or COLLAGE + V-JEPA 2. Using the same encoder for the baselines would help separate the gain from the representation and the gain from the retrieval design.
  • For DCT/MMR, the Top-K ablation in Table 4 removes MMR, the DCT block, and the language term at once. Should have the following comparisons, with the encoder, segmentation, candidate pool, and language term fixed:
    • no redundancy penalty (λ = 1)
    • semantic-only de-duplication (β = 0)
    • semantic + DCT de-duplication (full RVSR)

P.S.Parts of this already exist (β = 0 in Table 4; λ = 1 in Table 19), but not in one controlled setting: Table 19 reports only deltas from a separate compute environment. Without this, the gain of the whole selector should not be attributed to MMR alone.

5 comments
AuthorsOpenReview ↗

Response to Reviewer zMeM (1/4)

Thank you for your careful assessment and specific questions about behavioral relevance and the contributions of individual components. We have added state-based retrieval and segmentation analyses, together with controlled comparisons. All policy results below use the manuscript's BC-Transformer protocol. For each policy seed, we first average success rate (SR) across the 10 LIBERO-10 tasks, then report the mean ± sample standard deviation across three policy seeds.

W1. Redundancy reduction and behavioral relevance

We agree that a task-level language score cannot distinguish grasping from placing within the same demonstration, because both segments share the same instruction. The language term measures consistency between task instructions, while video features score segment-level relevance.

To examine whether redundancy reduction selects incompatible action types, we derived action labels from robot and object states independently of retrieval features and scores. The labels cover grasping, transport, placing, and fixture manipulation.

We held the V-JEPA 2 features, KTS boundaries, top-30 relevance candidates, final selection size of five, relevance/language weights, and first selected candidate fixed. Starting from the full score 0.5 rel - 0.5 red + 0.3 lang, we compared:

  • No redundancy penalty (fixed weights): remove only the redundancy term, retaining 0.5 rel + 0.3 lang.
  • Semantic-only redundancy (β=0\beta = 0): retain the redundancy penalty using semantic features alone.
  • RVSR: use semantic and DCT features in the redundancy score.

The no-penalty control preserves the relative weighting of relevance and language.

We evaluated the following retrieval metrics:

  • Action match: the fraction of the five retrieved segments containing all evaluated action types required by the query, with opening and closing treated separately. The metric is object-agnostic: a grasping query can match a grasping segment involving a different object category.
  • Grasp/place confusion: counts candidates containing placing but not grasping for a query requiring grasping but not placing, or vice versa. The denominator includes all five retrieved candidates for every evaluated query, including queries containing both phases.
  • Redundancy among action-matched pairs: the fraction of pairs satisfying the manuscript's Action-DTW near-duplicate criterion among pairs in which both retrieved segments match the query's evaluated action types. Queries with fewer than two matching segments are excluded.

Controlled redundancy ablation

Selection ruleAction match ↑Grasp/place confusion ↓Redundancy among action-matched pairs ↓Policy SR ↑
No redundancy penalty (fixed weights)86.9%2.4%46.5%60.0 ± 1.8%
Semantic-only redundancy (β=0\beta = 0)84.5%2.3%35.3%63.0 ± 2.1%
RVSR83.7%2.5%25.7%66.4 ± 1.3%

Action match is evaluated on the 334 of the 336 shared queries that contain at least one evaluated action. Retrieval metrics are computed per query, averaged over eligible queries within each task, then averaged across the 10 tasks.

These results show a trade-off between action matching and redundancy reduction. Relative to the no-penalty control, RVSR reduces action match from 86.9% to 83.7%, while reducing redundancy among action-matched pairs from 46.5% to 25.7% and increasing mean SR from 60.0% to 66.4%. Grasp/place confusion changes from 2.4% to 2.5%. Including DCT in the redundancy score increases mean SR by 3.4 percentage points over semantic-only redundancy.

The revised manuscript reports this controlled ablation in Appendix E.1 (Table 23), with metric definitions in Appendix F.1.

AuthorsOpenReview ↗

Response to Reviewer zMeM (2/4)

W2. Direct evaluation of segmentation and retrieval quality

The state-based analysis in W1 directly evaluates retrieval at the level of action types. Its redundancy metric requires both segments in each pair to match the query's evaluated actions. The reduction from 46.5% to 25.7% therefore cannot be explained solely by selecting mismatched action types. The direction of the reduction is unchanged when restricting the comparison to queries eligible under all three conditions.

We also evaluated segmentation against state-defined action intervals in the same 50 target and 4,500 prior demonstrations. For grasping and placing, interval preservation measures the fraction of action intervals without an internal cut, counting a cut only when at least three frames of that action remain on each side. For transport, we measure the mean number of fragments per transport interval, using the same cut criterion.

Segmentation metricTarget: KTSTarget: EEFPrior: KTSPrior: EEF
Grasp interval preservation95.0%82.3%96.5%77.4%
Place interval preservation70.3%40.3%64.0%69.4%
Mean fragments per transport interval2.251.372.201.20

Preservation rates are computed per demonstration and averaged equally across demonstrations containing the corresponding action. Transport fragment counts are first averaged over intervals within each demonstration, then equally across demonstrations containing transport.

KTS preserves more grasping intervals in both datasets. Place interval preservation is higher with KTS on the target data but lower on the prior data. KTS's grasp and place preservation rates also exceed those of random segmentation matched to each demonstration's segment count. KTS divides transport intervals into more fragments, creating multiple retrieval units within an action phase.

The KTS and EEF segmentation diagnostics are reported in Appendix C.5.1 (Table 14) of the revised manuscript. In the policy-level segmentation ablation (Table 4), SR is 66.4% with KTS and 60.5% with EEF segmentation.

W4. Technical contribution and roles of the components

RVSR separates two decisions under a limited retrieval budget: how relevant a candidate is to the query, and how much it repeats executions already represented by selected candidates. Mean video features score query-candidate relevance, while semantic and low-frequency temporal features jointly assess candidate-to-candidate redundancy. The technical contribution is this role-specific representation and selection design for few-shot imitation learning, addressing candidate pools that contain relevant segments but are dominated by similar executions.

The controlled ablation in W1 isolates the redundancy penalty and the inclusion of DCT while holding the representation, segmentation, candidate pool, and relevance/language weights fixed. The fixed-boundary encoder-by-selector comparison in Q2 separately examines action matching and redundancy reduction under a common segmentation. Together, these experiments clarify the roles of the representation and selection rule, and evaluate how they combine in policy learning.

Section 5.5 of the revised manuscript summarizes these additional analyses.

AuthorsOpenReview ↗

Response to Reviewer zMeM (3/4)

Q1. Extension to multi-view RGB

To examine multi-view retrieval, we encoded agent-view and wrist-view videos separately, normalized the features from each view, and averaged them. We first fixed the agent-view KTS boundaries to evaluate retrieval and redundancy discrimination.

Here, redundancy among all selected pairs is the fraction of all 10 pairs formed by the five selected segments satisfying the manuscript's Action-DTW near-duplicate criterion. W1 instead considers only pairs in which both segments match the query's evaluated actions.

Retrieval featuresRVSR action match ↑Redundancy among all selected pairs: Top-K → RVSRROC-AUC: mean only → mean + DCT
Agent-view83.7%43.1% → 23.9%0.793 → 0.820
Agent + wrist80.6%40.7% → 20.1%0.795 → 0.844

ROC-AUC evaluates the 435 pairs among each query's top-30 agent-view relevance candidates, with Action-DTW near-duplicates as positives. Both view configurations and both scores use these same candidate pairs. We average query-level ROC-AUC within each task, then across the 10 tasks.

With fused features, including DCT improves redundancy discrimination, and RVSR reduces redundancy among all selected pairs relative to Top-K. Action match is lower with simple view averaging, at 80.6%, compared with 83.7% for agent-view features alone.

We then generated KTS boundaries from the fused features. For each view configuration, we used one penalty across all demonstrations, chosen to approximately match the total prior segment count of the agent-view reference. Results for the 4,500 prior demonstrations are:

Segmentation featuresGrasp interval preservationPlace interval preservation
Agent-view96.5%64.0%
Agent + wrist97.0%55.9%

Grasp interval preservation remains similar after view fusion. Place interval preservation decreases from 64.0% to 55.9%, while remaining above random segmentation matched to each demonstration's segment count, at approximately 49%.

These experiments evaluate multi-view retrieval and segmentation separately. A downstream policy benefit from view fusion has not been established by these diagnostics.

The multi-view analyses are reported in Appendix F.5 (Tables 34 and 35) of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.