acceptodds
Withdrawn from ICLR 2027

Hierarchical Replay Memory Construction for Continual Imitation Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Experience replay (ER) offers a simple and effective strategy for continual imitation learning, but its effectiveness depends critically on what is retained under a limited budget. Robot demonstrations provide complementary supervision across granularities: complete trajectories preserve execution context across task stages, recurring behaviors and transitions form shared structure across demonstrations, and local windows capture fine-grained state–action variation within these behaviors. Although prior work has explored data selection at different temporal scales, how to jointly organize them within replay memory remains underexplored. To address this gap, we propose Hierarchical Memory Retention (HMR), a hierarchical replay memory construction framework that progressively organizes retention from global context to local variation under a fixed budget. HMR first constructs a task-progress-aware behavioral structure by aligning demonstrations along task progress and clustering local action–state sequences, revealing recurring modes, stage-specific occurrences, and transitions. Guided by this structure, HMR retains a small set of complete demonstrations as contextual anchors and supplements them with local windows for reliable occurrences and transitions they underrepresent, establishing task-wide structural coverage. Remaining capacity is allocated according to behavioral frequency, event importance, and current coverage to preserve representative local variations. Extensive experiments on LIBERO task suites, together with cross-architecture and cross-domain evaluations, demonstrate consistent improvements over baseline methods under matched replay budgets, while ablations validate its key design choices.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Manipulation, Replay Memory Construction, Continual Imitation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WmCcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates how to efficiently store data for episodic replay in robot lifelong imitation learning. The convention is to store a few representative traces, but this work goes further by analyzing which “chunks” of demonstrations are more worthy of storing. And the proposed method stores a few representative demonstrations and then selects other chunks to mitigate the “weakness” of those demonstrations for each task.

Specifically, the anchor demonstrations are ranked by how many recurring behavioral changes they contain or how representative they are, the additional chunks are chosen to cover reliable behaviors and transitions that the anchors miss, and the remaining budget is filled with representative and diverse chunks from each behavioral group.

As a result, with a fixed 1000-demonstration budget, the proposed method works better. With two VLAs, it improves final success by about 5.5 points over prefix-based ER, the only VLA baseline.

Critical strengths
  1. The method only changes which chunks are stored for ER, so it is easy to plug in. It works with both a small BCTransformer trained from scratch and pretrained VLAs, and it explicitly keeps “rare” behaviors (e.g., regrasps and transitions between stages) that simply keeping the first or random chunks can miss.

  2. The gains are consistent. With BCTransformer, it achieves the best forward transfer and AUC on all four LIBERO suites, including over GSS, the strongest baseline (+3.5 AUC on average). With two VLAs, it also improves final success over ER by about 5.5 points.

Critical weaknesses

A simple baseline is missing: just keep 1,000 random chunks from each task. How does that perform? In other words, which design in the proposed method is the most critical?

The method is quite complicated, and its components are not clearly necessary. It relies on many hand-designed steps and hyperparameters, yet uniform filling beats the proposed filling on LIBERO-Spatial, and anchors add little on LIBERO-Goal. In addition, the anchor ratios used in the main table match the best ablation settings, which were computed on two of the three evaluation seeds.

In general, I feel that more ablations to understand which parts make the proposed method work, and only highlighting a few top contributors, would be better than presenting a complex method with many different small updates. I believe this is a very practical method, and data selection (or coreset selection, which is already quite a small field in continual learning) is what people would actually use in practical continual learning. However, the current work may mix too many things, and the insights are less clear.

Questions for authors

See weakness.

Reviewer_PmPiOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Hierarchical Memory Retention (HMR), a replay-memory construction framework for continual imitation learning under a fixed memory budget. The main motivation of this paper is that robot demonstrations contain complementary information at different temporal granularities, including complete execution context, recurring behavioral structures, and local state-action variations. HMR first identifies temporal behavioral structures by aligning demonstrations using dynamic time warping, clustering local state-action sequences into behavioral modes, and stage-specific occurrences. Based on the discovered structures, the method constructs replay memory in three stages: (1) selecting complete demonstrations as contextual anchors, (2) adding local chunks to cover underrepresented behavioral occurrences and transitions, and (3) allocating the remaining memory capacity across behavioral groups while selecting representative and complementary chunks. Experiments on LIBERO and RoboTwin-CL evaluate the proposed method with BCTransformer, Qwen-GR00T, and Qwen-OFT. The results show improvements over the evaluated replay baselines in several continual-learning metrics.

Critical strengths

S1. A multi-granularity perspective on replay-memory construction. The paper explores replay-memory construction from multiple temporal granularities, a perspective that has received increasing attention in recent robot data curation research but remains relatively underexplored in how to set replay data. The integration of trajectory-level context, behavioral structure, and local variation provides a reasonable direction for improving replay-memory selection.

S2. Evaluation across multiple policy architectures and benchmarks. The experiments cover four LIBERO suites and RoboTwin-CL, using BCTransformer and two VLA-based policies. The reported results generally show improvements over the evaluated baselines, suggesting potential applicability across different policy architectures and manipulation settings.

S3. Detailed implementation descriptions and component-level evaluations. The appendix provides very detailed implementation descriptions and mathematical formulations of the proposed framework. The paper also includes component-level ablations examining anchor proportions, structural coverage, and representative memory completion, which help assess the contributions of the main components.

Critical weaknesses

W1. Limited empirical support for hierarchical replay retention. The paper argues that full trajectories contain redundant supervision, while local chunks may lose cross-stage connections (Lines 42-49 and 76-81). However, the evidence comes mainly from the schematic in Figure 1(a). An analysis of real demonstrations would help establish how much behavioral redundancy exists, whether differences are concentrated in short segments, and whether these patterns hold across tasks. The paper also leaves unclear why preserving cross-stage transitions is particularly important for continual imitation learning. Although the structural-coverage ablation in Figure 5 shows gains, it does not separate the effect of transition coverage from occurrence coverage or other changes in the retained data.

W2. Limited validation of the discovered behavioral structures. Section 3.2 relies on several heuristic choices, including prototype averaging, DTW alignment, clustering, and peak detection. Although Appendix A.2 provides detailed implementation descriptions, the rationale and empirical validity of several key choices remain unclear.

  1. The method constructs a common task-progress prototype by averaging demonstration descriptors. It is unclear whether such averaging produces a meaningful reference when demonstrations follow different execution strategies or contain different behavioral sequences.
  2. DTW alignment is used to establish task-progress correspondence across demonstrations, but the paper does not verify whether matched regions actually correspond to the same semantic behavioral stages.
  3. The discovered behavioral modes and stage-specific occurrences are not sufficiently validated through qualitative examples or quantitative structural analyses.

W3. The experimental comparisons lack recent and closely related methods. These baselines do not sufficiently represent recent developments in structure-aware data selection and replay for robot learning. For example, the authors should consider including the following recent methods in the comparison:

  1. Yuheng Lei, Sitong Mao, Shunbo Zhou, Hongyuan Zhang, Xuelong Li, and Ping Luo. 2026. Dynamic Mixture of Progressive Parameter-Efficient Expert Library for Lifelong Robot Learning. Trans. on Machine Learning Research (2026).
  2. Kaushik Roy, Giovanni D’urso, Nicholas Lawrance, Brendan Tidd, and Peyman Moghadam. 2026. SPREAD: Subspace Representation Distillation for Lifelong Imitation Learning. In 2026 IEEE International Conference on Robotics and Automation (ICRA). IEEE.
  3. Fanqi Yu, Matteo Tiezzi, Tommaso Apicella, Cigdem Beyan, and Vittorio Murino. 2026. Lifelong imitation learning with multimodal latent replay and incremental adjustment. In IEEE/CVF CVPR. 6740–6749.
  4. Xudong Wang, Zebin Han, Zhiyu Liu, Gan Li, Jiahua Dong, Baichen Liu, Lianqing Liu, and Zhi Han. 2026. Lifelong language-conditioned robotic manipulation learning. In AAAI, Vol. 40. 18629–18637.

W4. Limited discussion of alternative replay representations. For example, feature-based replay approaches (Dynamic mixture of progressive parameter-efficient expert library for lifelong robot learning, TMLR 2026; Lifelong imitation learning with multimodal latent replay and incremental adjustment, CVPR 2026), which retain compact learned representations instead of original observations, are not sufficiently discussed. Besides, there are also many replay-free works not discussed, such as the papers proposed in W3.

W5. Limited analysis of what the replay memory preserves. The paper claims that its three-stage procedure preserves task context, structural coverage, and local variations, but the experiments mainly measure downstream policy performance. Figure 2 illustrates the intended process without showing what is retained in practice. A real-task case study would help clarify which behavioral occurrences and transitions the demonstration anchors cover, which ones they miss, and which are recovered by the structural--coverage chunks.

Questions for authors

I have some questions about this paper, and it would really help me understand the idea and model design if the authors could clarify them:

  1. In lines 42-49 and 76-81, could the authors clarify the meaning of "nonuniform demonstrations", "intermediate states", and "execution context"? In particular, what evidence supports the claim that demonstrations sharing similar behavioral sequences differ mainly in short local segments?

  2. In Section 3.2, how robust is the learned task-progress prototype to demonstrations following substantially different behavioral sequences? Could the authors visualize DTW alignments across multiple real demonstrations to show whether aligned regions correspond to meaningful task stages?

  3. In Section 3.3, could the authors report structural coverage before and after adding coverage chunks, including the number of missing and recovered occurrences and transitions?

  4. In Section 4.1, why are more recent structure-aware, phase-aware, feature replay approaches excluded from the experimental comparisons?

Reviewer_H6mDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies replay memory construction for continual imitation learning under a fixed per-task budget measured in training chunks. The proposed HMR first discovers a task-specific temporal behavioral structure: demonstrations are aligned to a progress prototype by DTW, local chunks are clustered by constrained DTW into behavioral modes, each mode is split into stage-specific occurrences by peaks in its aligned-progress distribution, and observed transitions between these groups form a directed graph. Guided by this structure, HMR builds memory in three stages: a few complete demonstrations are kept as anchors, chosen by a stability-tested comparison of event-based and central-diverse rankings; a greedy capped-coverage step (monotone submodular) adds chunks for reliable occurrences and transitions underrepresented by the anchors; and a completion step allocates the remaining budget across groups by a diminishing-marginal-utility rule and selects chunks by medoid proximity, importance, diversity, and demonstration and temporal spread.

Experiments use BCTransformer on the four LIBERO suites with b = 1,000 chunks per task, comparing against prefix-based ER, GSS, and DemInf. HMR obtains the highest mean FWT and AUC on all four suites, improving average FWT and AUC over ER by 7.15 and 12.95 points and over GSS by 2.65 and 3.52 points.

Critical strengths
  • How to divide a fixed replay budget among complete trajectories, recurring behavioral structure, and local windows is a relevant and underexplored question for continual imitation learning, and the anchor, coverage, and completion stages address it in an intuitive order. HMR achieves the highest mean FWT and AUC on all four LIBERO suites, and its gains over ER carry over to two VLA policies and a bimanual benchmark (e.g., Qwen-GR00T on LIBERO-Long, NBT 19.15 to 8.34).
  • The appendix specifies every construction step, constant, tie-breaking rule, and baseline adaptation, and it openly states limitations such as future-task normalization statistics, the extra data access of DemInf, and the restricted scope of the two propositions. Together with task-wise curves for every suite, this level of detail makes the method reproducible and the claims checkable.
Critical weaknesses
  • Because the budget is matched in overlapping stride-one chunks rather than in retained data, ER and DemInf keep roughly 1,000 unique frames per task, whereas the temporal-gap rules of HMR likely retain substantially more. Without reported frame counts, the gains may come from broader state coverage rather than from the hierarchy.
  • The baselines are weak and incomplete: the main ER uses prefix selection that keeps only the first few demonstrations, and random or evenly spaced chunk sampling is missing, even though Uniform completion already matches or exceeds representative completion (Spatial AUC 68.24 vs. 67.14, lower NBT on both suites). The VLA and RoboTwin-CL settings compare only against this ER, and PHASER is not evaluated.
  • Evidence of reduced forgetting is limited, as gains over GSS fall within one standard deviation in five of eight FWT and AUC comparisons and HMR forgets more than GSS and ER on LIBERO-Long (NBT 24.31 vs. 19.97 and 23.45). In every ablation, some reduced variant has lower NBT than the full method, suggesting that the stages aid acquisition rather than retention.
  • Support for complete-demonstration anchors is weak, since they add only 0.56 AUC on Goal (66.82 vs. 66.26) and the per-suite anchor ratios match the best ablation settings, suggesting tuning on evaluation tasks. Many hand-set constants, a single budget and task order, and no broader sensitivity analysis also leave the generality of the gains unclear.
Questions for authors
  • Could you report the unique frames retained per task by each method and add random or evenly spaced chunk ER baselines compared at matched unique-frame budgets?
  • Could GSS, DemInf, random or uniform chunk ER, and PHASER also be evaluated in the Qwen-GR00T and RoboTwin-CL settings, or could you explain why PHASER is not applicable?
  • Why does HMR forget more than ER and GSS on LIBERO-Long, and could you report final average success for BCTransformer and explain how replay selection alone raises FWT by up to 7 points?
  • Are the modes and route proportions in Figures 1 and 2 derived from real LIBERO data, and how well do the discovered modes align with meaningful events such as gripper state changes?
  • Given that BCTransformer consumes at most ten observations, what supervision does a complete-demonstration anchor provide beyond the same number of overlapping chunks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.