acceptodds
Under review as a conference paper at ICLR 2027

GROW: Grounding with Long-Term Awareness for Skill-Incremental Robotic Manipulation

Abstract

Skill-incremental robotic manipulation requires continuously acquiring new skills while preserving previously learned capabilities. A widely adopted paradigm formulates manipulation as keyframe-wise action prediction, where the policy predicts the next keyframe action from the current observation and instruction. Applying skill-incremental learning to such keyframe-based manipulation introduces a fundamental level mismatch: local keyframe supervision and replay provide limited explicit guidance for preserving the long-term semantics of complete skill execution. To address this discrepancy, we propose **GROW** (**Gr**ounding with L**o**ng-Term A**w**areness), a continual manipulation framework that incorporates long-horizon action semantics across perception, action prediction, and replay, enabling skill-level semantics to be better captured and preserved across stages. At the perception level, GROW introduces dual-horizon grounding to regularize cross-modal attention toward both immediate interaction targets and long-horizon task goals. An auxiliary action chain predictor further captures how each skill evolves toward completion through instruction-conditioned future action prediction. The resulting action-chain predictions are additionally exploited for chain-based replay herding, enabling replay selection based on trajectory-level skill semantics. Experiments on RLBench simulation and real-world manipulation tasks show that GROW consistently improves performance over prior skill-incremental learning approaches. Code will be released upon acceptance.

Primary area:
applications to robotics, autonomy, planning
Keywords:
incremental learning, multi-task robotic manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HJAROpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper talk about keyframe-based skill-incremental manipulation suffers from a level mismatch. Learning and replay operate on individual keyframes, while the knowledge to preserve spans entire skill executions. It proposes GROW, which adds long-horizon cues through dual-horizon attention grounding, an auxiliary action-chain predictor, and chain-based replay herding. Result gains over iManip and other baselines on RLBench and a real robot.

Critical strengths

The motivation is clear and well illustrated. The close the blue or red jar example concretely shows why local next-keyframe supervision can lose instruction-conditioned context.

The method is lightweight and self-supervised. All extra supervision comes from existing demonstrations, with no external models, and it plugs into multiple PerceiverIO backbones.

Evaluate on three backbones, five SIL settings, two ablation regimes, real-robot experiments.

Critical weaknesses

Table 8 shows large gains even when GROW’s components act only during base pretraining, and in the real-world study GROW already reaches 60% on the base task versus 20% for both baselines before any incremental update. This suggests much of the improvement comes from a stronger base representation rather than reduced forgetting. Results under joint multi-task training, and forgetting measured relative to each method’s own base, would be needed to support the SIL-specific claim.

The goal anchor is only the terminal end-effector position, and the action chain predicts translations only, which says little about skill progression in tasks like push buttons or stack blocks. The method is therefore closer to auxiliary future-waypoint prediction than to modeling skill-level semantics.

The tables show large unexplained variance, e.g., non-monotonic σ sensitivity in Table 11 and an ablation in Table 2 outperforming full GROW on New tasks.

Questions for authors

Please provide complete specifications of the real-world hardware: the exact name, model, manufacturing year, and production batch of the robot and sensors.

If the paper is accepted, can you guarantee that the trained models will be publicly released before the conference? Will all reported results be fully reproducible from the released materials?

Please provide the complete set of parameters (if you set potential hyperparametersto the machine) used for the experiment shown in the supplementary video grow_3_slide_the_block_onto_the_red_square.mp4, to support reproducibility.

Reviewer_B8fuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GROW addresses skill-incremental robotic manipulation using keyframe-based policies. The framework combines three components: attention grounding toward the next and terminal end-effector positions; an auxiliary, instruction-conditioned predictor of future keyframe positions whose features support immediate action prediction; and replay selection using pooled action-chain predictions. Evaluation covers ten RLBench skills, several incremental configurations, three related PerceiverIO backbones, and three real-world skills. Average overall success improves over the authors’ iManip reimplementation by 3.5, 3.8, and 10.7 percentage points for PerAct, GNFactor, and ManiGaussian, respectively.

Critical strengths
  • GROW connects perception, action prediction, and replay selection through future-action information already available in demonstrations. This avoids requiring separate grounding annotations or external perception models
  • Table 1 shows improvements in average overall success across all three backbones, with the largest gain on ManiGaussian: 32.3% vs 21.6% for reproduced iManip.
  • Tables 2 and 9 provide useful component and update-mechanism comparisons, while the real-world experiments establish practical feasibility.
Critical weaknesses
  • The ablations establish useful component contributions, but they do not fully distinguish long-horizon knowledge preservation from stronger initial representations, generic auxiliary supervision, or additional predictive capacity.
  • TAIL provides weaker corroboration: Appendix A.2.2 averages performance across all available experts rather than selecting the corresponding expert. My concern is that this introduces an expert-selection penalty even when individual experts retain their capabilities; original TAIL activates the corresponding task adapter. The changed setting needs clearer qualification.
  • The iManip reproduction gap is also consequential. The authors transparently acknowledge it and improve the reproduced baseline’s learning rate, which is helpful. Nevertheless, incremental training stops at 50,000 iterations, while the original iManip study reports approximately 60,000 iterations to convergence in its relevant freezing experiment and trains that experiment for 100,000 iterations.
  • Removing the chain predictor changes multiple functions together. A matched predictor restricted to the next keyframe would more directly test whether future-horizon information supplies the benefit. Table 9 helps address generic update choices, but does not provide a TIB control with exactly GROW’s stage-wise LoRA and calibration procedure.
Questions for authors
  • Were the hyperparameters discussed in Appendix C selected using separate validation episodes, or using the episodes reported in the final results?
  • Does the advantage over iManip persist when its reproduced implementation is trained to convergence, including a longer incremental schedule?
  • Can you provide independent training runs and uncertainty estimates for the principal comparison, particularly the smaller PerAct and GNFactor gains?
  • Does GROW outperform a matched auxiliary predictor trained only on the next keyframe, with comparable parameter count, language conditioning, and feature injection? Does its advantage persist against TIB using the identical stage-wise LoRA, merging, and calibration schedule?
  • Does Equation 12 pool raw logits, normalized probabilities, or decoded coordinates? How is the distance representation normalized?
Reviewer_WkiBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GROW addresses skill-incremental robotic manipulation by adding grounding supervision and future keyframe position prediction to existing policies. It also uses the predicted action chains to select representative replay samples.

Critical strengths

1,The motivation is clear: isolated keyframes may not preserve enough information about how a skill progresses toward completion. 2,The three components fit this motivation reasonably well, and the ablations help explain their contributions. 3,The improvements across three backbones are encouraging. On ManiGaussian, average overall success across incremental stages improves from 21.6% to 32.3% over iManip.

Critical weaknesses

1,Some implementation details need clarification. In particular, I would like more detail on keyframe extraction, replay budget allocation, and whether the same keyframe index corresponds to comparable execution stages across demonstrations.

2,The training pipeline has several moving parts. Besides the proposed components, it includes replay, distillation, LoRA merging, and calibration. A clearly matched replay-plus-distillation-and-LoRA baseline, together with training overhead measurements, would help justify the added complexity. Replay budgets should also account for the extra future-position labels.

3,The real-world evidence is limited. There are only three skills and ten rollouts per evaluation. GROW also starts with much higher base-task performance, making it harder to separate better initial learning from reduced forgetting. Repeated runs and explicit forgetting metrics would strengthen this analysis.

Questions for authors

See above

Comments on the paper

1 comment

Dear Reviewers,

Please review the authors' rebuttal and the other reviewers' comments, and participate in the discussion before it closes on November 18 (AoE). Kindly indicate whether your concerns have been adequately addressed and raise any remaining questions or concerns.

Thank you for your time and contributions.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.