Remember What Matters: On-Demand Memory for Long-Horizon Robotic Manipulation

Memory is essential for long-horizon robotic manipulation: robots must remember task progress and retrieve historical visual details. The key lies in accurately associating the current task with relevant memory. We propose OnCue, a hierarchical memory framework that supports task-guided recall. Language descriptions record task progress, while subtask indices address the corresponding historical visual memories; scene-level and object-level memory preserve scene layout, object appearance, and location. The high-level planner generates the current subtask, selects relevant completed stages for visual-memory retrieval, and predicts bounding boxes around task-relevant objects in the current observation for future memory storage; the low-level executor combines the current image, subtask, and retrieved visual memory to execute manipulation. We evaluate OnCue on RMBench and RoboTwin-MeM, achieving average success rates of 84.0% and 95.8%, respectively, outperforming the strongest baseline by 16.2 and 19.8 percentage points. We also evaluate on three tasks across two real robot platforms, achieving an average success rate of 88.3%. We further conduct ablation studies and demonstrate the importance of memory design for long-horizon robotic manipulation.