- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
OnCue handles the case where what the policy needs has already left the frame. Memory is an ordered list of finished subtasks in language, plus a bank of visual evidence keyed by subtask index: the first frame of each subtask, and crops and boxes of the objects the planner flagged. A LoRA-tuned Qwen3-VL planner reads the current image, the goal and that list, then returns the next subtask, which past indices to pull and what to keep. Retrieved features enter a DiT policy through a cross-attention branch that is skipped when nothing is retrieved, and a small classifier decides when a subtask is done. Reported numbers: 84.0% on RMBench, 95.8% on RoboTwin-MeM, 16.2 and 19.8 points over the strongest baseline, 88.3% on three real tasks across two arms. The claim underneath is that indexing evidence by subtask lets the planner pull only what the current step needs.
Critical strengths
- The two memories fail on different tasks, which is the right evidence for the complementarity claim. Pick Objects in Order drops to 0 without language memory, Observe and Pick Up to 3 without visual memory.
- Coverage is decent for this area: two simulators at 100 trials, real hardware on two platforms, and a failure taxonomy most papers skip.
Critical weaknesses
- Table 1 confounds too much. OnCue brings an 8B LoRA-tuned planner, SAM3 and DINOv3, trained per task on supervision nobody else gets: re-segmented subtask boundaries, retrieval indices, object boxes, none of which is explained. The 16.2 points could be the routing idea or could be the annotation. Same planner, same labels, naive memory would settle it.
- Selectivity is the thesis and is never ablated. Both variants in 4.3.1 simply delete a memory type. Missing are R_k set to every completed subtask, a fixed recent window, and a random set of matched size. The retrieval labels already exist, so precision and recall on the predicted indices would cost nothing and would show whether the planner routes or has memorised a per-task pattern.
- No timing anywhere, for an 8B planner plus SAM3 plus DINOv3 plus a DiT at every boundary, on platforms recorded at 15 and 30 Hz. RoboTwin-MeM episodes run past 1500 steps and every allowed index goes into the prompt, yet there is no curve for success rate against memory size.
- Everything is seen at 224 by 224, and the regions worth keeping are chosen by the 8B VLM and then tidied by SAM3. D.2 concedes that Pick Double X Times suffers because the real digits are much smaller than in simulation, which is a sampling problem in the current frame rather than a storage problem. Worth at least discussing spatially dynamic computation here, for instance AdaptiveNN (Wang et al., Emulating human-like adaptive vision for efficient and flexible machine visual perception, Nature Machine Intelligence 2025), which treats perception as coarse-to-fine fixations on task-relevant regions and stops once the evidence suffices, at up to 28x lower inference cost. A fixation policy is another answer to which regions matter, and could supply higher-resolution crops without a per-task 8B model in the loop. Not a condition of acceptance, but active vision is absent from 2.2.
- Single seed, no variance, ablations on two or three tasks; 96 against 94 is not a result. The w/o Classifier row at 0 everywhere is a broken control rather than an informative one. Also, Figures 3, 5, 6, 9 and 10 are tables that the text calls Tables, colliding with the appendix numbering; 4.4 ends mid-sentence; Mem-0 and RMBench share a citation.
Questions for authors
See Weakness
4 comments
Response to Reviewer WqW3 [2/2]
Q3:Clarification of Real-World Perception Challenges and Adaptive Vision
We appreciate the reviewer's insightful suggestion regarding adaptive visual perception. We agree that fine-grained visual recognition and historical memory retrieval address complementary challenges in robotic manipulation.
In Pick Double X Times, the numbered blocks used in our real-world experiments are considerably smaller than those in simulation, making digit recognition more challenging. This scale difference makes fine-grained digit recognition more challenging, independently of historical memory retrieval.
We also acknowledge the relevance of adaptive visual perception methods such as AdaptiveNN, which employ coarse-to-fine processing to focus on task-relevant regions. Such approaches are complementary to OnCue and offer a promising direction for improving fine-grained visual evidence acquisition.
We have clarified this perception challenge and incorporated a discussion of AdaptiveNN in the revised Real-World Evaluation section.
Q4: Evaluation Seeds, Classifier Controls, and Presentation Corrections
We thank the reviewer for raising concerns regarding evaluation reliability and the design of our ablation studies. First, we would like to clarify that each simulation task is evaluated across 100 distinct random seeds, following the benchmark-specific evaluation protocols of RMBench and RoboTwin-MeM. We have explicitly emphasized this setting in the revised paper to avoid any misunderstanding.
We also acknowledge that the original completion classifier ablation was not sufficiently informative, as the w/o Classifier variant completely disabled memory updates. To address this limitation, we have replaced it with two more meaningful fixed-interval update strategies: Subtask-specific Median, which uses the median execution length of the closest matching subtask type in the training set as the update interval, and Global Average, which uses a fixed interval computed from the average ratio of total execution steps to the number of subtasks in the training set. Both strategies retain the same Planner and VLA as OnCue, differing only in the memory update triggers. We have updated the corresponding ablation study in the paper and provided further implementation details in the Appendix.
| Update Strategy | Press Button | Rearrange Blocks | Cover Blocks |
|---|---|---|---|
| Oracle Completion | 96% | 99% | 99% |
| OnCue | 87% | 93% | 96% |
| Subtask-specific Median | 81% | 90% | 95% |
| Global Average | 45% | 70% | 84% |
As shown in the table, the performance gap between Oracle Completion and OnCue highlights the potential for further improvement in completion detection, while OnCue consistently outperforms both fixed-interval update strategies across all three tasks. Notably, Global Average performs substantially worse on Press Button and Rearrange Blocks, where subtask execution lengths vary considerably, making uniform update intervals more prone to premature transitions. These results suggest that completion-driven memory updates provide more appropriate update timing than predefined fixed intervals, thereby improving task performance.
Finally, we thank the reviewer for identifying several presentation and citation issues in our paper. We have corrected the reported formatting inconsistencies, table references, and incomplete sentence in the revised version. We would also like to clarify that Mem-0 was introduced in the RMBench paper, and therefore sharing the same citation is intentional and appropriate. We have clarified this relationship in the revised paper to avoid confusion.
I thank the authors for their detailed responses! After reading them in detail, I decided to maintain my positive score.
Thank you very much for your thoughtful follow-up and constructive feedback throughout the review process. We sincerely appreciate your time, careful consideration, and positive assessment of our work.
Response to Reviewer WqW3 [1/2]
We sincerely thank Reviewer WqW3 for the constructive feedback and detailed suggestions on experimental design, ablation studies, computational efficiency, and evaluation reliability. Following these suggestions, we have strengthened our ablation studies, clarified the experimental settings, and provided additional analyses. Below, we present detailed responses and additional experimental evidence. All changes in the revised paper have been highlighted in red for easy reference.
Q1: Effectiveness of Selective Historical Memory Retrieval
We appreciate the reviewer's constructive feedback on the effectiveness of selective historical memory retrieval. We agree that further evaluation is important for understanding its contribution to OnCue.
To this end, we have conducted additional ablation experiments comparing OnCue with four alternative retrieval strategies: (1) Oracle Source, which uses ground-truth historical source indices, object bounding boxes, and corresponding object crops; (2) Recent-k, which retrieves the most recent k historical memories; (3) Random-k, which randomly selects k historical memories; and (4) All History, which provides all available historical memories without selective retrieval. For these variants, the corresponding Planner and VLA are retrained under their respective retrieval configurations. We set k = 2 for Swap T and k = 3 for Pick Objects in Order.
As shown in the table, OnCue achieves performance comparable to Oracle Source, whereas Recent-k, Random-k, and All History exhibit substantial performance degradation. These results suggest that the Planner can effectively identify task-relevant historical sources and that selective retrieval provides greater benefits than simply using recent, random, or all available historical memories.
Furthermore, our ablation studies on memory representation, memory granularity, completion-driven updates, and historical retrieval provide complementary evidence supporting the effectiveness of OnCue's key components and design choices.
These additional experiments and analyses have been incorporated into the revised paper, with detailed implementation settings provided in the Appendix.
Q2: Inference Efficiency and Execution Protocol
We appreciate the reviewer's concerns regarding inference efficiency and memory scalability. We provide a component-wise inference time comparison between OnCue and a variant without Visual Memory.
As shown in the table, Visual Memory introduces additional encoding and action-prediction overhead. Each DiT inference generates 30 actions, while the Planner is invoked only at subtask transitions rather than at every action prediction. OnCue achieves an execution frequency of 1.93 Hz, compared with 2.38 Hz without Visual Memory.
Regarding memory scalability, OnCue does not create a new memory index at every execution step. Instead, memory indices and updates operate at the subtask level. Therefore, the number of memory entries depends on task decomposition rather than the total execution steps. For example, the number of subtask indices in Press Button does not exceed 20.
We have clarified the inference efficiency, execution protocol, and memory indexing mechanism in the revised paper.