- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
STAM aims to help robot policies make better use of visual information from earlier in a task. It maintains a coarse recurrent memory alongside a timestamped collection of fine-grained image patches, using a learned spatial-temporal recall map to retrieve relevant information when needed. It achieves strong results on MIKASA-Robo, improving the average success rate from 39.6% to 72.8% with π0.5 and from 38.6% to 66.6% with GR00T N1.5. The authors also test the method on several memory benchmarks and four real robot platforms. However, the paper does not clearly show whether the recall mechanism can filter out irrelevant information or how much memory is needed for longer trajectories.
Critical strengths
The paper has two main strengths. This method shows clear improvements across different benchmarks and robot policies, and the ablation studies provide useful evidence for the design of STAM.
1.The experimental results are strong across different tasks and models. STAM improves the success rate from 39.6% to 72.8% with π0.5 and from 38.6% to 66.6% with GR00T N1.5 on MIKASA-Robo. On MemMimic, the success rate increases from 82.1% to 94.2% with π0.5.
2.The ablation studies show the importance of the main components.The ablation studies show the importance of the main components. Removing fine memory leads to a large drop in performance, while using fixed-window or global recall also gives worse results. These findings support the use of fine-grained memory and adaptive recall in STAM.
Critical weaknesses
Two weaknesses remain: whether learned recall suppresses irrelevant history, and whether the growing patch buffer is practical over long episodes.
1.The authors did not test whether STAM can avoid using memory when it is not needed, or how it performs when the history contains irrelevant information. The OOD experiments only cover two tasks, and Flip Bottle only matches the baseline at 54.5%. Therefore, it is still unclear whether STAM can effectively decide when to ignore memory.
2.Each real-world robot task is tested only 11 times, which I think is a bit limited. Also, the image patch buffer grows as the episode gets longer, but the authors do not seem to report the peak memory usage or inference latency for long episodes. I think this is also an important part that should be evaluated.
Questions for authors
I still have some questions that I hope the authors can clarify. If these concerns are properly addressed with reasonable explanations and additional results, I would seriously consider increasing my rating.
1.What percentage of image patches have zero recall during evaluation? How does the recall time differ between memory-dependent tasks and cases where the history is irrelevant or disturbed? I hope the authors can provide some additional experiments to clarify this.
2.Could the authors provide experimental results with more trials for each real-world robot task, especially the two OOD tasks? This would help me better evaluate the stability and robustness of the method.
3.How do the peak memory usage of the image patch buffer and the end-to-end latency change as the episode gets longer? What happens to long-term recall and performance if the buffer size is limited? I hope the authors can provide more results and analysis on these questions.
I would carefully review them and consider increasing my rating.
2 comments
3. Peak buffer memory, inference latency and limited storage
Thank you for the question. We added Sec. 4.4, “Inference Efficiency,” and Fig. 7 to evaluate how memory usage and inference latency change with episode length and how limiting buffer capacity affects task performance. Even on a long-horizon task spanning 5,000 decision steps, peak buffer memory usage was only 8.3 GiB, well within the available memory. We observed no out-of-memory errors across the tested episode lengths. Peak usage was 1.54 GiB at 500 steps and 2.4 GiB at 1,000 steps. At each decision step, STAM stores 256 patch embeddings per camera view. With three views, a ViT hidden size of 1,152, and two bytes per BF16 value, this amounts to 3 × 256 × 1,152 × 2 = 1,769,472 bytes, or about 1.69 MiB per step.
Turning to inference efficiency, Fig. 7(a) shows that compiled inference latency on an NVIDIA A800 increased by only 7.8 ms over 5,000 decision steps, from 70.1 to 77.9 ms. STAM retrieves cached patch features directly, avoiding the cost of re-encoding past observations as the episode grows longer.
Figure 7(b) compares two storage strategies on SO-101 Tic-Tac-Toe with the same number of retained frames. A fixed-capacity sliding window evicts the oldest entry when full, while uniform temporal sampling stores patch features every k decision steps. At 60% frame retention, uniform sampling achieved 72.7% success and the sliding window achieved 23.0%, compared with 90.9% for the full buffer. Uniform sampling keeps observations from earlier stages of the task available for recall, even at a lower temporal resolution. This broader temporal coverage better preserved task performance under the same storage budget.
Thank you for your careful review and for asking when STAM uses history and how it performs over longer episodes. We address your questions in three parts. First, we show how often patches access memory, how far back they look, and how these choices change when historical cues are obscured. Second, we discuss the real-world evaluation, with particular attention to the two OOD tasks, Flip Bottle and Make Tea. Finally, we report how memory usage and inference latency change with episode length and how limiting buffer capacity affects long-horizon task performance.
1. Selective recall under clean and perturbed history
Thank you for asking when STAM leaves memory unused and how its recall changes when historical cues are disturbed. In response, we added a history-perturbation experiment to Sec. 4.2 and a new visualization in Fig. 5. We applied a dark gray circular mask to randomly selected historical frames while keeping the current observation unchanged. Figure 5 illustrates how patch-level recall can handle a local occlusion in a shell-game episode. If the mask obscures hand–cup contact in one frame, the patch at that location can retrieve evidence from another frame in the observed history where the contact is visible. Other patches can independently select different frames.
Table 4 quantifies how often STAM uses memory and how far back it looks. Under clean history, 99.4–100.0% of patch–timestep entries had zero recall on tasks where the current observation was sufficient, compared with 13.7–21.3% on history-dependent tasks. This helps interpret the Flip Bottle result you highlighted. STAM matched the original policy’s 54.5% success rate while leaving memory inactive for 99.4% of entries. On the rare occasions when a patch did access memory, its median look-back duration was only 0.13 s. On the three history-dependent tasks, the corresponding medians ranged from 7.67 to 14.17 s.
When we perturbed the history, the fraction of active patches increased in all six evaluations. On Flip Bottle, for example, it rose from 0.6% to 13.3%. On each of the three history-dependent tasks, STAM also looked further back, with median look-back durations reaching 15.31–22.40 s. Together, these results show that STAM rarely accessed history when the current observation was sufficient, but accessed memory across more patches when historical cues were obscured. Task success remained within 1.0 percentage point of the clean-history result in all six evaluations.
2. Additional real-world experiments
Thank you for the suggestion, we expanded our evaluation from 11 to 50 trials per real-world task. As you can see in the table, STAM with π0.5 succeeds in 31/50 trials (62.0%) on Flip Bottle and 27/50 (54.0%) on Make Tea. The original policy succeeds in 27/50 (54.0%) and 19/50 (38.0%), respectively. Average OOD success rises from 46.0% to 58.0%. Taken together, these results would support STAM’s robustness on tasks where history may be unreliable or only weakly relevant. The advantage appears on both tasks, with a larger benefit on Make Tea.
Group averages and individual OOD tasks. Task success rate (%).
Memory manipulation by capability. Task success rate (%).