acceptodds
Under review as a conference paper at ICLR 2027

STAM: Spatio-Temporal Adaptive Coarse-Fine Visual Memory for Generalist Robot Policies

Abstract

Generalist robot policies need historical visual evidence when the current observation is insufficient for action selection. However, compressing history can discard useful details, while indiscriminate history conditioning can introduce irrelevant information. We propose STAM, a plug-in spatio-temporal adaptive coarse-fine visual memory module for generalist robot policies. Its coarse-fine visual memory combines compressed historical context with directly retrievable visual details. A spatio-temporal recall map determines which image patches access memory and how far back to retrieve fine visual evidence. Layerwise coarse-fine memory fusion integrates both memory sources into the selected patches without modifying the downstream policy backbone. An action-guided three-stage training procedure learns recall decisions from action errors without manual recall-map annotations. Experiments across four simulation benchmarks and four real-world robot platforms demonstrate improved memory-dependent manipulation while maintaining strong general manipulation performance. On MIKASA-Robo, STAM improves average success by 33.2 and 28.0 percentage points over the original and GR00T N1.5 policies, respectively. Code will be available after acceptance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Robot manipulation, robot vision

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_tBioOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

STAM aims to help robot policies make better use of visual information from earlier in a task. It maintains a coarse recurrent memory alongside a timestamped collection of fine-grained image patches, using a learned spatial-temporal recall map to retrieve relevant information when needed. It achieves strong results on MIKASA-Robo, improving the average success rate from 39.6% to 72.8% with π0.5 and from 38.6% to 66.6% with GR00T N1.5. The authors also test the method on several memory benchmarks and four real robot platforms. However, the paper does not clearly show whether the recall mechanism can filter out irrelevant information or how much memory is needed for longer trajectories.

Critical strengths

The paper has two main strengths. This method shows clear improvements across different benchmarks and robot policies, and the ablation studies provide useful evidence for the design of STAM.

1.The experimental results are strong across different tasks and models. STAM improves the success rate from 39.6% to 72.8% with π0.5 and from 38.6% to 66.6% with GR00T N1.5 on MIKASA-Robo. On MemMimic, the success rate increases from 82.1% to 94.2% with π0.5.

2.The ablation studies show the importance of the main components.The ablation studies show the importance of the main components. Removing fine memory leads to a large drop in performance, while using fixed-window or global recall also gives worse results. These findings support the use of fine-grained memory and adaptive recall in STAM.

Critical weaknesses

Two weaknesses remain: whether learned recall suppresses irrelevant history, and whether the growing patch buffer is practical over long episodes.

1.The authors did not test whether STAM can avoid using memory when it is not needed, or how it performs when the history contains irrelevant information. The OOD experiments only cover two tasks, and Flip Bottle only matches the baseline at 54.5%. Therefore, it is still unclear whether STAM can effectively decide when to ignore memory.

2.Each real-world robot task is tested only 11 times, which I think is a bit limited. Also, the image patch buffer grows as the episode gets longer, but the authors do not seem to report the peak memory usage or inference latency for long episodes. I think this is also an important part that should be evaluated.

Questions for authors

I still have some questions that I hope the authors can clarify. If these concerns are properly addressed with reasonable explanations and additional results, I would seriously consider increasing my rating.

1.What percentage of image patches have zero recall during evaluation? How does the recall time differ between memory-dependent tasks and cases where the history is irrelevant or disturbed? I hope the authors can provide some additional experiments to clarify this.

2.Could the authors provide experimental results with more trials for each real-world robot task, especially the two OOD tasks? This would help me better evaluate the stability and robustness of the method.

3.How do the peak memory usage of the image patch buffer and the end-to-end latency change as the episode gets longer? What happens to long-term recall and performance if the buffer size is limited? I hope the authors can provide more results and analysis on these questions.

I would carefully review them and consider increasing my rating.

2 comments

Thank you for your careful review and for asking when STAM uses history and how it performs over longer episodes. We address your questions in three parts. First, we show how often patches access memory, how far back they look, and how these choices change when historical cues are obscured. Second, we discuss the real-world evaluation, with particular attention to the two OOD tasks, Flip Bottle and Make Tea. Finally, we report how memory usage and inference latency change with episode length and how limiting buffer capacity affects long-horizon task performance.

1. Selective recall under clean and perturbed history

Thank you for asking when STAM leaves memory unused and how its recall changes when historical cues are disturbed. In response, we added a history-perturbation experiment to Sec. 4.2 and a new visualization in Fig. 5. We applied a dark gray circular mask to randomly selected historical frames while keeping the current observation unchanged. Figure 5 illustrates how patch-level recall can handle a local occlusion in a shell-game episode. If the mask obscures hand–cup contact in one frame, the patch at that location can retrieve evidence from another frame in the observed history where the contact is visible. Other patches can independently select different frames.

Table 4 quantifies how often STAM uses memory and how far back it looks. Under clean history, 99.4–100.0% of patch–timestep entries had zero recall on tasks where the current observation was sufficient, compared with 13.7–21.3% on history-dependent tasks. This helps interpret the Flip Bottle result you highlighted. STAM matched the original policy’s 54.5% success rate while leaving memory inactive for 99.4% of entries. On the rare occasions when a patch did access memory, its median look-back duration was only 0.13 s. On the three history-dependent tasks, the corresponding medians ranged from 7.67 to 14.17 s.

When we perturbed the history, the fraction of active patches increased in all six evaluations. On Flip Bottle, for example, it rose from 0.6% to 13.3%. On each of the three history-dependent tasks, STAM also looked further back, with median look-back durations reaching 15.31–22.40 s. Together, these results show that STAM rarely accessed history when the current observation was sufficient, but accessed memory across more patches when historical cues were obscured. Task success remained within 1.0 percentage point of the clean-history result in all six evaluations.


2. Additional real-world experiments

Thank you for the suggestion, we expanded our evaluation from 11 to 50 trials per real-world task. As you can see in the table, STAM with π0.5\pi_{0.5} succeeds in 31/50 trials (62.0%) on Flip Bottle and 27/50 (54.0%) on Make Tea. The original policy succeeds in 27/50 (54.0%) and 19/50 (38.0%), respectively. Average OOD success rises from 46.0% to 58.0%. Taken together, these results would support STAM’s robustness on tasks where history may be unreliable or only weakly relevant. The advantage appears on both tasks, with a larger benefit on Make Tea.

Group averages and individual OOD tasks. Task success rate (%).

MethodGeneralOODMemoryFlip BottleMake Tea
π0\pi_066.037.068.854.020.0
π0\pi_0-FAST26.025.068.044.06.0
GR00T71.328.058.428.028.0
π0.5\pi_{0.5}71.346.070.054.038.0
Pi-Mem*71.337.073.246.028.0
STAM (π0.5\pi_{0.5})81.358.092.862.054.0
STAM (GR00T)78.746.085.648.044.0

Memory manipulation by capability. Task success rate (%).

MethodTarget recallOcclusion recoveryOrigin recallCount memoryProgress memory
π0\pi_058.078.082.054.072.0
π0\pi_0-FAST46.080.078.072.064.0
GR00T38.062.080.052.060.0
π0.5\pi_{0.5}42.082.088.070.068.0
Pi-Mem*50.080.090.072.074.0
STAM (π0.5\pi_{0.5})76.096.0100.098.094.0
STAM (GR00T)68.088.098.082.092.0

3. Peak buffer memory, inference latency and limited storage

Thank you for the question. We added Sec. 4.4, “Inference Efficiency,” and Fig. 7 to evaluate how memory usage and inference latency change with episode length and how limiting buffer capacity affects task performance. Even on a long-horizon task spanning 5,000 decision steps, peak buffer memory usage was only 8.3 GiB, well within the available memory. We observed no out-of-memory errors across the tested episode lengths. Peak usage was 1.54 GiB at 500 steps and 2.4 GiB at 1,000 steps. At each decision step, STAM stores 256 patch embeddings per camera view. With three views, a ViT hidden size of 1,152, and two bytes per BF16 value, this amounts to 3 × 256 × 1,152 × 2 = 1,769,472 bytes, or about 1.69 MiB per step.

Turning to inference efficiency, Fig. 7(a) shows that compiled inference latency on an NVIDIA A800 increased by only 7.8 ms over 5,000 decision steps, from 70.1 to 77.9 ms. STAM retrieves cached patch features directly, avoiding the cost of re-encoding past observations as the episode grows longer.

Figure 7(b) compares two storage strategies on SO-101 Tic-Tac-Toe with the same number of retained frames. A fixed-capacity sliding window evicts the oldest entry when full, while uniform temporal sampling stores patch features every k decision steps. At 60% frame retention, uniform sampling achieved 72.7% success and the sliding window achieved 23.0%, compared with 90.9% for the full buffer. Uniform sampling keeps observations from earlier stages of the task available for recall, even at a lower temporal resolution. This broader temporal coverage better preserved task performance under the same storage budget.

Reviewer_THT4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper looks at plugin architectures for endowing generalist robot policies with memory. The memory makes use of coarse (frame-level) and local (patch-level) tokens. The system produces solid improvements on flagship VLAs (PI0.5 and GROOT) on memory-driven simulation benchmarks like MIKASA-Robo. The overall approach appears to cache patch-level VIT latents, making use of a recall map to decide which sparse set of patches to keep in memory. Initially, recall is set randomly. But during later stages of training, patches are given a target recall psuedo-labels if including them produces better action predictions. The final model is trained for both action prediction and recall prediction (with pseudo-labels).

Critical strengths
  • It seems quite natural to treat sparse memory as a decision-theoretic task (that must decide what memory to keep), which can be supervised with a "privileged" expert that has knowledge of a memory token's impact on the target action task. While such methods have been explored in other settings like language, I have not seen this for robotics.
Critical weaknesses
  • I think this paper could better position itself with respect to past related work. For example, much of the text focuses on global and fine memory encoding, but this seems standard. Papers such as MemoryVLA are cited and described in the related work Sec 2.2, but the description doesn't provide much insight as to the unique design choices in STAM that really matter. Sentence 121-123 hint at this but I think this needs to be expanded, as this is the core argument of the paper. Maybe it would help to revisit comparisons with prior art in the diagnostics section in Sec 4.2

  • I am not sure if recall maps are updated over time; once a spacetime patch is not recalled, I am guessing it is no longer available as memory for future timesteps. Is this true?

  • Very minor gripe, but I take issue with the use of the term non-Markovian. Higher-order markov models can depend on a historical markov blanket and so can represent actions such as wiping the table 3 times. I know that in the robot literature, "non-markovian" colloquially means "non-first-order-markov", but I view this as mistake that should be corrected.

Questions for authors

Please clarify distinctions with respect to prior art.

1 comment

Thank you for your careful review and for encouraging us to clarify what distinguishes STAM from prior memory-augmented policies. We first explain how STAM’s patch-level recall differs from the memory mechanisms in MemoryVLA, MEM, and GMP. We then clarify how the recall map is updated at each decision step and why a patch skipped at one step remains available later. Finally, we explain our use of the term “history-dependent” for tasks in which the current observation alone may be insufficient to select an action.

1. Positioning relative to MemoryVLA, MEM and GMP

Thank you for suggesting a clearer positioning with respect to prior work. We have revised Sec. 2.2 to explicitly compare STAM with MemoryVLA, MEM, and GMP, and to distinguish complementary memory representations from the memory-access mechanism that is central to STAM. We have also expanded the diagnostic discussion in Sec. 4.2 to connect these distinctions to the global recall map ablation and discuss the benefit of allowing different image regions to make different recall decisions.

We agree that combining high-level historical context with fine-grained visual information has precedents. MemoryVLA already maintains perceptual and cognitive memories with attention-based retrieval and gated fusion, while MEM combines short-term video memory with long-term language memory. STAM’s specific contribution is an explicit spatio-temporal recall map: each view-patch can disable memory access or predict a continuous look-back duration, allowing different regions to retrieve visual evidence from different historical moments. This differs from MemoryVLA’s attention-based aggregation over memory-bank entries.

Action-error-based memory supervision also has a precedent in GMP. The distinction is the supervision target: GMP calibrates a binary history-use gate, whereas STAM scores candidate recall maps to obtain pseudo-labels specifying both patch activation and temporal offsets. We have made these distinctions explicit in the revised related-work discussion.

2. Can a patch skipped at one step be recalled later?

Thank you for raising this question. The recall map is predicted anew at every decision step from the evolving SSMC history state. A zero recall value only disables memory access for the corresponding view-patch at the current step; it does not remove historical features from memory.

Specifically, the episodic patch buffer stores the view-patch features from each decision step, regardless of which patches access memory at that step. Previously stored features remain available for subsequent retrieval. Thus, a historical patch that is not recalled at one step can still be recalled at a later step if the newly predicted recall map selects it. The recall map controls which historical evidence is used for the current decision, rather than which evidence is retained for future decisions.

3. Terminology for tasks that require history

Thank you for this clarification. We agree that requiring historical information does not by itself imply non-Markovian dynamics. We have revised the terminology to “history-dependent tasks” to state our intended meaning more precisely: the current observation alone may be insufficient for action selection.


Reviewer_2royOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a hierarchical (coarse-fine) plug-in visual memory module called STAM, compatible with diverse VLAs (experiments with π0.5 and GR00T N1.5).
The coarse memory is a compact summary of past frames while the fine memory stores past images. At inference time, at every step, a learned recall map predicts how many past frames to use for each image region. The recall map is trained via a search over candidate maps to reduce pretrained policy action error. STAM is evaluated on LIBERO / LIBERO-MEM, MIKASA-Robo, MemMimic and on real robots.

Critical strengths

Selective, sparse memory is a reasonable approach to the problem. The method is efficient. Results reported across multiple VLAs. Diverse evaluation including real robots.

Critical weaknesses

Novelty:

  • Multiple highly relevant works are not cited or compared. Some examples: MemER (Sridhar et al., ICLR 2026), SAM2Act / MemoryBench, copycat / causal confusion (Wen et al. 2020; de Haan et al. 2019), video-LLM memory banks and token retrieval (MA-LMM, MovieChat)
  • The paper is very close to GMP and MEM (btoh cited), the contribution over these is somewhat incremental.

Methodology, results:

  • The recall does not seem to depend on the current frame, an ablation would be nice.
  • If I red the method correctly, M_{t,ℓ}[p] seems to be a single toksen, which would make the cross-attention degenerate.
  • The retrieval assumes a static camera because fo the fixed index.
  • Training details for SSMC are missing, reducing reproducibility.
  • The buffer still keeps all the frames, there is no selectivity for the memory storage.
  • LIBERO-MEM does not report π0.5 and GR00T without memory, we do not knwo the gian over the baseline.
  • Pi-Mem* (visual-memory-only) performs below the plain π0.5 on LIBERO, which suggests the baselines may be not tuned.
  • on MemMimic, no memory should be guessing, but π0.5 without memory scores 100 %, there is probably a cue leakage
  • on ARX shell game the STAM performance is at chance level, the baselines fail at 0 %, which is a manipulation failure. STAM might not be remembering anything useful here
  • STAM gains consistently more on π0.5 than on GR00T, even where the baseline performance is comparable, no intuition on this is given
  • some ablations, liek the fine memory one, show performance lower than the no-memory baseline

Presentation:

Fig. 4(a) should have measured distributions, not “illustrative”.

Fig. 2 is hard to parse and somewhat overloaded:

  • The pipeline overview (top-left) blends with rest.
  • The right half mixes the fine grid, current embeddings and coarse-fused features without a clear layer axis.
  • "Current Image Patches" has no incoming arrow.
  • τ_t is undefined in the fig.
  • Fig. 3: it would help to color-match each backbone's baseline and +STAM bars
  • statements like "a failure mode that a reviewer would naturally worry about" in the appendix
Questions for authors
  1. Please add the memory-less baselines on LIBERO-MEM
  2. Explain the no-memory MemMimic results and why ablated variants score below the no-memory backbone
  3. Include missing baselines
  4. Show per-seed results for ablations
  5. Show that the patch-level recall (main difference against GMP and MEM) is what actually helps.
4 comments

Thank you for your careful review and for raising specific questions about STAM’s design and evaluation. We address your comments in four parts below. First, we report the memoryless baselines and discuss the MemMimic and fine-memory ablation results. Second, we compare STAM with other memory-augmented policies and report results across training seeds. Third, we examine the contribution of patch-level recall and clarify how memory fusion works. Finally, we describe the camera setup, SSMC training procedure, and buffer storage strategy.

1. Baselines and MemMimic

Q1. Memoryless baselines on LIBERO-MEM

On LIBERO-MEM, we score each rollout by the fraction of subgoals completed and report the average across the ten tasks. For each backbone, the memoryless baseline and its STAM counterpart use the same pretrained checkpoint, training demonstrations, and evaluation protocol, including the same camera views, robot state inputs, initial states, and episode step limits. The memoryless π0.5\pi_{0.5} and GR00T N1.5 baselines achieve 8.0% and 5.0% average subgoal completion, respectively. Adding STAM raises these scores to 32.4% and 28.7%, gains of 24.4 and 23.7 percentage points. We have added both memoryless baselines to Table 1 of the revised paper.

Q2. Why can memoryless policies succeed on Match Color and Place Back?

The released MemMimic scorer checks whether the cube was lifted and placed in the correct bin, so the two-second pause in the demonstrations is optional for scoring. In Place Back, the robot can lift the cube and lower it into the same bin. In Match Color, the layout after shuffling is fixed, so the robot can choose its destination while the original color is still visible. These shortcuts offer a plausible explanation for the near-perfect scores of both original backbones. STAM brings larger gains when the right action is harder to infer from the current observation and depends more on what happened earlier. Its selective recall lets the policy retrieve the original color after an unpredictable delay or revisit the outcome of a previous push or fling to adjust the next motion. On Random Delay, this improved success from 78% to 98% with π0.5\pi_{0.5} and from 69% to 91% with GR00T.

Q2, continued. Why can removing fine memory hurt more than using no memory?

In the w/o Fine Memory ablation, we remove only the fine-memory retrieval path. Coarse memory summarizes the visual history at each patch location in a recurrent SSMC state. Fine memory lets the policy retrieve the feature at that location from a specific earlier frame. The ablated variant retains coarse memory and the recall map, so active patches still receive a compressed summary of their history.

In this respect, the coarse-only variant follows the same approach as most recurrent-memory policies, which use a compressed history to guide the current action. Compression can weaken a task-relevant cue, such as an object’s earlier color, while retaining irrelevant details that the policy can overfit to during training. For example, after a failed attempt, the trajectory leading to that failure remains encoded in memory. If the policy relies too heavily on that history, it may repeat the same motion, fail again, and get caught in a cycle of repeated failures. Removing fine retrieval also prevents the policy from revisiting the specific frame where the relevant cue was visible. On MIKASA-Robo, the coarse-only variant achieves 21.6% average success, compared with 39.6% for the no-memory π0.5\pi_{0.5} backbone and 72.8% for full STAM (Table 2 and Appendix Table S7).


2. Additional comparisons and training seeds

Q3. Additional baselines and the distinction from GMP and MEM

Thank you for suggesting additional comparisons. We additionally evaluated MemER (ICLR ’26) on all five MIKASA-Robo tasks and report the results in the table below. MA-LMM and MovieChat are long-video understanding models, while Wen et al. (2020) and de Haan et al. (2019) study spurious correlations in imitation learning. Neither group is applicable as a policy baseline for robot manipulation.

The original submission also reports comparisons across simulation and real-world experiments with GMP (CoRL ’26), Pi-Mem* (MEM, CoRL ’26), MemoryVLA (ICLR ’26), CronusVLA (AAAI ’26), SlotVLA (ICRA ’26), Naive E-SlotSSM (AAAI ’26), Past-Token Prediction (CoRL ’25), history-conditioned Diffusion Policy (RSS ’23), BC-RNN (CoRL ’21), and Fixed-Window Recall.

STAM’s main distinction from GMP and MEM is how it controls access to history. GMP uses action prediction error to calibrate a binary history-use gate at each decision step, while MEM combines short-term video memory with long-term language memory. STAM uses action prediction error to learn which image patches should access memory and how far back each should look. This lets STAM recover different historical details for different image regions within the same decision and use current visual features alone where they are sufficient.

The table below compares MemER with the policies in our MIKASA-Robo evaluation across all five tasks. STAM with π0.5\pi_{0.5} achieves 72.8% average success, compared with 67.8% for GMP, 56.8% for MemER, and 47.2% for Pi-Mem*. The Global Recall Map ablation in Sec. 4.5 isolates the effect of patch-level recall within STAM. Sharing one recall decision across the image while retaining both memory paths reduces average success from 72.8% to 57.8%.

MIKASA-Robo task success rates (%) and their mean across five tasks.

MethodShellGameTouchInterceptMediumRememberColor3RememberColor5RememberColor9Mean
CronusVLA32.05.031.013.09.018.0
MemoryVLA88.024.044.030.020.041.2
Pi-Mem* (π0.5\pi_{0.5})52.060.061.051.012.047.2
MemER92.048.069.060.015.056.8
GMP98.083.080.061.017.067.8
STAM (GR00T N1.5)94.085.079.057.018.066.6
STAM (π0.5\pi_{0.5})96.088.087.073.020.072.8

Q4. Per-seed results

Across all benchmarks, the STAM results reported in our paper are averaged over three independent training seeds. We train and evaluate each policy separately and then average the three evaluation scores. The table below gives the individual scores for full STAM on LIBERO-MEM. Mean subgoal completion was 32.4% with π0.5\pi_{0.5} and 28.7% with GR00T N1.5, with sample standard deviations of 1.6 and 1.1 percentage points, respectively. For both backbones, every seed’s score was within 1.8 percentage points of the corresponding mean reported in Table 1.

Full STAM on LIBERO-MEM. Values are average subgoal completion (%) across ten tasks.

Full STAM backboneSeed 42Seed 43Seed 44Mean
π0.5\pi_{0.5}34.231.931.132.4
GR00T N1.527.928.230.028.7

3. Patch-level recall and fusion

Q5. Contribution of patch-level recall

Thank you for asking us to clarify the contribution of patch-level recall. GMP gates history use at each decision step, while MEM encodes recent observations with a video encoder. STAM makes the recall decision at the patch level. Each patch can skip memory access or choose how far back to look, so different image regions can retrieve evidence from different moments within the same decision.

To measure the benefit of this design, we compare two STAM configurations that use the same π0.5\pi_{0.5} backbone and retain both coarse and fine memory. STAM with a Global Recall Map shares one recall decision across the image and achieves 57.8% average success on MIKASA-Robo. Full STAM with patch-level recall achieves 72.8%, with gains on all five tasks (Sec. 4.5, Fig. 8, and Appendix Table S7). These gains show the benefit of region-specific recall. Within a single decision, the policy can combine relevant visual cues from different points in the episode, giving it access to evidence that a single shared look-back time may miss.

We added two analyses in Sec. 4.2 (Table 4) to show how STAM uses this flexibility. First, under clean history, STAM activates more patches and looks further back on history-dependent tasks than on tasks where the current observation is sufficient. Second, when we perturb only the historical observations, STAM activates more patches and the median look-back duration of active patches increases. This pattern is consistent with STAM seeking alternative evidence in additional regions or earlier observations when some historical evidence is degraded. With a shared look-back time, changing the retrieved frame for an affected region also changes it for every other region. Patch-level recall lets STAM adjust retrieval for the affected region while still accessing useful evidence elsewhere. Across all six evaluations in Table 4, perturbing history increased the fraction of patches accessing memory. The largest drop in success rate was 1.0 percentage point on Random Delay, from 98.0% to 97.0%.

Single-token cross-attention

Yes. With one memory token per patch, the attention weight is always 1, so it does not depend on the query. The resulting operation is the gated residual projection we intended: we apply a learned projection to the selected memory token and add the gated result to the current patch feature.


4. Camera views, training and buffer storage

Same-index retrieval and camera motion

Thank you for raising this point. Following common practice in memory-augmented robot policies such as MemoryVLA, CronusVLA, and GMP, we use fixed third-person views for historical patch retrieval. We do not apply same-index retrieval to moving wrist-camera views. Once the recall map selects a historical timestamp for a patch, STAM retrieves the token at the same grid location in the same camera view. For moving cameras, we plan to explore pose-aware patch alignment so that STAM can retrieve historical evidence from the corresponding scene region as the viewpoint changes.

SSMC training details

Thank you for asking about the SSMC training setup. Appendix B.1 describes how the SSMC is trained across the three stages. In Stage 1, we train it jointly with the policy using the action prediction loss. We freeze it during recall pseudo-label search in Stage 2 and train it jointly with the policy and recall head in Stage 3. We will release the source code for the SSMC along with the training configurations, including the model dimensions and hyperparameters used in our experiments.

Buffer storage and selective retrieval

By default, STAM stores patch features at every decision step and defers selection until retrieval. In a memory-dependent task, a decision at step 2,000 may require the color of a block seen at step 200. A write-time filter must decide at step 200 whether to keep that patch, before the policy knows which detail it will need later. If the patch is discarded, the policy can no longer retrieve that color cue. STAM keeps the patch available so that, at step 2,000, its recall map can select the relevant image region and look back to step 200.

Even on a long-horizon task spanning 5,000 decision steps, peak memory usage was only 8.3 GiB, well within the memory budget of our evaluation setup. At 500 and 1,000 steps, peak usage was 1.54 GiB and 2.4 GiB, respectively. Each step stores approximately 1.69 MiB of patch features, calculated as 3 views × 256 patches per view × 1,152 features per patch × 2 bytes per BF16 value = 1,769,472 bytes. At retrieval time, each active patch reads one cached token, so the number of tokens retrieved per decision depends on how many patches are active. In the newly added Sec. 4.4 (Inference Efficiency), Fig. 7(a) shows that compiled inference latency increased from 70.1 to 77.9 ms over 5,000 decision steps.

We also added a buffer-size experiment on SO-101 Tic-Tac-Toe in Fig. 7(b). A fixed-capacity sliding window evicts the oldest entry when full, while uniform temporal sampling stores features every k steps. At 60% frame retention, uniform sampling achieved 72.7% success, compared with 23.0% for the sliding window. With the same number of stored frames, the policy performed substantially better when it could still retrieve evidence from earlier stages of the episode. We plan to explore adaptive retention strategies that further reduce memory usage while preserving cues that later decisions may require.


Comments on the paper

1 comment

We thank the three reviewers for their careful reading of STAM. Reviewer 2roy described selective, sparse memory as a reasonable approach to history-dependent manipulation and highlighted the method’s efficiency. Reviewer THT4 singled out how STAM learns to use memory: during training, the effect of a candidate recall decision on action prediction provides a supervision signal for that decision. THT4 found this a natural way to formulate sparse memory use and noted that, although familiar with related ideas in language modeling, they had not previously encountered this approach in robotics.

Reviewer 2roy also highlighted the breadth of the evaluation across multiple vision-language-action policies and real robots. Reviewer tBio emphasized the consistency of the gains across tasks and backbones. On MIKASA-Robo, STAM raised average success from 39.6% to 72.8% with π0.5\pi_{0.5} and from 38.6% to 66.6% with GR00T N1.5. On MemMimic, success with π0.5\pi_{0.5} rose from 82.1% to 94.2%. and from 38.6% to 66.6% with GR00T N1.5. On MemMimic, success with rose from 82.1% to 94.2%.

Reviewer tBio further highlighted the ablations as evidence for two specific design choices. Removing fine memory led to a large performance drop, supporting the value of retaining visual details that the policy can retrieve later. Replacing adaptive patch-level recall with a fixed temporal window or an image-wide recall decision also reduced success, supporting the value of letting individual regions select how far back to look.

Summary of revisions and additional experiments

  • Method positioning and patch-level recall (§§2.2 and 4.2). We expanded the comparison with MemoryVLA, MEM, and GMP and connected these design differences to the Global Recall Map ablation. With the backbone and both memory paths held fixed, allowing each patch to select its own recall time raises average success on MIKASA-Robo from 57.8% to 72.8%, with gains on all five tasks. We also adopted the term history-dependent for tasks in which the current observation alone may be insufficient for action selection.

  • Memoryless baselines and additional comparisons. In §4.1, Table 1 now includes the original π0.5\pi_{0.5} and GR00T N1.5 policies on LIBERO-MEM. Across ten tasks, STAM raises average subgoal completion from 8.0% to 32.4% with π0.5\pi_{0.5} and from 5.0% to 28.7% with GR00T N1.5. In our response to Reviewer 2roy, we also report a new evaluation of MemER on all five MIKASA-Robo tasks. Mean success is 56.8% for MemER, 67.8% for GMP, and 72.8% for STAM with π0.5\pi_{0.5}. The same response reports individual training-seed results for full STAM on LIBERO-MEM with both backbones.

  • Selective recall under clean and perturbed history (§4.2, new Table 4 and Fig. 5). Table 4 quantifies recall across six tasks and suites, and Fig. 5 illustrates local occlusion of hand–cup contact in a shell-game episode. Under clean history, 99.4–100.0% of patch–timestep entries had zero recall when the current observation was sufficient, compared with 13.7–21.3% on history-dependent tasks. Perturbing historical frames increased the fraction of active patches in all six evaluations. STAM also looked further back on all three history-dependent tasks. Success rates remained within one percentage point of their clean-history values.

  • Inference efficiency and buffer storage (new §4.4 and Fig. 7). In our reviewer responses, we report peak patch-buffer usage of 1.54 GiB at 500 decision steps, 2.4 GiB at 1,000 steps, and 8.3 GiB at 5,000 steps. Each step adds only 1.69 MiB of cached patch features, so even over 5,000 steps the full buffer remained well within the memory budget of our evaluation setup. Figure 7(a) reports compiled inference latency on an NVIDIA A800, which rose from 70.1 to 77.9 ms as episode length increased to 5,000 decision steps. Figure 7(b) compares two ways to limit buffer storage on SO-101 Tic-Tac-Toe. At 60% frame retention, uniform temporal sampling achieved 72.7% success, compared with 23.0% for a sliding window. With the same number of stored frames, sampling across the episode preserved access to earlier observations and yielded higher task success.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.