- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SimpleMemVLA, a vision-language-action framework that uses a pretrained VLM's native video context as memory for long-horizon manipulation. Instead of storing selected observations in a retrieval bank, compressing history into a fixed token budget, or maintaining a recurrent memory state, the method preserves sampled historical frames in timestamped video format until the current decision.
To connect long visual history to action generation, the model generates a textual description of the current sub-task. The contextual hidden states and token embeddings of this generated span condition a flow-matching action head, together with the current proprioceptive state. For efficient inference, the method prefills the shared historical prefix during action execution and reuses the resulting KV cache across consecutive decisions.
The paper evaluates SimpleMemVLA on four memory-centric benchmarks, two general-purpose manipulation benchmarks, and two real-world dual-arm manipulation tasks. It reports strong results on the memory benchmarks, including 88.3% success on RoboMME, compared with 31.5%, 22.6%, and 20.6% for the authors' controlled retrieval, token-compression, and recurrent-state variants, respectively. History interventions further show that removing or replacing relevant past observations changes the model's behavior in the corresponding direction. The paper also reports a reduction in decision latency from 1.02 s to 0.68 s through streaming inference.
The main contribution is a simple memory design that defers the selection of relevant historical evidence until the current decision, supported by controlled comparisons, intervention studies, and deployment experiments.
Critical strengths
1. Simple and effective memory design. The paper proposes using the VLM's native video context as memory, avoiding dedicated retrieval, compression, or recurrent memory modules. This is a simple and well-motivated approach that achieves strong results on multiple memory-centric benchmarks.
2. Comprehensive evaluation of memory utilization. The authors conduct controlled comparisons and history-intervention experiments, showing that removing or replacing relevant historical evidence predictably changes the policy's behavior. These experiments provide useful evidence that the model actively uses historical visual information for decision-making.
Critical weaknesses
1. Limited methodological novelty beyond existing long-context inference techniques.
The proposed system largely relies on established components: Prefix caching and overlapping prefill with action execution are standard techniques in LLM/VLM inference systems (SGLang[1] / PagedAttention[2]). The SWA variant further adopts conventional sliding-window attention and cache eviction to bound memory usage[3]. While this is useful for deployment, it appears to be a straightforward application of existing techniques rather than a distinct algorithmic contribution.
[1] Zheng, L., et al. “SGLang: Efficient Execution of Structured Language Model Programs.” NeurIPS, 2024. https://arxiv.org/abs/2312.07104
[2] Kwon, W., et al. “Efficient Memory Management for Large Language Model Serving with PagedAttention.” SOSP, 2023. https://arxiv.org/abs/2309.06180
[3] Jiang, A. Q., et al. “Mistral 7B.” 2023. https://arxiv.org/abs/2310.06825
2. Insufficient resource-matched evaluation of the proposed memory design.
SimpleMemVLA relies on processing long video histories, which increases inference latency and GPU memory consumption as the context grows. The authors introduce SWA to bound the attention window and cache usage, but this also limits direct access to older observations. However, they do not compare SWA with retrieval, compression, or recurrent memory methods under matched token budgets. It therefore remains unclear whether the proposed approach offers a better trade-off between memory capacity, inference efficiency, and task performance.
Questions for authors
-
How does SWA compare with retrieval, token compression, and recurrent memory under matched token budgets?
-
Can the authors report task success, GPU memory consumption, and inference latency to establish whether their approach offers a better efficiency–performance trade-off?
-
What temporal context does the annotating VLM use to generate sub-task labels, and does policy training use the same history length, frame-sampling rate, and timestamp convention as deployment?
-
How do the authors ensure that the training setup adequately supervises decisions requiring evidence from distant observations?
10 comments
Response - 2-1
Q2: Insufficient resource-matched evaluation of the proposed memory design. SimpleMemVLA relies on processing long video histories, which increases inference latency and GPU memory consumption as the context grows. The authors introduce SWA to bound the attention window and cache usage, but this also limits direct access to older observations. However, they do not compare SWA with retrieval, compression, or recurrent memory methods under matched token budgets. It therefore remains unclear whether the proposed approach offers a better trade-off between memory capacity, inference efficiency, and task performance.
A2: Native video performs strongly at compact budgets: approximately 700 history tokens yield 47.9% success, compared with 38.3% for 512 compression tokens and 36.0% for 256 recurrent-state tokens.
The 16-frame native configuration exceeds Perceiver compression by 9.6 percentage points while receiving 7.5 times fewer source frames (16 versus the compressor's complete 120-frame history). Retaining all 120 sampled frames raises native-video success to 88.3%. We evaluate all interfaces on the same 16 RoboMME tasks with the backbone, training data, sub-task supervision, and action-head architecture fixed.
The complete comparison below reports 13 configurations on all 16 RoboMME tasks, including nine added configurations. Retrieval variants use 8, 16, 32, or 120 sampled frames, compression variants use 64–512 memory tokens, and recurrent-state variants use 16–256 memory tokens. In the native video format, every two sampled frames form one temporal patch with 80 visual tokens after spatial merging, approximately 6–8 timestamp tokens, and two visual delimiters. Thus, 16, 32, and 120 sampled frames correspond to approximately 0.7K, 1.4K, and 5.3K historical native-interface tokens, respectively; after adding the current views and prompt, the 60-second native-video configuration contains approximately 5.6K input tokens. Native-video and retrieval budgets count the encoded visual tokens, timestamps, and delimiters supplied to the backbone; compression and recurrent-state budgets count their output memory/state tokens. These history-token counts are reported separately from deployment latency and peak GPU memory.
The performance curves provide the clearest comparison. Within the same 60-second history span, timestamped uniform retrieval rises from 47.9% with 16 frames to 55.1% with 32 frames, while query-conditioned retrieval reaches 56.5% at the same 32-frame budget. Retaining all 120 frames in the native video interface reaches 88.3%. By contrast, compression improves from 34.3% with 256 tokens to 38.3% with 512 tokens, while recurrent-state memory improves from 33.4% with 64 tokens to 36.0% with 256 tokens. Native video therefore continues to gain substantially as more historical evidence is retained, whereas the tested compression and recurrent-state variants show smaller gains at their larger budgets.
The matched deployment measurements show a modest incremental decision-path cost for visual memory. On H100, Prefix Prefill with a 60-second history takes 0.699 s versus 0.674 s for the same-checkpoint single-frame baseline, adding 25 ms (3.7%) of decision-path latency and 0.78 GB (6.6%) of peak GPU memory. Background history prefill is excluded from these decision-path timings. A controlled, same-checkpoint SWA runtime ablation takes 0.739 s under the same measurement boundary. Separately, the trained SWA deployment uses a 5,888-token attention window, reaches 91.0% RMBench success versus 94.0% for standard deployment, and has a 0.92 s median latency over 153 consecutive decisions including background maintenance.
The capacity sweep and deployment measurements establish two useful operating points: compact native history already exceeds the tested learned compression and recurrent-state configurations, and retaining the full sampled history gives the highest measured success. Prefix reduces the incremental decision-path cost, while SWA supports incremental updates with a bounded active cache.
Response - 2-2
A2: Complete RoboMME comparison: 13 configurations, including 9 added in the rebuttal.
Task groups: Counting (BinF, PickX, SwingX, StopC); Permanence (VU, BU, VUS, BUS); Reference (PickH, VR, VPB, VPO); Imitation (MoveC, InsP, PatL, RouteS).
Retrieval-based memory
| Memory mechanism | Budget | BinF | PickX | SwingX | StopC | VU | BU | VUS | BUS | PickH | VR | VPB | VPO | MoveC | InsP | PatL | RouteS | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Retrieval (uniform, original) | 8 frames | 38 | 34 | 76 | 38 | 76 | 58 | 20 | 14 | 18 | 0 | 18 | 18 | 68 | 8 | 10 | 10 | 31.5 |
| Retrieval (uniform, no TS) | 16 frames | 48 | 86 | 72 | 54 | 74 | 42 | 22 | 24 | 30 | 22 | 44 | 36 | 74 | 14 | 16 | 12 | 41.9 |
| Retrieval (uniform, TS) | 16 frames | 48 | 100 | 84 | 62 | 82 | 56 | 32 | 26 | 28 | 20 | 58 | 48 | 88 | 12 | 14 | 8 | 47.9 |
| Retrieval (uniform, TS) | 32 frames | 52 | 100 | 86 | 74 | 84 | 62 | 48 | 38 | 32 | 34 | 66 | 54 | 84 | 24 | 26 | 18 | 55.1 |
| Retrieval (query, TS) | 32 frames | 44 | 98 | 72 | 68 | 64 | 68 | 52 | 52 | 38 | 48 | 64 | 62 | 76 | 32 | 38 | 28 | 56.5 |
| Native video context | 120 frames | 78 | 100 | 96 | 92 | 100 | 100 | 90 | 94 | 90 | 64 | 86 | 90 | 94 | 46 | 94 | 98 | 88.3 |
Compression-based memory
| Memory mechanism | Budget | BinF | PickX | SwingX | StopC | VU | BU | VUS | BUS | PickH | VR | VPB | VPO | MoveC | InsP | PatL | RouteS | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Compression (original) | 64 tokens | 26 | 50 | 92 | 16 | 18 | 12 | 14 | 10 | 16 | 6 | 16 | 12 | 56 | 0 | 10 | 8 | 22.6 |
| Compression (Perceiver, TE) | 64 tokens | 32 | 52 | 72 | 16 | 18 | 24 | 14 | 12 | 18 | 12 | 24 | 18 | 52 | 12 | 8 | 10 | 24.6 |
| Compression (Perceiver, TE) | 256 tokens | 44 | 92 | 92 | 36 | 26 | 32 | 10 | 14 | 22 | 20 | 38 | 32 | 62 | 4 | 10 | 14 | 34.3 |
| Compression (Perceiver, TE) | 512 tokens | 42 | 94 | 90 | 38 | 24 | 32 | 12 | 26 | 34 | 32 | 42 | 34 | 56 | 10 | 24 | 22 | 38.3 |
Recurrent-state memory
| Memory mechanism | Budget | BinF | PickX | SwingX | StopC | VU | BU | VUS | BUS | PickH | VR | VPB | VPO | MoveC | InsP | PatL | RouteS | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Recurrent-state (original) | 16 tokens | 26 | 36 | 56 | 6 | 26 | 24 | 30 | 16 | 12 | 4 | 34 | 20 | 16 | 0 | 10 | 14 | 20.6 |
| Recurrent-state$^{+}$ | 64 tokens | 46 | 42 | 66 | 14 | 44 | 52 | 52 | 48 | 26 | 18 | 26 | 28 | 24 | 8 | 18 | 22 | 33.4 |
| Recurrent-state$^{+}$ | 256 tokens | 58 | 54 | 46 | 24 | 48 | 44 | 56 | 52 | 24 | 16 | 34 | 32 | 32 | 6 | 22 | 28 | 36.0 |
marks the nine newly added configurations. Success rates (%), with 50 episodes per task. Avg. is the macro-average over all 16 tasks. Bold marks the highest result in each column, including ties. Rows marked original and the SimpleMemVLA native row report results from the original paper.
Retrieval budgets count sampled frames supplied to the VLM; compression and recurrent-state budgets count memory tokens. TS: timestamps; TE: temporal encoding. The original retrieval baseline is unordered and has no timestamps; the new retrieval variants preserve temporal order. Perceiver denotes the strengthened two-layer architecture. The 120-frame row is SimpleMemVLA native: it retains the full sampled history without a separate retrieval module.
Task names: BinF = BinFill; PickX = PickXtimes; SwingX = SwingXtimes; StopC = StopCube; VU = VideoUnmask; BU = ButtonUnmask; VUS = VideoUnmaskSwap; BUS = ButtonUnmaskSwap; PickH = PickHighlight; VR = VideoRepick; VPB = VideoPlaceButton; VPO = VideoPlaceOrder; MoveC = MoveCube; InsP = InsertPeg; PatL = PatternLock; RouteS = RouteStick.
In the revision, we will retain the complete task-level table, foreground the closest-budget comparison with explicit token accounting, and distinguish the controlled 0.739 s runtime measurement from the 0.92 s continuous SWA deployment result.
Response - 1
Q1: Limited methodological novelty beyond existing long-context inference techniques. The proposed system largely relies on established components: Prefix caching and overlapping prefill with action execution are standard techniques in LLM/VLM inference systems (SGLang[1] / PagedAttention[2]). The SWA variant further adopts conventional sliding-window attention and cache eviction to bound memory usage[3]. While this is useful for deployment, it appears to be a straightforward application of existing techniques rather than a distinct algorithmic contribution.
A1: The novelty is the native-video memory interface and its exact, trainable streaming realization; prefix caching and sliding-window attention are enabling primitives.
A pretrained VLM can use its native video context as effective embodied memory without a dedicated memory module. This is a non-obvious empirical finding: the existing RoboMME comparison contains fourteen memory variants built on a shared π0.5 backbone, but does not include the raw native-video context as a memory baseline. Among the reported alternatives, the strongest reaches 44.5% average success, compared with 88.3% for attending directly to the timestamped visual stream.
We strengthened the comparisons using the same backbone, training data, sub-task supervision, and action-head architecture. Retrieval restores temporal order and timestamps and adds query-conditioned selection; compression uses a Perceiver architecture; and compression and recurrent-state variants receive larger memory budgets. The strongest retrieval-based, compression-based, and recurrent-state alternatives achieve 56.5%, 38.3%, and 36.0% average success, respectively, while native video context reaches 88.3%. The native-context interface therefore remains effective after the competing interfaces are strengthened.
Results are mean success rates across the 16 RoboMME tasks.
The technically non-trivial part is making native-context memory exact and trainable for embodied control. First, the streaming implementation reproduces the full-history decision: token and position IDs match at integer exactness, activations agree up to kernel-level numerical noise, and generated outputs are identical to full recomputation. This equivalence applies to the complete VLM–sub-task–action pipeline and is not provided automatically by a generic inference-serving system.
Second, the SWA variant requires training and deployment context constructions to match. Because the linear-attention layers must integrate the episode from its first frame, ordinary independent-window training would create a train–deployment mismatch. We therefore use episode-level packing: the episode is forwarded once as a shared trunk, and B random decision anchors fork from the corresponding cached states. Each branch attends to the same bounded window used at deployment, while episode-absolute timestamps keep cached temporal patches immutable. This construction allows one new temporal patch to be appended and one expired cache block to be evicted at each decision without rebuilding the retained context.
Our contribution is therefore to establish and deploy a native-video memory interface in which the pretrained backbone retains the full temporal evidence, selects relevant information at decision time, and passes the resulting contextual hidden states to a standard flow-matching action head. The systematic comparisons show that this simple interface can outperform substantially more specialized memory mechanisms under the evaluated protocols.
In the revision, we will separate the enabling inference primitives from the native-context memory finding, and describe exact streaming equivalence and episode-level SWA packing as the implementation contributions required for deployable native visual memory.