acceptodds
Under review as a conference paper at ICLR 2027

SimpleMemVLA: A Simple but Effective Native-Video Memory for Vision-Language-Action Models

41 views

Abstract

Long-horizon manipulation is partially observable: the information needed to choose the next action may appear only in observations from minutes earlier. Existing memory mechanisms for VLAs, such as retrieval banks, learned compressors and recurrent states, must decide what to keep from the past before knowing what a future decision will require. They were motivated by the assumption that minute-scale history is too large to process directly, which no longer holds for modern VLM backbones. We propose SimpleMemVLA, a VLA without a dedicated memory module that uses the backbone's native video context directly as memory. It keeps the sampled history intact in the timestamped video format the backbone was pretrained to process, routes the evidence it finds to a standard flow-matching action head through the hidden states of a generated sub-task, and prefills the history shared by consecutive decisions during action execution, keeping latency close to that of a single-frame VLA. SimpleMemVLA achieves state-of-the-art results on four memory benchmarks without loss on general-purpose control, and with the same backbone and training setup it outperforms retrieval, compression and recurrent-state methods by a wide margin. History interventions show that the policy reads specific evidence from its past and follows edited histories without parameter updates, a visual form of in-context learning. On a physical dual-arm robot, it completes two tasks whose decisive evidence disappears before the robot acts.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Robot Learning, Long-Horizon Manipulation, Visual Memory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5wBnOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SimpleMemVLA, a vision-language-action framework that uses a pretrained VLM's native video context as memory for long-horizon manipulation. Instead of storing selected observations in a retrieval bank, compressing history into a fixed token budget, or maintaining a recurrent memory state, the method preserves sampled historical frames in timestamped video format until the current decision.

To connect long visual history to action generation, the model generates a textual description of the current sub-task. The contextual hidden states and token embeddings of this generated span condition a flow-matching action head, together with the current proprioceptive state. For efficient inference, the method prefills the shared historical prefix during action execution and reuses the resulting KV cache across consecutive decisions.

The paper evaluates SimpleMemVLA on four memory-centric benchmarks, two general-purpose manipulation benchmarks, and two real-world dual-arm manipulation tasks. It reports strong results on the memory benchmarks, including 88.3% success on RoboMME, compared with 31.5%, 22.6%, and 20.6% for the authors' controlled retrieval, token-compression, and recurrent-state variants, respectively. History interventions further show that removing or replacing relevant past observations changes the model's behavior in the corresponding direction. The paper also reports a reduction in decision latency from 1.02 s to 0.68 s through streaming inference.

The main contribution is a simple memory design that defers the selection of relevant historical evidence until the current decision, supported by controlled comparisons, intervention studies, and deployment experiments.

Critical strengths

1. Simple and effective memory design. The paper proposes using the VLM's native video context as memory, avoiding dedicated retrieval, compression, or recurrent memory modules. This is a simple and well-motivated approach that achieves strong results on multiple memory-centric benchmarks.

2. Comprehensive evaluation of memory utilization. The authors conduct controlled comparisons and history-intervention experiments, showing that removing or replacing relevant historical evidence predictably changes the policy's behavior. These experiments provide useful evidence that the model actively uses historical visual information for decision-making.

Critical weaknesses

1. Limited methodological novelty beyond existing long-context inference techniques.

The proposed system largely relies on established components: Prefix caching and overlapping prefill with action execution are standard techniques in LLM/VLM inference systems (SGLang[1] / PagedAttention[2]). The SWA variant further adopts conventional sliding-window attention and cache eviction to bound memory usage[3]. While this is useful for deployment, it appears to be a straightforward application of existing techniques rather than a distinct algorithmic contribution.

[1] Zheng, L., et al. “SGLang: Efficient Execution of Structured Language Model Programs.” NeurIPS, 2024. https://arxiv.org/abs/2312.07104

[2] Kwon, W., et al. “Efficient Memory Management for Large Language Model Serving with PagedAttention.” SOSP, 2023. https://arxiv.org/abs/2309.06180

[3] Jiang, A. Q., et al. “Mistral 7B.” 2023. https://arxiv.org/abs/2310.06825

2. Insufficient resource-matched evaluation of the proposed memory design.

SimpleMemVLA relies on processing long video histories, which increases inference latency and GPU memory consumption as the context grows. The authors introduce SWA to bound the attention window and cache usage, but this also limits direct access to older observations. However, they do not compare SWA with retrieval, compression, or recurrent memory methods under matched token budgets. It therefore remains unclear whether the proposed approach offers a better trade-off between memory capacity, inference efficiency, and task performance.

Questions for authors
  1. How does SWA compare with retrieval, token compression, and recurrent memory under matched token budgets?

  2. Can the authors report task success, GPU memory consumption, and inference latency to establish whether their approach offers a better efficiency–performance trade-off?

  3. What temporal context does the annotating VLM use to generate sub-task labels, and does policy training use the same history length, frame-sampling rate, and timestamp convention as deployment?

  4. How do the authors ensure that the training setup adequately supervises decisions requiring evidence from distant observations?

10 comments
AuthorsOpenReview ↗

Response - 1

Q1: Limited methodological novelty beyond existing long-context inference techniques. The proposed system largely relies on established components: Prefix caching and overlapping prefill with action execution are standard techniques in LLM/VLM inference systems (SGLang[1] / PagedAttention[2]). The SWA variant further adopts conventional sliding-window attention and cache eviction to bound memory usage[3]. While this is useful for deployment, it appears to be a straightforward application of existing techniques rather than a distinct algorithmic contribution.

[1] Zheng, L., et al. “SGLang: Efficient Execution of Structured Language Model Programs.” NeurIPS, 2024. https://arxiv.org/abs/2312.07104

[2] Kwon, W., et al. “Efficient Memory Management for Large Language Model Serving with PagedAttention.” SOSP, 2023. https://arxiv.org/abs/2309.06180

[3] Jiang, A. Q., et al. “Mistral 7B.” 2023. https://arxiv.org/abs/2310.06825

A1: The novelty is the native-video memory interface and its exact, trainable streaming realization; prefix caching and sliding-window attention are enabling primitives.

A pretrained VLM can use its native video context as effective embodied memory without a dedicated memory module. This is a non-obvious empirical finding: the existing RoboMME comparison contains fourteen memory variants built on a shared π0.5\pi_{0.5} backbone, but does not include the raw native-video context as a memory baseline. Among the reported alternatives, the strongest reaches 44.5% average success, compared with 88.3% for attending directly to the timestamped visual stream.

We strengthened the comparisons using the same backbone, training data, sub-task supervision, and action-head architecture. Retrieval restores temporal order and timestamps and adds query-conditioned selection; compression uses a Perceiver architecture; and compression and recurrent-state variants receive larger memory budgets. The strongest retrieval-based, compression-based, and recurrent-state alternatives achieve 56.5%, 38.3%, and 36.0% average success, respectively, while native video context reaches 88.3%. The native-context interface therefore remains effective after the competing interfaces are strengthened.

Memory interfaceHistory representationSuccess (%)
Retrieval (query-conditioned)32 selected frames56.5
Compression (Perceiver)512 compressed tokens38.3
Recurrent-state memory256 state tokens36.0
SimpleMemVLA (native video)120 sampled frames88.3

Results are mean success rates across the 16 RoboMME tasks.

The technically non-trivial part is making native-context memory exact and trainable for embodied control. First, the streaming implementation reproduces the full-history decision: token and position IDs match at integer exactness, activations agree up to kernel-level numerical noise, and generated outputs are identical to full recomputation. This equivalence applies to the complete VLM–sub-task–action pipeline and is not provided automatically by a generic inference-serving system.

Second, the SWA variant requires training and deployment context constructions to match. Because the linear-attention layers must integrate the episode from its first frame, ordinary independent-window training would create a train–deployment mismatch. We therefore use episode-level packing: the episode is forwarded once as a shared trunk, and BB random decision anchors fork from the corresponding cached states. Each branch attends to the same bounded window used at deployment, while episode-absolute timestamps keep cached temporal patches immutable. This construction allows one new temporal patch to be appended and one expired cache block to be evicted at each decision without rebuilding the retained context.

Our contribution is therefore to establish and deploy a native-video memory interface in which the pretrained backbone retains the full temporal evidence, selects relevant information at decision time, and passes the resulting contextual hidden states to a standard flow-matching action head. The systematic comparisons show that this simple interface can outperform substantially more specialized memory mechanisms under the evaluated protocols.

In the revision, we will separate the enabling inference primitives from the native-context memory finding, and describe exact streaming equivalence and episode-level SWA packing as the implementation contributions required for deployable native visual memory.

AuthorsOpenReview ↗

Response - 2-1

Q2: Insufficient resource-matched evaluation of the proposed memory design. SimpleMemVLA relies on processing long video histories, which increases inference latency and GPU memory consumption as the context grows. The authors introduce SWA to bound the attention window and cache usage, but this also limits direct access to older observations. However, they do not compare SWA with retrieval, compression, or recurrent memory methods under matched token budgets. It therefore remains unclear whether the proposed approach offers a better trade-off between memory capacity, inference efficiency, and task performance.

A2: Native video performs strongly at compact budgets: approximately 700 history tokens yield 47.9% success, compared with 38.3% for 512 compression tokens and 36.0% for 256 recurrent-state tokens.

The 16-frame native configuration exceeds Perceiver compression by 9.6 percentage points while receiving 7.5 times fewer source frames (16 versus the compressor's complete 120-frame history). Retaining all 120 sampled frames raises native-video success to 88.3%. We evaluate all interfaces on the same 16 RoboMME tasks with the backbone, training data, sub-task supervision, and action-head architecture fixed.

The complete comparison below reports 13 configurations on all 16 RoboMME tasks, including nine added configurations. Retrieval variants use 8, 16, 32, or 120 sampled frames, compression variants use 64–512 memory tokens, and recurrent-state variants use 16–256 memory tokens. In the native video format, every two sampled frames form one temporal patch with 80 visual tokens after spatial merging, approximately 6–8 timestamp tokens, and two visual delimiters. Thus, 16, 32, and 120 sampled frames correspond to approximately 0.7K, 1.4K, and 5.3K historical native-interface tokens, respectively; after adding the current views and prompt, the 60-second native-video configuration contains approximately 5.6K input tokens. Native-video and retrieval budgets count the encoded visual tokens, timestamps, and delimiters supplied to the backbone; compression and recurrent-state budgets count their output memory/state tokens. These history-token counts are reported separately from deployment latency and peak GPU memory.

The performance curves provide the clearest comparison. Within the same 60-second history span, timestamped uniform retrieval rises from 47.9% with 16 frames to 55.1% with 32 frames, while query-conditioned retrieval reaches 56.5% at the same 32-frame budget. Retaining all 120 frames in the native video interface reaches 88.3%. By contrast, compression improves from 34.3% with 256 tokens to 38.3% with 512 tokens, while recurrent-state memory improves from 33.4% with 64 tokens to 36.0% with 256 tokens. Native video therefore continues to gain substantially as more historical evidence is retained, whereas the tested compression and recurrent-state variants show smaller gains at their larger budgets.

The matched deployment measurements show a modest incremental decision-path cost for visual memory. On H100, Prefix Prefill with a 60-second history takes 0.699 s versus 0.674 s for the same-checkpoint single-frame baseline, adding 25 ms (3.7%) of decision-path latency and 0.78 GB (6.6%) of peak GPU memory. Background history prefill is excluded from these decision-path timings. A controlled, same-checkpoint SWA runtime ablation takes 0.739 s under the same measurement boundary. Separately, the trained SWA deployment uses a 5,888-token attention window, reaches 91.0% RMBench success versus 94.0% for standard deployment, and has a 0.92 s median latency over 153 consecutive decisions including background maintenance.

The capacity sweep and deployment measurements establish two useful operating points: compact native history already exceeds the tested learned compression and recurrent-state configurations, and retaining the full sampled history gives the highest measured success. Prefix reduces the incremental decision-path cost, while SWA supports incremental updates with a bounded active cache.

AuthorsOpenReview ↗

Response - 2-2

A2: Complete RoboMME comparison: 13 configurations, including 9 added in the rebuttal.

Task groups: Counting (BinF, PickX, SwingX, StopC); Permanence (VU, BU, VUS, BUS); Reference (PickH, VR, VPB, VPO); Imitation (MoveC, InsP, PatL, RouteS).

Retrieval-based memory

Memory mechanismBudgetBinFPickXSwingXStopCVUBUVUSBUSPickHVRVPBVPOMoveCInsPPatLRouteSAvg.
Retrieval (uniform, original)8 frames38347638765820141801818688101031.5
Retrieval (uniform, no TS)+^{+}16 frames4886725474422224302244367414161241.9
Retrieval (uniform, TS)+^{+}16 frames4810084628256322628205848881214847.9
Retrieval (uniform, TS)+^{+}32 frames52100867484624838323466548424261855.1
Retrieval (query, TS)+^{+}32 frames4498726864685252384864627632382856.5
Native video context120 frames7810096921001009094906486909446949888.3

Compression-based memory

Memory mechanismBudgetBinFPickXSwingXStopCVUBUVUSBUSPickHVRVPBVPOMoveCInsPPatLRouteSAvg.
Compression (original)64 tokens2650921618121410166161256010822.6
Compression (Perceiver, TE)+^{+}64 tokens325272161824141218122418521281024.6
Compression (Perceiver, TE)+^{+}256 tokens449292362632101422203832624101434.3
Compression (Perceiver, TE)+^{+}512 tokens4294903824321226343242345610242238.3

Recurrent-state memory

Memory mechanismBudgetBinFPickXSwingXStopCVUBUVUSBUSPickHVRVPBVPOMoveCInsPPatLRouteSAvg.
Recurrent-state (original)16 tokens2636566262430161243420160101420.6
Recurrent-state$^{+}$64 tokens464266144452524826182628248182233.4
Recurrent-state$^{+}$256 tokens585446244844565224163432326222836.0

+^{+} marks the nine newly added configurations. Success rates (%), with 50 episodes per task. Avg. is the macro-average over all 16 tasks. Bold marks the highest result in each column, including ties. Rows marked original and the SimpleMemVLA native row report results from the original paper.

Retrieval budgets count sampled frames supplied to the VLM; compression and recurrent-state budgets count memory tokens. TS: timestamps; TE: temporal encoding. The original retrieval baseline is unordered and has no timestamps; the new retrieval variants preserve temporal order. Perceiver denotes the strengthened two-layer architecture. The 120-frame row is SimpleMemVLA native: it retains the full sampled history without a separate retrieval module.

Task names: BinF = BinFill; PickX = PickXtimes; SwingX = SwingXtimes; StopC = StopCube; VU = VideoUnmask; BU = ButtonUnmask; VUS = VideoUnmaskSwap; BUS = ButtonUnmaskSwap; PickH = PickHighlight; VR = VideoRepick; VPB = VideoPlaceButton; VPO = VideoPlaceOrder; MoveC = MoveCube; InsP = InsertPeg; PatL = PatternLock; RouteS = RouteStick.

In the revision, we will retain the complete task-level table, foreground the closest-budget comparison with explicit token accounting, and distinguish the controlled 0.739 s runtime measurement from the 0.92 s continuous SWA deployment result.

Reviewer_LrjDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the use of historical information in partially observable robotic manipulation. SimpleMemVLA feeds sampled historical observations from a head-mounted camera into Qwen3.5-4B as timestamped video and conveys historical information to a flow-matching action head through the contextual hidden states associated with the generated subtask text. To reduce decision latency, it also prefills the history prefix required for the next decision while actions are being executed.

Critical strengths
  1. The simple approach yields substantial gains on RMBench, RoboMME, MIKASA-Robo, and RoboMemArena, demonstrating that directly leveraging pretrained video context can effectively support robotic memory tasks.
  2. History intervention experiments show that the policy accesses and relies on task-relevant historical information while disregarding irrelevant distractors, rather than making decisions solely based on current observations or a fixed sequence of steps.
Critical weaknesses
  1. The controlled comparisons simultaneously alter historical coverage, temporal structure, and representational capacity, preventing attribution of the performance gap to “write-time commitment.”The “retrieval” baseline in Section 4.4 uses only eight uniformly sampled frames and removes temporal order and timestamps, while the compression and recurrent baselines are restricted to 64 and 16 tokens, respectively. These comparisons therefore do not isolate write-time commitment as the cause of the performance gap.

  2. Low decision-path latency with long histories does not establish sustained real-time operation.The standard method uses window-relative timestamps, requiring the history to be re-encoded as the window slides; background prefilling therefore retains a computational cost that grows with context length. In the 45-minute experiment in Figure 5, background processing exceeds the action-execution budget, and decision-path latency itself reaches 1.18 seconds. These results do not demonstrate that the system can sustain the indicated control cadence.

Questions for authors
  1. Are the eight frames always sampled uniformly? Is there any mechanism for querying the history based on the current instruction or decision context?
  2. How do you rule out differences in action-execution capabilities between GroundSG and the proposed policy?
  3. Once the history window is full, how much history is actually re-encoded at each decision as old frames are evicted and window-relative timestamps change?
9 comments
AuthorsOpenReview ↗

Response - 1-1

Q1: The controlled comparisons simultaneously alter historical coverage, temporal structure, and representational capacity, preventing attribution of the performance gap to “write-time commitment.”The “retrieval” baseline in Section 4.4 uses only eight uniformly sampled frames and removes temporal order and timestamps, while the compression and recurrent baselines are restricted to 64 and 16 tokens, respectively. These comparisons therefore do not isolate write-time commitment as the cause of the performance gap.

A1: The strengthened controls favor preserving native historical evidence for read-time use: native video outperforms learned compression at the closest tested token budgets, while query-conditioned selection adds a modest gain.

Native video reaches 47.9% with approximately 700 history tokens (16 frames), exceeding Perceiver compression at 512 memory tokens (38.3%) by 9.6 percentage points despite receiving 7.5 times fewer source frames (16 versus 120). Under a fully matched 32-frame setup, conditioning frame selection on the instruction and current observations improves success from 55.1% to 56.5%, a 1.4-point gain.

The original comparison combined historical coverage, temporal structure, and representational capacity. We expanded all three memory families using the same backbone, training data, sub-task supervision, and action-head protocol to examine these factors. The strongest tested retrieval, compression, and recurrent-state variants reach 56.5%, 38.3%, and 36.0% on RoboMME, respectively; retaining all 120 sampled frames in native video context reaches 88.3%. The table below summarizes the history representations supplied to the backbone.

MethodReported interface budgetAverage success (%)
Recurrent state256 state tokens36.0
Learned compression512 memory tokens38.3
Native video∼700\sim 700 tokens (16 frames)47.9
Native video∼1400\sim 1400 tokens (32 frames)55.1
Native video∼5300\sim 5300 tokens (120 frames)88.3

Native-video budgets count visual tokens after temporal grouping and spatial merging, together with timestamps and visual delimiters. Each two-frame temporal patch contributes 80 visual tokens, approximately 6–8 timestamp tokens, and two delimiters, giving approximately 0.7K, 1.4K, and 5.3K history tokens for 16, 32, and 120 frames. Compression and recurrent-state budgets count the resulting memory/state tokens supplied to the backbone. Current views and prompt tokens are additional to these history budgets.

Retrieval: temporal structure and sampling density. The new retrieval variants preserve chronological order and use the same native video-processing pathway as SimpleMemVLA. Uniform selection samples two-frame temporal units across the same 60-second history window, with the prompt and collator aligned to the native configuration. At the same 16-frame budget, the timestamped native-video variant reaches 47.9%, compared with 41.9% for the earlier variant. With order and timestamps preserved, increasing coverage to 32 and 120 frames raises success to 55.1% and 88.3%, respectively.

Query-conditioned retrieval uses the instruction and current observations to select historical frames, retains the current temporal unit, and preserves chronological order and timestamps. Its 56.5% result uses the same 60-second candidate history and 32-frame budget as uniform retrieval at 55.1%. This comparison measures the effect of context-dependent selection; the native 120-frame result shows the larger gain obtained by retaining denser evidence.

Compression: architecture and output capacity. We strengthen compression with two Perceiver layers containing cross-attention and feed-forward blocks, temporal encoding before compression, and current observations aligned with SimpleMemVLA. At the same 64-token output budget, these changes improve success from 22.6% to 24.6%. Increasing the output budget to 256 and 512 tokens raises success to 34.3% and 38.3%, respectively. The compressor receives all 120 source frames before producing its memory representation.

Recurrent-state memory: gains from larger states. Success rises from 20.6% with 16 state tokens to 33.4% with 64 and 36.0% with 256. Increasing state capacity helps, with a smaller gain over the upper part of the tested range.

AuthorsOpenReview ↗

Response - 1-2

A1: The closest tested budgets reveal an advantage for preserving sampled evidence in its native format. The 16-frame native configuration retains its sampled evidence without an additional learned compressor and leads the 120-frame Perceiver configuration by 9.6 points, despite receiving 7.5 times fewer source frames. Doubling the compressor output from 256 to 512 tokens adds 4.0 points. An illustrative extrapolation linear in log⁡2\log_2 token budget gives approximately 40.1% at 700 tokens, still 7.8 points below native video; this is an estimate, while the measured comparison is 47.9% versus 38.3%. Together with the matched retrieval result, this pattern supports preserving evidence for decision-conditioned read-time use: the learned compressor reduces the history before the backbone reads it, whereas the native interface leaves the sampled evidence available to that read.

Complete task-level RoboMME results.

Counting: BinF, PickX, SwingX, StopC; Permanence: VU, BU, VUS, BUS; Reference: PickH, VR, VPB, VPO; Imitation: MoveC, InsP, PatL, RouteS.

Retrieval-based memory

Memory mechanismBudgetBinFPickXSwingXStopCVUBUVUSBUSPickHVRVPBVPOMoveCInsPPatLRouteSAvg.
Retrieval (uniform, original)8 frames38347638765820141801818688101031.5
Retrieval (uniform, no TS)16 frames4886725474422224302244367414161241.9
Retrieval (uniform, TS)16 frames4810084628256322628205848881214847.9
Retrieval (uniform, TS)32 frames52100867484624838323466548424261855.1
Retrieval (query, TS)32 frames4498726864685252384864627632382856.5
Retrieval (SimpleMemVLA native)120 frames7810096921001009094906486909446949888.3

Compression-based memory

Memory mechanismBudgetBinFPickXSwingXStopCVUBUVUSBUSPickHVRVPBVPOMoveCInsPPatLRouteSAvg.
Compression (original)64 tokens2650921618121410166161256010822.6
Compression (Perceiver, TE)64 tokens325272161824141218122418521281024.6
Compression (Perceiver, TE)256 tokens449292362632101422203832624101434.3
Compression (Perceiver, TE)512 tokens4294903824321226343242345610242238.3

Recurrent-state memory

Memory mechanismBudgetBinFPickXSwingXStopCVUBUVUSBUSPickHVRVPBVPOMoveCInsPPatLRouteSAvg.
Recurrent-state (original)16 tokens2636566262430161243420160101420.6
Recurrent-state64 tokens464266144452524826182628248182233.4
Recurrent-state256 tokens585446244844565224163432326222836.0

Retrieval budgets count sampled frames supplied to the VLM; compression and recurrent-state budgets count memory tokens. TS: timestamps; TE: temporal encoding. The original retrieval baseline is unordered and has no timestamps; the new retrieval variants preserve temporal order. Perceiver denotes the strengthened two-layer architecture. The 120-frame retrieval configuration selects all frames sampled at 2 FPS over the 60-second history window and preserves their temporal order and timestamps, making it equivalent to SimpleMemVLA's native video-context memory.

Task names: BinF = BinFill; PickX = PickXtimes; SwingX = SwingXtimes; StopC = StopCube; VU = VideoUnmask; BU = ButtonUnmask; VUS = VideoUnmaskSwap; BUS = ButtonUnmaskSwap; PickH = PickHighlight; VR = VideoRepick; VPB = VideoPlaceButton; VPO = VideoPlaceOrder; MoveC = MoveCube; InsP = InsertPeg; PatL = PatternLock; RouteS = RouteStick.

In the revision, we will foreground the closest-budget native-versus-compression comparison, retain the expanded table and token accounting, and state that dense native history with read-time integration is the strongest measured interface in this setting, with write-time commitment presented as a mechanistic interpretation of the observed pattern.

AuthorsOpenReview ↗

Response - 2

Q2: Low decision-path latency with long histories does not establish sustained real-time operation.The standard method uses window-relative timestamps, requiring the history to be re-encoded as the window slides; background prefilling therefore retains a computational cost that grows with context length. In the 45-minute experiment in Figure 5, background processing exceeds the action-execution budget, and decision-path latency itself reaches 1.18 seconds. These results do not demonstrate that the system can sustain the indicated control cadence.

A2: The 45-minute experiment is a scaling probe, whereas sustained real-time operation is evaluated in the bounded SWA schedule: over 153 consecutive decisions, the full streaming pipeline has a 0.92 s median decision latency against a 0.96 s action-execution interval.

The standard Prefix implementation uses window-relative timestamps, so its background history prefill must rebuild the retained prefix as the window slides. Prefix Prefill moves this work off the decision path, but the background workload still grows with the retained context length. SWA uses episode-absolute timestamps and monotonically increasing positions; after a temporal patch is encoded, its tokens and cache entries remain valid until eviction. Each decision therefore appends one new patch and evicts one expired block, keeping the active cache and per-step update cost bounded independently of episode duration.

The following controlled comparison measures the decision path on one H100. All configurations use the same BF16 checkpoint, batch size 1, and action-generation settings. We report medians over the same 30 paired observations after warm-up.

Latency

Total input tokens (10310^3 tokens)Output text (10310^3 tokens)SF (second)Full (second)Prefix (second)SWA (second)
1.3620.0090.3990.4270.4090.446
1.3620.0210.6740.6870.6790.720
1.3620.0421.1531.1581.1541.194
5.6340.0090.3990.5620.4160.464
5.6340.0210.6740.8430.6990.739
5.6340.0421.1531.3261.1811.225
10.9900.0090.3990.7670.4310.470
10.9900.0210.6741.0440.7080.746
10.9900.0421.1531.5211.1901.230

SF: single-frame input (0.364K tokens); Full: full-history recomputation; Prefix: streaming prefix prefill. For each output length, the same SF result is reused across history lengths. The raw input-token column includes the history before SWA eviction; SWA uses a 5.888K-token active attention window, so its resident cache remains fixed as the raw history grows. The table reports decision-path latency with background history prefill excluded.

For the representative 60-second configuration with 5.634K raw input tokens and 21 output tokens, Prefix Prefill takes 0.699 s versus 0.674 s for the same-architecture single-frame baseline. The controlled SWA entry is 0.739 s for this decision path; it isolates the decision computation after background prefill. In the continuous SWA run, the 0.92 s median includes the full streaming schedule, including background maintenance, and is therefore the relevant sustained-operation measurement.

The artificial 45-minute input experiment exceeds the action-execution budget in standard Prefix mode, and its decision-path latency reaches 1.18 s. This result identifies the scaling limit of window-relative prefix prefill and motivates the SWA schedule, whose episode-absolute timestamps and append-and-evict updates keep the active context bounded.

In the revision, we will label the 45-minute experiment as a context-length scaling probe, report the 0.92 s continuous-SWA measurement alongside its 0.96 s execution interval, and distinguish it from the controlled 0.739 s decision-path entry in the latency table.

Reviewer_JP1NOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SimpleMemVLA, a simple yet effective approach to incorporating long-term visual memory into VLA models. Instead of introducing dedicated memory modules or learned memory compression, the method directly uses temporally sampled historical observations through the VLM backbone's native video interface. Historical information is incorporated into action prediction through generated sub-task representations, while shared-prefix prefill and an exploratory SWA variant are investigated to improve inference efficiency. Experiments on memory-centric manipulation benchmarks demonstrate strong performance, suggesting that native video context can serve as an effective memory mechanism for VLA models.

Critical strengths
  • The idea of using the backbone's native video interface to represent visual history aligns well with the design philosophy of modern LLMs and VLMs, which increasingly leverage long-context modeling capabilities rather than relying on task-specific memory modules. This is a simple yet promising direction that is worth exploration.
  • The proposed method achieves strong results on memory-dependent manipulation benchmarks, providing convincing evidence that directly incorporating historical visual observations can effectively improve a VLA model's ability to reason about past events and perform memory-dependent tasks.
  • The use of shared-prefix prefill and the exploration of SWA are valuable contributions toward making long-context VLA inference more practical. The efficiency experiments demonstrate meaningful reductions in inference latency and illustrate the potential of these techniques for processing long visual histories.
Critical weaknesses
  • The reported latency of 0.68 s for the 60-second memory configuration is still relatively high compared with modern VLA models, some of which can achieve inference frequencies of around 5 Hz with comparably sized VLM backbones and flow-matching action decoders. While the proposed prefill strategy substantially reduces the overhead of processing historical observations, it would be valuable to provide a more detailed latency breakdown and a direct comparison against a single-frame VLA using the same model architecture, hardware, and inference settings. This would help quantify the additional inference cost introduced by visual memory. In this regard, I find the claim in the abstract that the method keeps latency "close to that of a single-frame VLA" somewhat too strong without a more explicit comparison.
  • The evaluation of general-purpose, non-memory-intensive manipulation tasks is limited to LIBERO and LIBERO-Plus, which are relatively homogeneous in terms of their task distributions. Preserving the general-purpose manipulation capabilities of the base policy is an important property of a memory-augmented VLA model. Therefore, including additional commonly used benchmarks with more diverse task settings would provide stronger evidence that incorporating visual memory does not compromise general-purpose control performance. Furthermore, I strongly recommend an ablation comparing the same model with and without historical visual observations on these general-purpose benchmarks to isolate the effect of memory on tasks that do not explicitly require it.
  • The paper would benefit from discussing and comparing against the concurrent work NativeMEM: Native Memory Compression for Long-Horizon Robotic Manipulation (Wang et al., 2026). Both approaches leverage native visual representations to incorporate long-term observation histories into VLA models. A discussion of their conceptual similarities and differences, particularly regarding memory representation, computational efficiency, and manipulation performance, would help clarify the novelty and practical advantages of the proposed method.
Questions for authors
  • How is the token budget of approximately 5.6K tokens for a 60-second history calculated? For example, assuming a memory sampling frequency of 2 FPS and approximately 0.5K tokens per frame, a 60-second history would contain roughly 60K tokens.
  • Figure 2, bottom-left subfigure: Why does the "new obs to decoder" component include green blocks representing language tokens? Are these tokens newly generated at each decision step, or are they reused from the previous inference step?
  • Figure 2, bottom-right subfigure: Why is there an idle period between prefill and decoding in the streaming inference pipeline? My understanding is that prefill should ideally be performed immediately before decoding to incorporate the most recent observations. If prefill is completed earlier, how does the method ensure that the latest visual information is not omitted when predicting the current action chunk?

Minor suggestions

  • The abbreviation "SWA" appears before being explicitly defined. I suggest introducing its full name, "Sliding-Window Attention (SWA)," at its first occurrence to improve readability.
10 comments
AuthorsOpenReview ↗

Response - 1

Q1: The reported latency of 0.68 s for the 60-second memory configuration is still relatively high compared with modern VLA models, some of which can achieve inference frequencies of around 5 Hz with comparably sized VLM backbones and flow-matching action decoders. While the proposed prefill strategy substantially reduces the overhead of processing historical observations, it would be valuable to provide a more detailed latency breakdown and a direct comparison against a single-frame VLA using the same model architecture, hardware, and inference settings. This would help quantify the additional inference cost introduced by visual memory. In this regard, I find the claim in the abstract that the method keeps latency "close to that of a single-frame VLA" somewhat too strong without a more explicit comparison.

A1: Under matched model, hardware, and inference settings, “close to a single-frame VLA” refers to the marginal cost of adding visual memory: the 60-s history adds only 25 ms (3.7%) of decision latency and 0.78 GB (6.6%) of peak GPU memory.

Using the same BF16 checkpoint, H100 GPU, batch size 1, 30-step action chunks, and ten action-denoising steps, the single-frame baseline takes 0.674 s, whereas Prefix Prefill with the 60-s visual history takes 0.699 s. For the representative setting with 5,634 input tokens and 21 output tokens, the latency breakdown is:

SFPrefix
Visual encoding (s)0.0130.024
Decision-time prefill (s)0.0410.041
Text decoding and conditioning preparation (s)0.5290.541
Action DiT (s)0.0910.092
Total decision latency (s)0.6740.699
Latency excluding text stage (s)0.1450.158

Text decoding and conditioning preparation account for 0.529 s, or 78% of the single-frame decision cycle, and this dominant stage is present in both configurations; adding visual memory increases it by only 0.012 s. Excluding this shared text stage, the visual, prefill, and action-processing path takes 0.145–0.158 s, corresponding to approximately 6–7 Hz for this portion of the cycle. The measured increase from adding visual memory is therefore small relative to the end-to-end latency already required by sub-task generation.

In the revision, we will state in the abstract that prefilling the shared visual history during action execution adds only 25 ms (3.7%) to decision latency relative to the same model with a single current frame under matched inference settings. We will also add the matched latency, memory, and component breakdown to the latency discussion and clarify that the 6–7 Hz figure refers to the non-text portion of the decision path.

AuthorsOpenReview ↗

Response - 2

Q2: The evaluation of general-purpose, non-memory-intensive manipulation tasks is limited to LIBERO and LIBERO-Plus, which are relatively homogeneous in terms of their task distributions. Preserving the general-purpose manipulation capabilities of the base policy is an important property of a memory-augmented VLA model. Therefore, including additional commonly used benchmarks with more diverse task settings would provide stronger evidence that incorporating visual memory does not compromise general-purpose control performance. Furthermore, I strongly recommend an ablation comparing the same model with and without historical visual observations on these general-purpose benchmarks to isolate the effect of memory on tasks that do not explicitly require it.

A2: Adding visual history preserves general-purpose control and improves the long-horizon LIBERO suite by 6.2 percentage points under a matched SWA ablation.

We trained current-only and history-conditioned SWA variants with identical architectures, training procedures, and evaluation settings, using 500 trials per suite. The current-only variant receives only the current observation, whereas the history variant additionally receives the retained visual history. Their success rates are:

Observation settingSpatialObjectGoalLongAverage
Current only97.095.897.088.294.5
With history96.898.897.694.496.9
Difference (pp)−0.2-0.2+3.0+3.0+0.6+0.6+6.2+6.2+2.4+2.4

The history variant matches the current-only variant on the non-Long suites within the observed variation (97.0/95.8/97.0 versus 96.8/98.8/97.6) and raises Long success from 88.2% to 94.4%. The average therefore increases from 94.5% to 96.9%. These results show that adding visual history preserves general-purpose manipulation and provides its clearest benefit when the task requires longer temporal context.

For broader task coverage, we additionally evaluated SimpleMemVLA on RoboDojo-Sim, which contains 42 tasks across five capability dimensions. SimpleMemVLA obtains a 33.71 score on the Memory dimension, compared with 5.895.89 for π0.5\pi_{0.5}, 4.764.76 for X-VLA, and 4.924.92 for EventVLA. This result extends the evaluation beyond the LIBERO family; the leaderboard models use different backbones and pretraining setups, so we use it as complementary coverage evidence rather than as a matched history ablation.

In the revision, we will present the matched LIBERO current-only/history comparison as the direct evidence for general-purpose control preservation and include the RoboDojo-Sim result as broader task coverage with its cross-model evaluation protocol stated explicitly.

AuthorsOpenReview ↗

Response - 3

Q3: The paper would benefit from discussing and comparing against the concurrent work NativeMEM: Native Memory Compression for Long-Horizon Robotic Manipulation (Wang et al., 2026). Both approaches leverage native visual representations to incorporate long-term observation histories into VLA models. A discussion of their conceptual similarities and differences, particularly regarding memory representation, computational efficiency, and manipulation performance, would help clarify the novelty and practical advantages of the proposed method.

A3: NativeMEM and SimpleMemVLA both build long-term memory from pretrained visual representations, but SimpleMemVLA retains native timestamped video while NativeMEM learns a dedicated memory tokenizer; under the reported protocols, SimpleMemVLA is stronger on the shared tasks.

Memory representation. NativeMEM learns an action-supervised tokenizer initialized from the VLA's vision encoder and compresses each historical frame-view pair into one memory token. SimpleMemVLA directly supplies sampled, temporally ordered, timestamped observations through the backbone's native video interface, without an additional learned memory compressor. This preserves the input structure used during video pretraining and lets the backbone integrate historical evidence with the current decision context.

Computational efficiency. NativeMEM reduces the historical token budget through learned compression. SimpleMemVLA retains a larger native video context and overlaps shared-history prefill with action execution, moving history processing off the critical decision path. The two methods therefore trade compact memory representations against dense native context with overlapped computation.

Manipulation performance. Our evaluation covers all ten RMBench tasks, whereas NativeMEM reports results on six. We compare the six shared RMBench tasks and the four LIBERO suites below. The reported success rates (%) are:

RMBench: memory-centric manipulation

TaskNativeMEMSimpleMemVLA
Observe and Pickup44.065.0
Rearrange Block98.0100.0
Put Back Block100.0100.0
Swap Blocks94.0100.0
Swap T69.093.0
Press Button5.0$^\dagger$100.0
Average (6 tasks)68.393.0

LIBERO: general-purpose manipulation

SuiteNativeMEM$_{128}$NativeMEM$_{512}$SimpleMemVLA
Spatial94.495.298.2
Object98.298.298.8
Goal94.495.698.0
Long (LIBERO-10)88.687.095.0
Average93.994.097.5

The RMBench averages are computed over the six listed tasks. †^\dagger NativeMEM uses external number recognition on Press Button. The subscripts denote NativeMEM's memory-token budgets on LIBERO. SimpleMemVLA obtains higher average success on the shared RMBench tasks (93.0% versus 68.3%) and on LIBERO (97.5% versus NativeMEM's best reported average of 94.0%).

In the revision, we will add this representation, efficiency, and shared-task comparison to the related-work and experimental discussion, together with the evaluation protocols for both methods.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Brief Summary to Rebuttal

We provide a summary of our main contributions and rebuttal updates.

Our central contribution is to show that a pretrained VLM’s native video-context understanding can directly serve as robotic memory, achieving leading results on four memory benchmarks without a dedicated memory module. During the rebuttal, we strengthened the evidence and addressed the main concerns in three areas:

  • Stronger controlled comparisons. We expanded the evaluation to 13 memory configurations, including query-conditioned retrieval and stronger compression and recurrent baselines, and added a fixed-executor oracle control. We also refined the interpretation to account for differences in historical coverage and capacity.

  • Explicit deployment costs. Matched H100 measurements show that a 60-second history adds 25 ms (3.7%) of decision-path latency over single-frame input. We additionally report component timings, GPU memory, and continuous SWA operation, distinguishing decision latency from background processing.

  • Broader validation and clearer protocols. We added current-only versus history-conditioned LIBERO experiments, broader RoboDojo evaluation, and comparisons with NativeMEM.

We also clarified the token-budget calculation; prompt-token reuse and sub-task generation in Figure 2; prefill scheduling and incorporation of the latest observations; history re-encoding as the window slides; the distinction between the 45-minute scaling probe and continuous operation; memory-budget definitions and the scope of cross-method comparisons; the role of established caching techniques relative to our contribution; and the annotation context, causal policy inputs, temporal sampling, timestamp conventions, and supervision for distant historical evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.