acceptodds
Withdrawn from ICLR 2027

Hybrid Memory-Augmented Vision-Language-Action Models for Long-Horizon Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-Language-Action (VLA) models have emerged as a powerful paradigm for closed-loop robot control, yet their limited memory remains a critical bottleneck for long-horizon embodied manipulation. Existing memory-augmented VLAs typically rely on short temporal windows, recurrent memory, or explicit visual memories. These approaches either emphasize recent context, compress away long-range evidence, or retain redundant historical observations, making it difficult to jointly maintain temporal continuity and preserve task-relevant evidence over long horizons. To address these limitations, we propose HERM-VLA, a Hybrid Episodic–Recurrent Memory Vision-Language-Action Model that combines recurrent state propagation with explicit episodic evidence retention. The Recurrent Working Memory maintains a compact latent state to track task evolution, while the Episodic Evidence Memory preserves informative historical context as compact snapshots. An action-aware evidence consolidation mechanism selects action-relevant visual features together with cognitive and spatial context, reducing redundant historical information. The resulting episodic memory is summarized and integrated with the recurrent state and current context for action prediction, enabling effective long-horizon control. Experiments on long-horizon manipulation tasks demonstrate the effectiveness of HERM-VLA and the complementary roles of its two memory components.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision Language Action Models, Embodied Agents

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_taz7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HERM-VLA combines recurrent state tracking with episodic snapshots of selected visual tokens, cognitive features, and coarse spatial descriptors. A temporal Q-Former summarizes the snapshots for action prediction. Experiments on RoboCasa365 and RMBench report gains over the listed baselines.

Critical strengths
  1. Table 1 reports gains over GR00T N1.5 on seen compositions (14.8% to 29.0%) and unseen compositions (2.7% to 11.2%). RMBench success is 68.6% (Table 2).
  2. Table 3 shows that adding episodic memory to the recurrent model raises success on the RoboCasa365 Atomic target split from 66.22% to 68.56%, supporting the hybrid design.
Critical weaknesses
  1. Table 3 omits an episodic memory alone baseline, and the ablations cover only Atomic tasks. The action attention selector is not compared with random or uniform selection at the same token budget.
  2. Table 1 does not specify the averaging weights: 37.6% is reported, while the unweighted category mean is 36.27%. It is also unclear whether Table 3's 59.11% memory-free baseline uses the same evaluation settings as Table 1's 50.7% GR00T N1.5 result.
Questions for authors
  1. How does episodic memory alone perform with matched capacity and training?
  2. Does action attention outperform random or uniform selection at the same token budget?
  3. How are Table 1's averages computed, and are the Table 1 and Table 3 baseline settings comparable?
  4. Does Table 2 use EventVLA's visual anchors only configuration, and is the 0.8 percentage point difference consistent across repeated evaluations?
Reviewer_wqWaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes HERM-VLA, a hybrid memory-augmented VLA for long-horizon manipulation that separates two roles of memory. A Recurrent Working Memory maintains a compact latent state of a few tokens, which is updated every action chunk through gated cross-attention to track the evolving task state. An Episodic Evidence Memory stores snapshot which consists of cognitive tokens and are aggregated by a temporal Q-Former. Built on GR00T N1.5, HERM-VLA achieves 37.6% average success on RoboCasa365. Built on Qwen-OFT, it achieves 68.6% on RMBench.

Critical strengths

Strengths

  • The paper is easy to follow.
  • The target problem that existing memory designs face a trade-off between maintaining temporal continuity and selectively preserving task-relevant evidence is quite intuitive and decoupling these two roles into complementary memories is a reasonable design choice.
Critical weaknesses

Weaknesses

[W1] Incomplete and indirect comparison. In Table 1, HERM-VLA is built on GR00T N1.5 but compared against stronger or more recent models such as GR00T N1.6 and RLDX-1. These comparisons therefore mix the effect of the memory design with differences in backbone. Against the strongest baseline, RLDX-1, the improvement is marginal (37.6% vs. 36.0% on average). On RMBench, the improvement over EventVLA is similarly small (68.6% vs. 67.8%). Moreover, the two backbones are never evaluated on the same benchmark, so the generality of the gains across backbones is not established. In addition, there are also no real-world experiments, which leaves open whether the method is effective for long-horizon execution beyond simulation.

[W2] Limited memory baselines. Many memory-augmented VLAs are discussed in the related work. Yet the main RoboCasa365 comparison (Table 1) includes only one method with an explicit history mechanism (RLDX-1). Other memory baselines appear only partially, in the RMBench table (Table 2). It is therefore unclear whether HERM-VLA’s advantage holds against existing memory designs under the same backbone and benchmark.

Questions for authors

Questions

[Q1] Why was GR00T N1.5 chosen as the backbone? Do the gains remain when HERM-VLA is applied to more recent or stronger backbones (e.g., GR00T N1.6, π0.5), and when both backbones are evaluated on the same benchmark?

[Q2] Is HERM-VLA still advantageous over existing memory-augmented VLAs (e.g., MemoryVLA, EventVLA, ReMem-VLA) when they are implemented on the same backbone and evaluated on RoboCasa365?

[Q3] How long are the episodes in the evaluated benchmarks, and how far back can the episodic bank retain evidence under its replacement strategy? It would be helpful to evaluate on recent memory benchmarks with substantially longer horizons (e.g., RoboDojo) and to report performance as a function of episode length.

[Q4] What is the additional inference latency, memory footprint, and training cost of HERM-VLA relative to the GR00T N1.5 backbone?

Reviewer_QR9JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents HERM-VLA, a memory-augmented VLA policy combining recurrent working memory with an explicit episodic evidence bank. A gated update propagates compact recurrent states, while each episodic snapshot stores action-attention-selected visual tokens, cognitive tokens, and coarse spatial descriptors. A temporal Q-Former summarizes the snapshots, and the resulting memory tokens are integrated with current observations for action prediction. The method also uses auxiliary action and progress prediction losses. Experiments instantiate the framework on GR00T N1.5 for RoboCasa365 and Qwen-OFT for RMBench, with component ablations and one qualitative rollout.

Critical strengths
  1. A reasonable design for complementary memory requirements. Compact recurrent states and explicitly retained evidence address different aspects of partial observability. The combination is well motivated, and retaining both visual detail and higher-level context is a plausible way to support decisions that depend on earlier observations.
  2. Promising results on two simulation benchmarks and two backbone instantiations. On RoboCasa365, average success increases from 23.9% for GR00T N1.5 to 37.6%, including a Composite Unseen improvement from 2.7% to 11.2% (Table 1). The RMBench result of 68.6% and the snapshot-composition ablations provide additional evidence that the framework merits investigation.
Critical weaknesses
  1. No real-robot validation of the proposed memory mechanism. Both evaluated benchmarks are simulated, and Section 5 explicitly leaves real-world long-horizon evaluation to future work. Consequently, the paper does not establish whether retained evidence and recurrent states remain reliable under real visual noise, occlusion, execution errors, and accumulated state mismatch. This limits the practical support for a robot-control contribution. Simulation-only evaluation is not inherently invalid, but the current evidence does not establish physical deployment robustness.
  2. The novelty of the hybrid memory mechanism is not sufficiently distinguished from prior work. MemoryVLA already combines working representations with a perceptual-cognitive memory bank; ReMem-VLA propagates recurrent query-based memory; and EventVLA selectively preserves visual evidence. These works are already cited, so the issue is the incremental contribution rather than missing citations. HERM-VLA's recurrent/episodic combination and action-aware selection may be useful, but the paper needs stronger evidence that this particular combination provides a distinctive advantage beyond assembling established components.
  3. The visual explanation and qualitative evidence are insufficient for the central memory claim. The paper contains an overview diagram (Figure 1) and one stacking rollout with attention maps (Figure 2). The rollout shows current evidence selection, but does not trace which earlier snapshot is later accessed, what information it supplies, or when it changes the action. There is also no visual comparison with a memory baseline or failure analysis under delayed recall. The missing evidence is therefore a concrete account of how memory resolves ambiguity, rather than simply a larger number of figures.
  4. The ablations do not adequately establish the claimed complementarity and action-aware selection benefit. Table 3 includes no-memory, recurrent-only, and hybrid configurations, but no episodic-only configuration. Moreover, the ablations are restricted to RoboCasa365's Atomic split rather than Composite tasks or RMBench. There is no matched-budget comparison of action-attention selection against random or uniform visual-token selection. Thus, the existing gains do not sufficiently isolate the hybrid mechanism or show that current-action relevance preserves evidence useful for later decisions.
  5. Statistical and deployment reporting is incomplete. The gains over the strongest reported alternatives are 1.6 percentage points on RoboCasa365 average success and 0.8 points on RMBench, but evaluation episode counts, repeated-run variation, and confidence intervals are not reported. The paper also provides no measured inference latency or peak memory despite motivating compact evidence storage by computational cost. These omissions make the reliability of the incremental gains and the practical cost of the added memory modules difficult to assess.
Questions for authors
  1. Deployment validation: Is any real-robot evaluation available on tasks requiring recall after the relevant cue disappears? If not, what evidence supports robustness beyond the simulated settings, and how should the practical claims be scoped?
  2. Contribution and attribution: What is the key capability introduced beyond the cited memory approaches? Under the same backbone and comparable memory budget, how does the hybrid design compare with episodic-only memory, and does action-aware token selection outperform a simple selection control on memory-dependent tasks?
  3. Visual evidence: Can you show a representative rollout tracing evidence storage, later memory access, and the resulting action, together with a baseline or failure case? This would clarify when the proposed memory is necessary rather than merely present.
  4. Evaluation and cost clarification: How many episodes and training seeds support the reported success rates, and what uncertainty accompanies the small gains over RLDX-1 and EventVLA? Please also report inference latency and peak memory relative to the corresponding backbone. Finally, what protocol difference explains the no-memory Atomic result of 59.11% in Table 3 versus GR00T N1.5's 50.7% in Table 1?
Reviewer_m7weOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HERM-VLA, a vision–language–action model with a hybrid memory architecture for long-horizon robotic manipulation. Its recurrent working memory carries a compact hidden state across action chunks to track task progress. Its episodic evidence memory stores visual features selected by action attention, together with cognitive features and coarse-grained spatial descriptors. A temporal Q-Former aggregates these historical representations to inform subsequent action prediction. The authors evaluate HERM-VLA on RoboCasa365 and RMBench, using GR00T N1.5 and Qwen-OFT as the respective backbones. On RoboCasa365, HERM-VLA achieves an average success rate of 37.6%, compared with 36.0% for the strongest reported baseline. On RMBench, it achieves the highest average success rate among the methods compared.

Critical strengths
  1. Recurrent memory tracks task progress across successive action chunks, while episodic snapshots preserve historical evidence for subsequent action prediction. As shown in Table 3, augmenting recurrent memory with episodic memory increases the atomic-task success rate from 66.22% to 68.56%.

  2. As shown in Table 5, with the visual and cognitive token configurations held fixed, adding a 2×22 \times 2 grid of spatial descriptors per view increases the success rate from 65.78% to 68.56%. This result suggests that coarse-grained spatial layout provides information complementary to the selected local visual features.

Critical weaknesses
  1. The component ablation in Table 3 is limited to atomic tasks in RoboCasa365 and omits an episodic-memory-only control. It therefore measures only the incremental benefit of adding episodic memory to recurrent memory. The available results do not establish whether hybrid memory outperforms either component in isolation or whether its gains are concentrated in tasks that require recall after long delays. Moreover, Table 5 varies the contents of episodic snapshots, and Figure 2 presents a single qualitative example. Neither compares action-attention-based token selection with alternative selection rules under a matched storage budget. The evidence is therefore insufficient to substantiate the paper’s claims about the roles of hybrid memory and action-aware token selection in long-horizon tasks.

  2. In Table 1, HERM-VLA exceeds the strongest baseline by 1.6 percentage points in average success rate. On unseen composite tasks, it achieves a success rate of 11.2%, compared with 9.0% for the strongest baseline in that category. These point estimates alone provide limited insight into the sensitivity of the reported gains to training seeds and evaluation trajectories. Furthermore, GR00T N1.5 in Table 1 and the no-memory control in Table 3 achieve atomic-task success rates of 50.7% and 59.11%, respectively, a difference of 8.41 percentage points. Clarifying the training and evaluation configurations for both settings is necessary to determine how much of the improvement over the backbone reported in Table 1 can be attributed to the proposed method.

Questions for authors
  1. To clarify the comparability of the two baselines, could the authors specify how GR00T N1.5 in Table 1 and the no-memory model in Table 3 differ in initialization, training data, number of training steps, auxiliary losses, and evaluation trajectories? Under training and evaluation settings matched to those of the full model, what success rates does the no-memory model achieve on seen and unseen composite tasks, respectively?

  2. To isolate the respective contributions of hybrid memory and action-attention-based token selection, could the authors report results for the recurrent-memory-only model, the episodic-memory-only model, and the full model separately on RMBench’s single-step recall and multi-step integration tasks? With the number of retained tokens per view and the snapshot capacity held fixed, how much does action-attention-based selection improve performance over uniform or random selection? If repeated runs are available, please also report the number of runs and the corresponding uncertainty estimates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.