acceptodds
Withdrawn from ICLR 2027

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

26 views

Abstract

Mainstream Vision-Language-Action (VLA) models predict actions primarily from the current observation under a Markovian assumption, thus struggling with long-horizon, temporally dependent tasks. Existing memory-augmented VLAs either expand the observation window or retrieve history from the memory bank as auxiliary policy-side context. However, they leave memory outside the native latent embedding space of VLA reasoning, preventing historical experience from being fluidly interleaved with multimodal reasoning and action formation. To this end, we introduce LaMem-VLA, a latent-memory-native framework that reconstructs historical experience into latent memory tokens and directly interweaves them with VLA reasoning. At its core, LaMem-VLA introduces four coordinated components: (i) a curator that organizes historical experience into two complementary short-term and long-term memory vaults; (ii) a seeker that queries both vaults using the multimodal cognition to retrieve context-relevant evidence; (iii) a condenser that reconstructs the retrieved evidence into compact short-term and long-term latent memory tokens; and (iv) a weaver that injects these memory tokens with the current observation and instruction into one continuous embedding sequence. By representing, retrieving, and consuming historical experience entirely in the same continuous latent space, LaMem-VLA enables memory to directly participate in VLA reasoning and guide action generation. Experiments on SimplerEnv, LIBERO and MIKASA-Robo demonstrate state-of-the-art success rates of 73.9%, 97.6%, and 51.4%, respectively. In real-world evaluation, LaMem-VLA ranks best on general and long-horizon suites. The code will be released.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Model, Robotic Manipulation, Latent Memory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZyBJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces LaMem-VLA, a new design in the family of memorizable robot generalists. The core is to take the representation of VLA as a nature to construct so-called memory, and reconstruct the historical experience into a latent part that carries memory and guides reasoning. The method consists 4 components that sequentially and explicitly generate and utilize these latent memory tokens in the same representation space for memory-based action generation. The experiments show the effectiveness in both simulation benchmarks and real-world tasks of different purpose (generalizable and long-horizon.)

Critical strengths
  1. As a new way to explore memory utilization in robot generalists, this work seems clear in its process pipeline and interests people to research on the native representation rather than adding other external modules while some comparison may lack further analysis (see Weakness 1). The insight is rather clear and well motivated.

  2. The experiments are conducted properly, and the presentation is detailed. The improvements on MIKASA-Robo and real-world long-horizon tasks are encouraging. The ablation and choice sweeps help me understand the mechanism and clarify the contribution of this framework.

Critical weaknesses
  1. For method novelty, as far as I know, the method itself is not novel enough. After removing decorative expressions, the essence of LaMem-VLA is to move the injection point of memory from "after-VLM" (the work MemoryVLA by Shi et al.) to "before-VLM". The backbone, dual memory back, and specific sequence memory processing all follow MemoryVLA.

  2. For method techniques and motivation. Firstly, the claimed point of pushing memory into the VLM input sequence is well explored, as the authors also classified. Recent works like ContextVLA (Jang et al.) also used similar techniques. Well the author didn't compare with this method. And the "short term" and "long term" description didn't have support in the paper, the capacity are same, two vaults are same in time scale. The labelled "milestones, progress, goals, semantics" in Figure 2 also lack explanation and analysis. The description of memory building and latency analysis confused me of how it work, is it 2 times VLM forward?

  3. For the experiment. The evaluation results as claimed are selected alone, if I understand this correctly, all the simulation results of LaMem-VLA are tuned and tested while MemoryVLA lacks a fair comparison. If the training budgets are different, the gain is relatively small and not promising.

  4. For writing, this paper includes many terms that distract my attention while reading, and I can't understand the proposed framework after many times of back-and-forth checking and alignment. The logic and technical framework should be better positioned and written clearly.

Questions for authors
  1. Regaring the method, Section 3.4 states that the VLA backbone produces Q^con_t from X_t and I, and Eq. 8 then processes S_t with the VLM again. Is Q^con_t obtained from a full LLaMA-7B forward pass on the memory-free sequence, or from the vision encoder or input embeddings? Is the H^action written to the long-term vault taken from that pass or from VLM(S_t)? Please also give a latency breakdown for Table 11 (backbone, memory modules, DiT), and state whether N_con is the full token sequence.

  2. The claimed difference from MemoryVLA and ContextVLA is the injection point. Can you provide a controlled comparison in which the same memory content (same vaults, retrieval, and condenser) is injected (a) before the VLM as in LaMem-VLA, (b) after the VLM as in MemoryVLA, and (c) as compressed past-frame tokens as in ContextVLA, all trained under the same protocol? Without this, the benefit cannot be attributed to the proposed mechanism rather than to extra training or checkpoint selection.

Reviewer_F4nxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LaMem-VLA retrieves visual and action-state histories, compresses them into a small set of tokens, and feeds them into the VLM before action prediction. It reports improvements across three simulation benchmarks and six real-world tasks.

Critical strengths
  • Clear, well-specified architecture; the appendix gives masks, hyper-parameters and update rules in enough detail to reimplement.
  • MIKASA-Robo, a genuinely memory-dependent benchmark, is included, and the +10.2 gain there is the paper's most convincing result.
  • Real-world long-horizon suite is sensibly designed (occlusion, aliased states, elapsed-time dependence) and shows a meaningful +15.3 margin.
Critical weaknesses
  1. The method adds little beyond MemoryVLA, and the novelty claim is overstated.
  • Most of the system is inherited from MemoryVLA: backbone, action head, separate stores for perception and cognition, adjacent-similarity merging, timestep encoding of memory keys, episode-ordered dataloader, all three benchmarks, and even the real-world tasks (e.g., cover-block, sequential buttons).
  • The only real change is where memory enters the model. This paper keeps memory "in the native latent space". "Weaving" latent memory tokens into the context is also close to MemGen.
  • Directly relevant work is missing as comparisons. MemoryVLA++, MEM (2603.03596) and ReMem-VLA are not compared; OptimusVLA [25] (98.6 on LIBERO), HAMLET [21], ContextVLA [58] and MemER [24] are cited but not compared.
  1. "Short-/long-term" is a misnomer. Both vaults span the whole episode with the same capacity, write rate and merge rule; one simply stores pre-LLM visual tokens and the other post-LLM action states, i.e., MemoryVLA's perceptual/cognitive split renamed.

  2. Similarity-based retrieval undermines the motivation. Memory is supposedly needed when the current view is ambiguous, but cosine Top-KK against that same view returns frames that look alike and are just as ambiguous, while the informative ones are by definition dissimilar. Since hard Top-KK is non-differentiable, the query cannot learn to do better, and with K=8K{=}8 of L=16L{=}16 retrieval hardly filters anyway. A comparison against recency, random, or retrieve-all baselines is needed.

Minor Weakness:

  1. Fig. 2 misspells "Conderser" twice; the Fig. 3 legend writes "LaMemVLA".
  2. The motivation paragraph is repeated nearly verbatim in §1, §2 and §3.2; the curator/seeker/condenser/weaver/vault naming reads as decorative. Tighten.

Sources:

Questions for authors
  1. QtconQ^{\text{con}}_t is produced by "the VLA backbone" before memory injection, then the memory-augmented sequence StS_t is run through the VLM again. Is this two 7B forward passes per step? If so, how is latency only +0.076 s over CogACT?
  2. op-KK selection is non-differentiable, and each step's new unit is written only after that step's prediction.
  • How does B\mathcal{B} learn to retrieve useful units?
  • Does the gradient pass back through vault states across the 16-frame group? If not, CsC_s never receives any training signal.
  1. Baseline fairness. In the real-world experiments and in Table 5, did baselines get the same data, the same number of steps, the same best-of-N checkpoint selection, and the same memory length (64 for the long-horizon tasks)?
  2. Retrieval evidence. Can you give quantitative retrieval evidence, e.g., recall of frames known to be decisive on MIKASA, and failure modes when visually similar distractors are present? Fig. 6 shows only one qualitative example.
Reviewer_TEjmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents LaMem-VLA, a framework designed to improve the use of historical information in vision-language-action models. It stores visual features and action hidden states in separate memory banks, retrieves context-relevant evidence, and compresses it into fixed-length dual-source memory tokens. These tokens are injected into the VLM input sequence to guide action representation learning and subsequent diffusion-based action generation. Compared with methods that fuse historical features at the action prediction stage, its main distinction is allowing memory to participate directly in the VLM’s internal representation computation. Evaluations on simulation benchmarks and real-world robotic tasks support its effectiveness in long-horizon manipulation.

Critical strengths
  1. The paper identifies a limitation of methods that incorporate memory primarily into the action expert: historical information has limited involvement in the VLM’s reasoning process. This motivates integrating memory directly into the VLM.
  2. The framework comprises four clearly defined components—curator, seeker, condenser, and weaver—with distinct roles in memory management, retrieval, compression, and integration.
  3. The paper is well organized, making the proposed method accessible and understandable.
Critical weaknesses
  1. The proposed method maintains two memory banks that store historical visual features and action-token representations, respectively. Although the action tokens are described as memory-grounded, both banks accumulate information from the beginning of each episode, follow similar update and consolidation mechanisms, and support the retrieval of multiple units for subsequent compression. It therefore remains unclear how these mechanisms establish distinct temporal scopes beyond differences in the stored representations. The authors should clarify what makes one memory short-term and the other long-term, and provide evidence supporting this distinction.
  2. Although the method targets memory-augmented VLAs, most simulation and ablation experiments use benchmarks that do not explicitly require historical information to resolve the current task state. Moreover, most comparison baselines lack dedicated memory mechanisms. These evaluations therefore provide limited evidence for the proposed method’s advantages in memory-dependent decision-making. More extensive experiments on tasks with explicit temporal dependencies, together with comparisons against representative memory-augmented VLAs, would strengthen the empirical support.
Questions for authors

The supplementary ablation study shows that increasing memory capacity improves performance on LIBERO Long-10 before reaching a plateau, whereas performance on SimplerEnv decreases at (L=32) and (L=64). Given the subsequent retrieval mechanism, one might expect a larger memory bank to preserve access to relevant evidence while filtering out unnecessary candidates. Does this degradation indicate limitations in the memory design, consolidation strategy, or retrieval mechanism?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.