acceptodds
Under review as a conference paper at ICLR 2027

Keep Everything, Read One Latent: Verbatim Episode Memory for World-Action Models

18 views

Abstract

World-action models (WAMs) typically act from a bounded window of recent ob- servations, which makes their decisions effectively Markovian. We show that a WAM becomes history-dependent with verbatim episode memory (VEM). Every observation, from the demonstration and from execution alike, stays in context as its encoded video latent. After temporal sampling and native VAE encoding, no additional memory compression, selection, or eviction is applied. Training uses expert actions and future video targets, without memory annotations or subgoal supervision. The video expert integrates the causal history into the current latent, while the action expert reads only this fixed-size, history-conditioned state. VEM achieves state-of-the-art overall performance on RoboMME, with a 62.5% aver- age success rate across 16 tasks. Placing the demonstration and execution history on one stream elicits strong imitation capabilities, yielding a 78.2% success rate on the Imitation Suite. This outperforms policies conditioned on ground-truth sub- goals on both our backbone (49.5%) and a language-conditioned VLA (64.0%). In a retrieval test, a single initial observation still steers behavior after more than twice the length of the longest training episode of irrelevant history.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world-action models, robot learning, long-context memory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Z7uGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes VEM, a WAM Architecture & System that directly keeps all historical visual information for video model while keeps only current proprioception and the observation of information comes from the video model for the action expert. The system abandoned any selectors / compactors on the history visual observation, directly uses the video model to attend all history information. VEM also support imitation learning since its history prefix can be directly appended with demonstration videos. On RoboMME, VEM achieves an average success rate of 62.5% across all 16 tasks, outperforming the strongest reported baseline (44.5%) and a fixed-budget variant (53.3%). The paper further introduces a controlled behavioral retrieval experiment, RoboNIAH, and reports strong performance on imitation tasks.

Critical strengths

1. Clear writing and well-motivated memory design

  • The paper has a clear presentation. The design insight of VEM is simple and well-motivated. The idea of keeping the whole visual history and pass through only one current latent frame to the action expert is conceptually clean and architecturally reasonable (for training and inference).

2. Strong empirical results and ablations

  • The improvements on RoboMME are substantial. The ablation results on the fixed-budget and full-history directly show that keeping full observations improves performance over limited context.

3. Interesting findings on demonstration-conditioned imitation

  • The idea of combining demonstrations and execution observations into a unified stream is attractive. The results on InsertPeg and RouteStick indicate that visual demonstrations can be directly plug in for informative guidance.

4. A useful evaluation of long-range behavioral retrieval

  • The RoboNIAH experiment is a good addition. Through varying the distance between a informative observation and the subsequent decision, and comparing with a matched key-frame-excluded experiment, we can see more targeted evidence of long-range information leveraged.
Critical weaknesses

1. Problem formulation and lack of experiments with comparable overheads

  • A key reason why other methods use a memory selector/compactor is the desire to ensure their systems can run stably for a long period of time. While the cost of VEM will eventually increase (and potentially causes failures and crashes) because it continuously records and attends to all historical frames. Since their computational budgets are different, the experiment results can not directly prove that the memory design of VEM is better. I suggest that comparisons between a resource constraint VEM variant and previous works that shares a comparable overheads could be more convincible.

2. Lack of a detailed performance analysis

  • The author reports the latency result on the long history inference. However, It is better to provide more details on memory usages, TOPS and output fps since the KV Cache scales both memory usage and computational budget up linearly.

3. Task-specific performance bias

  • The method exhibits more powerful performance on episodic retrieval while being less powerful on some continuous state-tracking tasks. Is this a side effect resulting from attending to all historical frames? (e.g. if the model cannot distinguish the previous states from latest states well, it could use old information over new information) Could authors provide more analysis on this?
Questions for authors
  • How does VEM model the super-long historical frames that exceeds the base model's capability. (e.g. use an extrapolated positional embedding for the KV cache?)

  • Could the authors provide more explanations on why the method performs particularly well on episodic retrieval but remains weak on continuous state-tracking tasks?

Reviewer_bn4BOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes verbatim episode memory for world-action models, retaining all sampled demonstration and execution video latents in a causal stream. The video backbone integrates history into the current latent, which alone conditions action generation. RoboMME evaluations, matched-backbone ablations, and long-range retrieval tests support the design, achieving 62.5% overall success and 78.2% on imitation tasks.

Critical strengths

Clear design supported by a matched-backbone comparison. Retaining the full episode improves success by 9.2 percentage points over bounded history on the same backbone, providing direct evidence that preserving historical observations benefits control.

Controlled evidence of long-range retrieval. The needle/no-needle experiment shows that an early observation can guide behavior beyond training episode lengths. This supports meaningful use of distant history rather than merely gains from recent context.

Critical weaknesses

The fixed-size action interface is validated only with bounded history. The comparison against direct history access uses 13 latents, rather than full episodes, and does not establish equivalence. This leaves a central claim—effective compression of long histories into the current latent—insufficiently tested.

Scalability remains limited. Retaining all latents produces a growing KV cache, while closed-loop execution collapses at 257 latents in the retrieval test. Without memory-scaling measurements and stronger long-horizon validation, the practicality of “keep everything” remains uncertain.

Questions for authors

NA

Reviewer_WTLuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates whether retaining a robot's visual history improves manipulation tasks that require remembering earlier observations. The proposed verbatim episode memory (VEM) retains encoded observations from both a demonstration and the robot's subsequent execution. A video model uses this history to form a representation of the current observation, which then guides the action model. This design aims to preserve earlier evidence while limiting the visual input that the action model processes at each decision. With VEM, a world-action model built on Wan2.2 achieves a higher average success rate on RoboMME than the same model with a limited memory budget. Further experiments show that the policy can follow visual demonstrations and use an earlier cue after a prolonged delay.

Critical strengths

Simple yet effective design: The model is trained using expert actions and future observations, without keyframe labels or subgoal annotations. In the RoboNIAH experiment, it succeeds in 15 of 16 scenes after a prolonged scripted delay when the initial cue is provided, compared 7/16 when that cue is withheld. These controls provide evidence that earlier visual information contributes to the policy's decisions.

Critical weaknesses
  1. VEM extends Fast-WAM by conditioning the current observation representation on the full visual history, while retaining its action interface and auxiliary video-prediction training. The empirical gains are useful, but the methodological advance over Fast-WAM remains modest.
  2. The performance drop with the fixed-budget baseline suggests that aggressively limiting history discards useful information. However, real-world deployment cannot accommodate indefinitely growing history because of latency, memory capacity, and memory-bandwidth constraints. This leave concerns on the practicality of this method.
Questions for authors
  1. Are there any experiments showing the balance point in number of recent latents to cache while preserving most of VEM’s performance gains? Similar to language models, will there be a point where the context is too long for the model that ultimately adversely affect the model performance?
  2. In Table 4, the comparison between f0f_0-only and full-prefix action reads is conducted only under the fixed budget context. As an additional ablation, how do these two action-read interfaces compare when the video expert retains the full episode as in VEM? Would restricting the action expert to f0f_0 have any accuracy trade-off alongside its efficiency benefits in the full-history setting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.