acceptodds
Withdrawn from ICLR 2027

LTM: Latent Transition Based In-Context Memory for Vision-Language-Action Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

Robotic manipulation requires remembering interactions that are no longer observable in the current scene. Existing memory mechanisms face a trade-off between retaining detailed histories and maintaining a compact policy context. We introduce Latent Transition Memory (LTM), which represents interaction history as an ordered sequence of compact latent transitions between consecutive observations. A frozen latent action encoder extracts each transition, and a lightweight adapter maps it to a single memory token, preserving a record of observed changes at one token per transition. We instantiate LTM on as LTM-, using causal attention and persistent KV caching for incremental inference, with sparse visual anchors to ground video demonstrations. Across two pretrained latent encoders, LTM- achieves up to 58.54% success on RoboMME and 80.5% on LIBERO-Mem, exceeding the reported memory-free baseline by 40.61 percentage points and the strongest compared LIBERO-Mem baseline by 38.0 points, respectively. On a real robot, it improves average exact-count accuracy from 11.25% to 82.50% in a pick-and-place \(x\) times task. Ablations support the contribution of transition content beyond temporal cues, while a controlled swap diagnostic shows that relevant motion information can remain recoverable even when downstream state tracking fails. These findings establish latent transitions as an effective, compact memory representation and highlight temporal composition as a remaining challenge.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA memory, latent action model, embodied intelligence

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zCkAOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Latent Transition Memory (LTM), a memory mechanism for VLAs that represents interaction history using compact latent transitions between consecutive observations. A frozen latent action encoder extracts a transition representation from each observation pair, which is projected into a single memory token and provided to the policy as an ordered memory sequence. For video-conditioned tasks, sparse visual anchors are additionally used to provide visual grounding.

Critical strengths
  1. The paper addresses an important limitation of current VLAs under partial observability.
  2. The proposed memory representation is compact and inference-efficient.
Critical weaknesses
  1. The performance seems marginal. In particular, the proposed method performs worse than PonderPounce on the RoboMME benchmark.

  2. Several recent works [1-4] already compress past observations into compact representations in a simpler way and compress the observations significantly. This paper does not include a comparison to these approaches or an in-depth discussion of why the latent action provides better context to VLAs.

  3. A large gain seems to come from temporal cues rather than transition-specific content. Replacing the latent transitions with fixed random representations still achieves 44.50% success, compared with 57.67% for the full model. In several counting tasks, the random-token variant is even competitive with or better than the full model. This suggests that sequence length or temporal position alone explains a substantial portion of the improvement, making the contribution of the learned transition semantics less clear.

  4. In real-world experiments, pi0.5 significantly underperform even in x=1 problem, but the proposed method gets 95%. This unusually large gap suggests that factors beyond long-term memory, such as training stability or temporal conditioning, may contribute substantially to the reported improvement.

  5. Minor issue: The main result tables are difficult to interpret because they contain too many task-level scores without sufficiently summarizing the key trends. Moreover, the table captions provide limited information about the experimental settings under which these results are obtained, making the comparisons difficult to follow without repeatedly referring back to the main text.

[1] Jang et al., ContextVLA: Vision-Language-Action Model with Amortized Multi-Frame Context, 2025
[2] Torne et al., MEM: Multi-Scale Embodied Memory for Vision Language Action Models, 2026
[3] Kwon et al., Exploring High-Order Self-Similarity for Video Understanding, 2026
[4] Physical intelligence, π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities, 2026

Questions for authors

Please answer the weaknesses part.

Reviewer_1YbhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a visual memory problem in VLA models. It motivate the need of visual memory by showing some examples, such as when the starting and ending frames of an action look similar, the model needs a mechanism to remember what happened in between. It proposes to represent the transition using latent transition vectors from a pretrained encoder, which is learned by reconstructing the next frame from the previous frame and the transition vector. These vectors are then provided as input to the policy, serving as memory of scene changes between consecutive observations. Results show improvements on RoboMME, LIBERO Mem, and a real robot counting task.

Critical strengths
  1. The paper targets an important question. The scenario where interaction info is absent from observation seems to be common in physical AI. Also, the proposal of using compact representations to model such information is interesting and worth investigating.
  2. The approach is simple and shows useful results across benchmarks and a real robot task.
  3. The ablations and discussions, such as that preserving information about past changes does not necessarily mean a policy can use it to track the current state, is valuable and could inspire future research problems.
Critical weaknesses
  1. The novelty of the solution seems limited. The transition encoder comes from previous work, and the main idea is to feed its vectors into the policy as memory. Also, prior works such as HiF VLA already uses motion info to represent history.
    1. Also, there are baselines with even simpler design we should test. For example, what if we just use pooled image feature for each frame or the difference between consecutive image features. This would make it clearer whether the proposed representation is actually better.
  2. It is still unclear how much the model relies on timing to solve the counting tasks. Random memory tokens already achieve 44.5%, compared with 57.67% for the full model. A simple time or step count input would be a useful baseline, especially for the real robot task. Testing pauses or failed attempts would also help show whether the model remembers what actually happened, rather than when an action usually finishes.
  3. The efficiency results are not complete. The reported 67 ms only measures policy inference, but getting the transition vectors also takes time. The paper should measure the full system latency, including the encoder running alongside action execution. Also, the memory keeps growing, so it would help to show how latency and GPU memory usage change with longer histories.
  4. The method still struggles with some object tracking and swap tasks. The Cup Swap study is useful, but it uses a different model and additional supervision. It therefore does not fully explain why the robot policy fails. Testing this explanation on the actual robot task observations would make the discussion more convincing.
Questions for authors

Please refer to the Major Weaknesses section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 1Ybh

  1. Our primary contribution is to propose using a Latent Action Model (LAM) as the memory representation for VLAs, along with several inference optimizations, including lagged inputs and KV-cache reuse. In fact, our experimental baselines already include methods that use pooled visual tokens to represent historical information, such as Frame Sampling and Token Dropping, both of which perform relatively poorly (with the best result reaching only 44.51%). We additionally evaluated a transition representation based on the video compression encoding approach proposed in HiF-VLA, which achieved a slightly better result of 51.3%, but still fell substantially short of our method.

  2. We also observed this phenomenon in our experiments. For certain tasks, such as counting, feeding fixed random transitions according to timestamps can achieve relatively good performance. Upon further analysis, we found that this behavior reflects a dataset-specific shortcut. Since the manipulation tasks in the training set are performed at approximately the same speed and contain no external disturbances, the model can exploit temporal cues alone, learning a shortcut that associates the number of manipulations with their duration. However, after introducing random disturbances and variations in execution speed into both the training and test sets to invalidate this temporal shortcut, the success rate of the ablation relying solely on temporal cues dropped to 24% on the PickXtimes and SwingXtimes tasks, whereas the model using correct transitions maintained a success rate of 94%. This demonstrates that transitions provide meaningful information beyond temporal cues alone.

  3. In fact, we have already discussed the latency details in the paragraph analyzing the effect of lag in our ablation study. With lag = 5, the transition encoding latency can be largely hidden through asynchronous inference, overlapping with policy execution and computation, while introducing almost no degradation in performance. Please refer to the corresponding section of our paper for further details.

We sincerely thank you for your careful review and constructive feedback. We will further refine and improve the paper accordingly.

Reviewer_1YbhOpenReview ↗

Review update

Thank you for the response. The new counting experiment helps, but a few concerns remain:

  1. The two responses report different counting results. Please clarify the settings and number of trials.
  2. It is unclear whether the new HiF VLA comparison uses matched training settings and memory budgets.
  3. Full system latency and memory usage with longer histories remain unclear.

I will therefore maintain my current score.

Reviewer_HrLKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper innovatively proposes Latent Transition Memory (LTM), which explicitly stores interaction-related information rather than directly retaining historical visual observations. This design effectively reduces the contextual overhead while preserving essential information about past interactions. Building upon this mechanism, the authors further develop the LTM- π model, which achieves substantial performance improvements over multiple baseline methods across various benchmarks.

Critical strengths
  1. The motivation of this article is clear.
  2. The shallow representation idea alleviates data heterogeneous.
  3. The experimental results also verify the effectiveness of LTM.
Critical weaknesses
  1. Is LTM primarily suitable for scenarios with relatively limited environmental changes? In environments involving rapid scene changes, multiple moving objects, or frequent interactions, would a larger memory capacity be required to preserve sufficient information about historical interactions? The authors are encouraged to clarify how the complexity and frequency of environmental changes affect the information retention capability and memory requirements of LTM, particularly in long-horizon manipulation tasks.

  2. The paper uses pretrained Latent Action Models to encode transitions between consecutive observations into memory tokens. However, existing works such as LAPA [1] and DreamDojo [2] have already explored latent action representations, while HAMLET [3] has investigated token-based memory for VLA models. Therefore, the proposed method appears to combine these existing techniques. The authors should better clarify the novelty of LTM and explain its advantages over these related approaches.

  3. Although each transition is compressed into a single memory token, the memory sequence still grows linearly with the number of interactions. It is therefore unclear whether LTM remains efficient in long-horizon tasks. The authors should provide further analysis of memory consumption and inference overhead as the history length increases.

  4. Although LTM-π achieves promising results, some baselines are taken from previous studies under different experimental settings. Moreover, the main ablation studies use only one random seed. The authors should provide more controlled comparisons and multi-seed experiments to better validate the effectiveness of LTM.

  5. Some mistakes:

(1) Chung et al. (2026a) and Chung et al. (2026b) has the same topics and DOI

(2) Section 4.2.1, Page 8, "For real-world test" should be "For the real-world tests".

(3) Abstract, Page 1, "achieves up to 58.54% success on RoboMME and 80.5% on LIBERO-Mem", I think "achieves average success rates of up to 58.54% on RoboMME and 80.5% on LIBERO-Mem" is better.

[1] Ye, S., Jang, J., Jeon, B., et al. (2025). Latent Action Pretraining from Videos. International Conference on Learning Representations (ICLR 2025).

[2] Gao, S., Liang, W., Zheng, K., et al. (2026). DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos. Proceedings of the 43rd International Conference on Machine Learning (ICML 2026), PMLR 306, 33340–33366.

[3] Koo, M., Choi, D., Kim, T., Lee, K., Kim, C., Seo, Y., & Shin, J. (2026). HAMLET: Switch Your Vision-Language-Action Model into a History-Aware Policy. International Conference on Learning Representations (ICLR 2026).

Questions for authors

As noted in the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.