acceptodds
Under review as a conference paper at ICLR 2027

Remember Smarter: Visual History Compressor and Hyperbolic Experience Space for Robotic Memory

Abstract

Zero-shot transfer in complex long-horizon manipulation depends on reusing successful experiences acquired while executing simpler subtasks. This setting requires a policy to retain visual context throughout the current trajectory and retrieve relevant experience from other trajectories. However, many VLA models use only a bounded set of recent visual frames and lack an explicit mechanism for reusing successful prior trajectories. Mamba scales linearly with sequence length and is therefore well suited to compressing visual histories, while hyperbolic space can represent hierarchical relations among related experiences. Based on these properties, we propose Remember Smarter (RS), a module that adds a visual history compressor and a hyperbolic experience memory to existing VLA models. The visual branch uses a bidirectional spatial Mamba and a causal temporal Mamba to summarize patch sequences from multiple camera views. Residual cross-attention adds this summary to the VLM states used for action generation while preserving the original sequence of visual tokens. The experience branch encodes final-layer VLM states from successful simple-subtask trajectories into a Poincaré VAE latent space and organizes the resulting codes with a HypHC-inspired binary topology. Retrieved experiences are converted into prompt tokens for action generation. When integrated with , RS improves the overall success rate on LIBERO-Plus by 16.9 percentage points over the base policy. RS also improves real-robot performance, including zero-shot transfer to task compositions that require maintaining object-name grounding, coping with visual occlusion, or recognizing state changes that are visually subtle or unobservable, such as button presses.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotics, visual compressor, experience memory, hyperbolic space

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_wpTeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the limited ability of vision-language-action (VLA) models to retain historical information and reuse experience across tasks in long-horizon robotic manipulation. It proposes Remember Smarter (RS), a memory augmentation module that uses spatial and temporal Mamba to compress multi-view visual histories and organizes successful trajectories in hyperbolic space for asynchronous retrieval. Retrieved experiences are converted into prompt tokens to support action generation. Experiments with π0 show that RS improves overall success on LIBERO-Plus from 53.6% to 70.6% and yields gains on real-robot tasks involving counting, occlusion, and complex task composition.

Critical strengths
  1. The Mamba-based visual history compressor provides an efficient way to equip VLA models with temporal memory, with computational cost scaling linearly with sequence length.
  2. The motivation for introducing experience memory is compelling: reusing successful experiences from simpler subtasks offers a promising way to support generalization to complex, long-horizon task compositions.
Critical weaknesses
  1. The proposed method is relatively straightforward, combining Mamba-based compression of historical tokens with the introduction of “experience” tokens, and its novelty appears limited. The necessity of the Poincaré VAE also remains unclear. Comparisons with simpler encoders and ablations of variational modeling, hyperbolic geometry, and hierarchical organization are needed.

  2. The experimental evaluation has substantial limitations. Improvements over a memory-free baseline, without achieving state-of-the-art performance, are insufficient to establish the method’s superiority. Moreover, LIBERO-Plus primarily evaluates robustness rather than memory-dependent reasoning, making it difficult to attribute the gains to the proposed memory mechanisms. Explicitly memory-dependent tasks are needed. The real-robot experiments include only π0 and component ablations; comparisons with existing memory-augmented methods are necessary to establish the approach’s relative advantages.

Questions for authors
  1. Experience tokens are masked during policy training and enabled only at inference, while the experience branch is trained separately. Does this introduce a training–inference mismatch, and how does the VLA learn to use these tokens without joint optimization?
  2. Lines 422–425 appear inconsistent with Figure 5: the text attributes results to π0+RS, while the corresponding values are labeled as π0+VMC in the figure. Please clarify the method labels and reported values, and enlarge the figure’s font for readability.
Reviewer_82shOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces Remember Smarter (RS), a plug-and-play memory module for VLA policies that combines a Mamba-based visual-history compressor, hyperbolic experience space construction, Poincare VAE, HypHC routing, and entailment cones for reusable experience. Integrated with pi0, RS improves robustness on LIBERO-Plus and shows gains on real-robot compositional tasks.

Critical strengths
  • The proposed method improves performance of pi0-LIBERO, especially on the robustness benchmark of LIBERO-Plus.
  • The experiments cover both simulation and real-world manipulation tasks.
Critical weaknesses
  • The motivation for each design choice is unclear (e.g., Mamba for visual-history compression, hyperbolic experience space construction, Poincare VAE, HypHC routing, entailment cones, etc). I view the contribution of the paper as a task-specific integration of existing techniques rather than a clear key innovation.

  • The main LIBERO-Plus experiment additionally adapts the model from a fine-tuned -LIBERO checkpoint, which makes the comparison with reported baselines difficult to interpret as a fair comparison. Also, the proposed method underperforms the reported MemoryVLA, MemoryVLA++ at Table1, but this is not discussed sufficiently.

  • The paper lacks controlled comparison with prior works targeting long-horizon or manipulation or robustness, to clarify the specific benefits of RS.

  • The details of how the experience-reuse mechanism was used in the main LIBERO-Plus experiment is insufficiently explained.

  • To support the plug-and-play claim, it would be important to demonstrate RS on additional VLA backbones beyond pi0, such as pi0-FAST or GR00T-family models.

Minor:

  • The boundary between the preliminaries and method sections is somewhat unclear, with substantial method-specific details already introduced in the preliminaries.
  • Some figures are difficult to interpret without additional context. For example, the full complex-task instruction corresponding to Figure 3 should be explicitly provided.
Questions for authors

See weaknesses also

  • What are the additional training computes for each stage of RS compared to the base model’s fine-tuning compute? Can the authors provide a compute-matched comparison for the main experiments?

  • Can the authors provide controlled comparison with prior works targeting long-horizon manipulation or robustness?

  • The main motivation of RS is long-horizon manipulation, while the primary benchmark in Table 1 evaluates robustness on LIBERO-Plus. Can the authors clarify why visual history memory and cross-episode experience memory are expected to improve robustness to these perturbations?

  • What is the contribution of each major design choice? e.g., using Mamba for visual-history compression instead of other architectures, the hyperbolic experience space design, and using the proposed hyperbolic experience space with P-VAE, HypHC-inspired routing, and entailment cones rather than simpler mechanisms?

  • Can the authors demonstrate the claimed plug-and-play nature of RS on at least one additional VLA backbone beyond pi0?

Reviewer_UqBeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Remember Smarter (RS) is a framework that adds visual-history memory to a VLA with a bidirectional spatial Mamba and causal temporal Mamba to encode the frames. It integrates this visual memory via residual attention with action-expert states used for action generation. Additionally, it adds cross-episodic memory by encoding the VLM hidden state at the final timeestep of a successful episode in a hyperbolic latent-space, and it retrieves successful episodes relevant to the current scene from this latent space. It generated prompt tokens by interpolated in latent space from the query towards the retrieved episode. They evaluate their method on LIBERO, LIBERO-Plus, and real-world experiments.

Critical strengths
  1. RS tackles two forms of memory: visual-memory and episodic memory. This are both important for long-horizon tasks in robotics.
  2. The recurrent memory stays fixed as the memory grow, which is useful for edge compute.
  3. They show improvement on long-horizon memory-based tasks over pi0.
Critical weaknesses
  1. The real-world and LIBERO Plus experiments lack a ablation on the retrieval mechanism.
  2. They do not compare to simpler retrieval methods. It would be useful to show comparisons with conventional Euclidean embeddings spaces such as SigLIP or VLM features using nearest neighbor retrieval. This could help establish the importance of the hyperbolic retrieval framework.
  3. The paper doesn't sufficiently explain how the policy learns to use the retrieved experience despite never being trained with them. Concretely, the experience tokens are masked during policy training and enabled during training.
Questions for authors
  1. Given that experience tokens are masked during training, how does the policy work with the retrieved experience during deployment?
  2. Can you provide visual-memory-only results on the real-world and LIBERO Plus experiments to isolate the contribution of the experience retrieval more effectively?
  3. Can you compare to simpler retrieval methods such as nearest-neighbor retrieval over VLM features in Euclidean space?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.