acceptodds
Under review as a conference paper at ICLR 2027

Memento: Distilling Memory Capability from VLMs to VLAs

Abstract

Sequential multimodal decision making often requires retaining information that is no longer available in the current observation. This poses a fundamental challenge for Vision-Language-Action (VLA) policies, whose predictions are typically conditioned on limited context despite many tasks depending on information accumulated over interaction histories. Addressing such history dependence requires retaining relevant context over time, yet naively extending the context is computationally prohibitive, and hierarchical architectures that pair a high-level Vision-Language-Model (VLM) with a low-level VLA incur the cost of an additional large model throughout inference. In this work, we introduce Memento, a framework that distills multi-layer representations of interaction histories from a pretrained VLM into a lightweight online memory module based on Test-Time Training (TTT). Memento learns to encode task-relevant historical information into fast weights through online updates. The resulting memory readout is projected into memory tokens, providing the downstream VLA policy with compact access to accumulated context at bounded inference cost. In our experiments, we instantiate Memento by distilling multi-layer representations from Qwen3.5-4B and integrating the resulting memory module with . Memento improves the average success rate from 19.33% to 41.04% on RoboMME and from 10.56% to 51.67% on RMBench, while maintaining comparable performance on LIBERO, where long-term interaction history is less critical. These results demonstrate that memory capability can be distilled from a pretrained VLM, endowing VLA policies with the ability to predict actions conditioned on interaction histories.

Primary area:
foundation or frontier models, including LLMs
Keywords:
distillation, VLMs, VLAs, memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wTt2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Memento trains an online fast-weight memory for a VLA by distilling history representations from a frozen VLM. Its read-before-write updates use momentum and forgetting. Layer-specific heads predict the teacher's centered final-token representations, which a connector maps to memory tokens. The policy is then fine-tuned with the memory's slow weights frozen, and deployment no longer needs the teacher. The experiments probe history recall, test control on RoboMME and RMBench, and check whether performance is preserved on LIBERO. RoboMME success rises from 19.33% to 41.04%. Latency is 44.36 ms per action chunk, compared with 40.41 ms for the base policy.

Critical strengths

The accuracy/latency tradeoff is what convinces me most. Memento roughly matches MemER's 40.88% success on RoboMME without the repeated VLM calls, and Counting improves from 48.67% to 68.50%. On RMBench, the larger improvement over Mem-0 is in tasks needing multiple observations: 41.25% against 25.00%. LIBERO performance is 97.45%.

Learning the memory content separately from teaching the VLA to use it is an interesting choice. The paper explains the read-before-write ordering clearly. I also liked the frozen-memory probes: StopCube count accuracy is 85%, versus 25% using the current observation, so there is evidence of retained information beyond the task-success numbers. Table 5 shows that the connector-only stage has a substantial effect.

Critical weaknesses

I lean weak reject at present because the comparison that should isolate distillation leaves too much uncertainty about optimization. Appendix A.4 does not give the Action-Supervised TTT truncation length or training budget, and says too little about its connector initialization and schedule. Memento itself drops from 68.50% to 26.84% on Counting when connector-only training is skipped. Given that sensitivity, difficulty learning to access memory could explain part of the action-supervised baseline's failure. It is hard to tell how much to attribute to the absence of VLM supervision. Distillation is the main distinction from existing fast-weight memory approaches, so this comparison needs tighter controls. The final-layer and training-stage ablations only cover Counting, which also leaves their explanation untested on the other suites.

I had a few concrete difficulties reconstructing the method from the appendix. I could not find the selected teacher layer indices. A.1 gives queries with D tokens, then a single 2560-dimensional readout, without saying how the tokenwise outputs are aggregated. A.5 mainly gives the RMBench specialist setup; the exact RoboMME training protocol remains unclear. For the probes, both dataset sizes and the unit of the train/test split are missing. Near-perfect state recall is hard to interpret without knowing whether complete trajectories were kept separate. The three evaluation seeds help assess rollout variability, but do not establish variation across independently trained memories. With these details missing, I cannot judge how reproducible the large differences in Table 5 are.

Questions for authors
  1. Please specify the Action-Supervised TTT truncation length, connector warm-up and total training budget. Are the latter two matched to Memento? Can you compare the supervision sources while controlling these choices?

  2. Which teacher layers do you distill, and how do the D tokenwise queries/readouts become a single memory vector? Please also give the corresponding RoboMME training settings.

  3. How many trajectories are used for the probes, and are train/test sets disjoint by complete trajectory? Could you check the main distillation ablation on a non-Counting suite and report variation across training runs?

3 comments

We sincerely thank the reviewer for recognizing the strengths of our work and for raising thoughtful and constructive questions, which provide us with an opportunity to clarify points in the manuscript that were not sufficiently explained. We will address the questions and the corresponding weaknesses.

Question 1 & Weakness related to action-supervised TTT and connector training.

We thank the reviewer for raising this question and providing us with the opportunity to clarify the action-supervised TTT baseline.

  1. Training configurations and budget.

To ensure a fair comparison, Action-Supervised TTT adopts the same memory module architecture and size as Memento. We also maintain comparable training budgets whenever possible. Specifically, both methods use 80K gradient steps for downstream VLA training. However, due to the inherent differences in their supervision signals, their memory training procedures are not entirely identical.

In Memento, the training procedure consists of three stages:

  1. Memory distillation: We independently train the memory module to predict Qwen’s history-conditioned representations, using them as distillation targets. This stage uses 20K gradient steps.
  2. Connector training: We train the connector for 30K gradient steps, with the first 10% used for warm-up. This stage bridges the representation gap between the memory module and the VLA, since the memory module’s output lies in Qwen’s representation space, which is not directly aligned with the VLA’s input space.
  3. VLA training: we jointly train the VLA and connector for 80K (15K warm-up) gradient steps to enable the policy to effectively utilize the learned memory representations for action prediction.

In contrast, action-supervised TTT follows a two-stage training:

  1. Action-supervised memory training: We freeze the VLA and train the memory module and connector jointly using the action loss. Gradients are backpropagated through the frozen VLA to both components. This stage uses 50K gradient steps, with the first 10% used for warm-up. We allocated additional training steps compared with Memento’s connector training because both the memory module and connector must be optimized simultaneously.
  2. VLA training: We jointly train the VLA and memory components for 80K gradient steps (15K warm-up) to enable the policy to utilize the learned memory signal for downstream action prediction.

For connector initialization, the projection weights are sampled from a normal distribution with a standard deviation of 0.002, while the scale and bias of the LayerNorm are both initialized to zero, ensuring the initial memory tokens are all zeros.

For action-supervised TTT, we use a TBPTT truncation length of 64 recurrent steps. Each recurrent step corresponds to one frame sampled every 16 action steps, covering approximately 1024 action steps. These 16 action steps correspond to the execution of a single action chunk generated by the VLA. In contrast, Memento does not require TBPTT during memory distillation, as the frozen VLM provides history-conditioned representations at each timestep as direct supervision targets. This avoids the need to backpropagate the downstream action loss through time when learning the memory module, which is one of the advantages of our distillation approach.

  1. Connector role in Memento and action-supervised TTT, and learning to access memory

We would like to clarify that the connector serves different roles in Memento and action-supervised TTT. In Memento, the memory module is trained independently of the VLA to produce Qwen representations, which the VLA cannot directly interpret. Therefore, a separate connector training stage is necessary to bridge the representation gap.

In contrast, under action supervision, the memory module and connector are directly connected to the VLA during memory module training. Since gradients from the action loss are backpropagated through the VLA to optimize the memory module and connector, they learn to produce representations compatible with the VLA’s input space during memory training itself. Thus, action-supervised TTT does not require an additional connector-only alignment stage. Nevertheless, we retain the connector in action-supervised TTT to maintain the same model architecture and size as Memento.

Therefore, the performance degradation observed when removing connector-only training from Memento (68.50% to 26.84%) does not imply that action-supervised TTT suffers from missing the connector-only training stage and cannot access memory. The connector-only training stage in Memento addresses a representation mismatch specifically introduced by independently distilling Qwen representations, whereas action-supervised TTT learns the memory representations and their alignment with the VLA jointly.

  1. Understanding the performance gap between the two supervision sources.

To better understand the difference between these supervision signals, we also investigated why action-supervised TTT did not achieve the expected performance. Specifically, we analyzed the memory tokens produced during the training of the memory module and connector. We found that, with our distillation method, the average pairwise cos similarity of memory tokens across different interaction histories gradually decreases during training and eventually stabilizes around 0.4. In contrast, with action-supervised TTT, the similarity remains high, converging to approximately 0.9. One possible explanation is that, when trained solely through the action loss, the memory module may learn relatively input-insensitive representations that help minimize the downstream action loss, rather than distinguishing different interaction histories. In contrast, through the preceding VLM distillation stage, our memory module has already learned to encode semantic information from interaction histories into meaningful representations, enabling it to better capture and preserve histories.

These observations suggest that the performance gap reflects differences in the difficulty of learning meaningful memory representations under different supervision signals, rather than simply whether the VLA can access the memory module’s output. Furthermore, our distillation method offers additional advantages in terms of training efficiency and simplicity, such as avoiding the need for backpropagation through time during VLA fine tuning.

Question 2 & Weakness related to implementation details.

We thank the reviewer for pointing out these missing implementation details. We are happy to clarify the teacher layer selection and the aggregation of tokenwise memory readouts.

Teacher layer selection: Qwen3.5-4B consists of 32 decoder layers, with each group of four layers containing three linear attention layers followed by one full attention layer. We select the outputs of full attention layers (4,8,12,16,20,24 and 28, together with the final layer).

Tokenwise readout aggregation: At each timestep, the memory module generates a query for each of the D input tokens in each of the four heads. Each query retrieves a 640-dim vector from the corresponding fast weight memory. Within each head, we average the D tokenwise readouts to obtain a single 640-dim vector. We then concatenate the four head vectors to form a 2560-dim memory representation, which is passed through a readout MLP to get the eight layer-specific predictions. The resulting 8 ×\times 2560 dim predictions are then passed through the connector to produce memory tokens in the VLA’s input space.

Training setting in RoboMME. RoboMME follows the same training pipeline, with several implementation details chosen to align with the benchmark’s own evaluation protocol. For RoboMME, the memory module takes SigLIP tokens from two camera views (front and wrist), together with the instruction embeddings. Demonstration video frames are written into memory before task execution (if the task has demonstration video) and memory is subsequently updated every 16 steps. This follows RoboMME’s evaluation protocol, in which the policy is queried every 16 environment steps and executes the first 16 actions of its predicted action chunk. We have updated Appendix A.5 of the revised manuscript to include more detailed implementation settings and training hyperparameters for each stage of the RoboMME pipeline.

Question 3 & Weakness related to probe experiments

We thank the reviewer for recognizing the value of our probing experiments and for highlighting the need for additional details. We are happy to provide further clarification on the probe datasets and train/test split.

For each task, we use 50 complete trajectories and conduct 5-fold cross-validation. In each fold, we train the probe on 40 trajectories and evaluate it on the remaining 10 trajectories, with no overlap in trajectories between the training and test sets. Each trajectory is held out for testing exactly once, and the reported results are averaged across all five folds. Importantly, each trajectory contributes only a single sample to the probing dataset, ensuring that the near-perfect state recall results cannot be attributed to leakage from correlated frames within the same trajectory.

Weakness related to ablation study.

We thank the reviewer for this valuable suggestion. We have indeed conducted the ablation experiments across all suites, although we only reported the Counting results in the original manuscript. We are happy to include the additional results in the revised manuscript. Overall, the results consistently show that multi-layer distillation contributes substantially to the final performance across different suites. We report the average performance and variation across two independent training runs, each evaluated three times.

Suitepi0.5Final Layerw/o stage 2Memento
Counting27.6727.6727.084.0427.08_{4.04}24.671.2724.67_{1.27}68.251.48\mathbf{68.25}_{1.48}
Performance20.5020.5019.581.9419.58_{1.94}16.752.2616.75_{2.26}30.920.54\mathbf{30.92}_{0.54}
Reference17.5017.5021.330.5721.33_{0.57}20.330.6720.33_{0.67}31.250.78\mathbf{31.25}_{0.78}
Imitation11.6711.6712.500.9012.50_{0.90}11.082.0411.08_{2.04}33.920.54\mathbf{33.92}_{0.54}

We hope that our responses have addressed the reviewer's concerns and clarified the questions raised. In response to the reviewer's valuable feedback, we have updated the manuscript, with all revisions highlighted in blue for clarity. These updates include more detailed descriptions of our architecture and implementation, additional ablation results across different evaluation suites, and further details on our probing experiments. We sincerely appreciate the reviewer's constructive feedback and welcome any further questions or suggestions, which we would be happy to address.

Reviewer_pTsnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for equipping VLA models with long-term memory through representation distillation from pretrained VLMs. The method employs a TTT module to maintain interaction history in online fast weights, supervised by multi-layer hidden representations extracted from a frozen VLM. A lightweight connector subsequently maps the distilled representations into memory tokens for downstream action prediction. The framework follows a three-stage training procedure and removes the teacher VLM during inference. Experiments on RoboMME, RMBench, and LIBERO demonstrate substantial improvements on history-dependent manipulation tasks with relatively low inference overhead.

Critical strengths
  1. Long-term memory is an important yet underexplored capability in current VLA models. The proposed approach provides a reasonable direction for incorporating historical information without repeatedly invoking a large VLM during inference.

  2. Memento improves the average success rate from 19.33% to 41.04% on RoboMME and from 10.56% to 51.67% on RMBench. The improvements on counting and multi-step memory tasks are particularly encouraging.

  3. The framework achieves comparable overall performance to hierarchical VLM-based memory approaches while substantially reducing inference latency. The additional latency over the original VLA backbone is relatively small.

  4. The representation probing experiments provide evidence that the memory module preserves historical information. The ablations on multi-layer distillation and staged training also provide useful insights into the design.

Critical weaknesses
  1. The evaluation is restricted to simulation benchmarks, and the RMBench experiments use separate memory modules for different task groups. It is unclear whether the distilled memory generalizes to unseen tasks, longer interaction horizons, or noisy observations.

  2. In Table 1, Memento substantially outperforms Action-Supervised TTT (41.04% vs. 19.96%), which is used to demonstrate the benefit of VLM representation distillation. However, the comparison does not fully isolate the supervision objective: Memento trains its memory through BPTT, whereas Action-Supervised TTT uses TBPTT. Moreover, it is unclear whether the two methods follow comparable connector-training procedures and optimization budgets. A controlled comparison with matched backpropagation horizons and training settings is needed to attribute the improvement specifically to VLM distillation.

  3. The ablations supporting multi-layer distillation and the necessity of the separate connector-training stage are conducted only on four RoboMME Counting tasks. Given the substantial performance differences between the ablated variants, it would be important to verify whether these conclusions extend to other memory categories.

  4. The main components, including TTT-based fast-weight memory, representation distillation, and memory-token conditioning, are largely established techniques. The primary contribution is their integration into a VLA framework. In particular, the distinction from concurrent TTT-based VLA memory methods such as RobotTTT appears to lie mainly in the supervision objective. A deeper explanation of why multi-layer VLM representations are particularly suitable for memory learning would strengthen the technical contribution.

Questions for authors
  1. Can the authors provide a more controlled comparison with Action-Supervised TTT, using matched optimization settings, backpropagation horizons, and training budgets? How much of the improvement can be attributed specifically to VLM representation distillation?

  2. How essential is the semantic knowledge of the pretrained VLM? For example, how would Memento perform with alternative representation targets, or with a history encoder trained without VLM distillation? Such comparisons could clarify whether the method truly transfers memory capability from the VLMs.

  3. How robust is the online memory to longer execution horizons? Does performance degrade when the deployment trajectories deviate substantially from those used for memory distillation?

5 comments

We sincerely thank the reviewer for their constructive feedback and valuable suggestions on our work, which have provided us with an opportunity to clarify points that were not sufficiently explained in the original manuscript and to provide additional details and results. We will first address the questions and their corresponding weakness, followed by responses to the remaining weaknesses individually.

Question 1 & Weakness 2. We sincerely thank the reviewer for raising this important question and providing us with the opportunity to clarify the training setup of our Action-Supervised TTT baseline and its comparison with Memento.

1. Training configurations and budget.

To ensure a fair comparison, Action-Supervised TTT adopts the same memory module architecture and size as Memento. We also maintain comparable training budgets whenever possible. Specifically, both methods use 80K gradient steps for downstream VLA training. However, due to the inherent differences in their supervision sources, their memory training procedures are not entirely identical.

In Memento, the training procedure consists of three stages:

  1. Memory distillation: We independently train the memory module to predict Qwen’s history-conditioned representations, using them as distillation targets. This stage uses 20K gradient steps.
  2. Connector training: We train the connector for 30K gradient steps, with the first 10% used for warm-up. This stage bridges the representation gap between the memory module and the VLA, since the memory module’s output lies in Qwen’s representation space, which is not directly aligned with the VLA’s input space.
  3. VLA training: we jointly train the VLA and connector for 80K (15K warm-up) gradient steps to enable the policy to effectively utilize the learned memory representations for action prediction.

In contrast, action-supervised TTT follows a two-stage training:

  1. Action-supervised memory training: We freeze the VLA and train the memory module and connector together using the action loss. Gradients are backpropagated through the frozen VLA to both components. This stage uses 50K gradient steps, with the first 10% used for warm-up. We allocated additional training steps compared with Memento’s connector training because both the memory module and connector must be optimized simultaneously.
  2. VLA training: We jointly train the VLA and memory components for 80K gradient steps (15K warm-up) to enable the policy to utilize the learned memory signal for downstream action prediction.

TBPTT is a commonly adopted training strategy for TTT models, particularly when learning from long interaction sequences, as also demonstrated by RoboTTT. In contrast, Memento does not require TBPTT during memory distillation, as the frozen VLM provides history-conditioned representations as direct supervision targets. Although memory distillation still involves backpropagation through the memory module, it avoids the need to propagate the downstream action loss through time when training the memory module. This simplifies the training process and represents an additional advantage of our distillation method.

2. Connector training in Memento and action-supervised TTT

We would like to clarify that the connector serves different roles in Memento and action-supervised TTT. In Memento, the memory module is trained independently of the VLA to produce Qwen representations, which the VLA cannot directly interpret. Therefore, a separate connector training stage is necessary to bridge the representation gap.

In contrast, under action supervision, the memory module and connector are directly connected to the VLA during memory module training. Since gradients from the action loss are backpropagated through the VLA to optimize the memory module and connector, they learn to produce representations compatible with the VLA’s input space during memory training itself. Thus, action-supervised TTT does not require an additional connector-only alignment stage. Nevertheless, we retain the connector in action-supervised TTT to maintain the same model architecture and size as Memento.

Therefore, the performance degradation observed when removing connector-only training from Memento does not imply that action-supervised TTT suffers from missing the connector-only training stage. The connector-only training stage in Memento addresses a representation mismatch specifically introduced by independently distilling Qwen representations, whereas action-supervised TTT learns the memory representations and their alignment with the VLA jointly.

3. Understanding the performance gap between the two supervision sources.

To better understand the difference between these supervision signals, we also investigated why action-supervised TTT did not achieve the expected performance. Specifically, we analyzed the memory tokens produced during the training of the memory module and connector. We found that, with our distillation method, the average pairwise cos similarity of memory tokens across different interaction histories gradually decreases during training and eventually stabilizes at 0.4. In contrast, with action-supervised TTT, the similarity remains high, converging to approximately 0.9. One possible explanation is that, when trained solely through the action loss, the memory module may learn relatively input-insensitive representations that help minimize the downstream action loss, rather than distinguishing different interaction histories. In contrast, through the preceding VLM distillation stage, our memory module has already learned to encode semantic information from interaction histories into meaningful representations, enabling it to better capture and preserve histories.

These observations suggest that the performance gap reflects differences in the difficulty of learning meaningful memory representations under different supervision signals, rather than whether the VLA can access the memory module’s output. Furthermore, our distillation method offers additional advantages in terms of training efficiency and simplicity, such as avoiding the need for backpropagation through time during VLA fine tuning.

Question 2 & Weakness 4. We thank the reviewer for these questions and for providing us with the opportunity to further clarify the motivation and technical contributions of our work.

Novelty and distinction from existing TTT-based VLA methods. While TTT-based memory and representation distillation have been studied independently, we believe the key contribution of Memento lies in introducing a new perspective on learning memory for VLA policies.

This distinction is particularly important compared with concurrent methods, which train TTT-based memory mechanisms through sequence-level action supervision. In contrast, Memento explicitly separates training memory from downstream VLA policies. By distilling history-conditioned representations from a pretrained VLM, our approach provides direct supervision for what information the memory module should preserve. We believe this capability-distillation formulation, rather than simply the integration of existing components, constitutes the central methodological contribution of our work.

How essential is the semantic knowledge of the VLM, and use alternative representation targets. We agree that alternative representation targets or history encoders could potentially serve as supervision sources for learning an online memory module. Our proposed framework is not inherently restricted to representations from pretrained VLMs. Other teacher models could also be incorporated if they can offer informative representations of task-relevant interaction histories.

We choose pretrained VLMs as teachers because they offer two important advantages. First, pretrained VLMs can process long multimodal interaction histories and produce representations that capture task-relevant semantic information. Second, their representations can be conditioned through task-oriented prompts, allowing us to guide the teacher toward historical information relevant to downstream VLA decision-making. Our probing experiments provide supporting evidence for this property: removing the task-oriented prompt reduces memory probing accuracy on three of the four evaluated tasks.

Importantly, our goal is not to establish that VLM representations are the only suitable supervision targets, but rather to investigate whether the memory capability of pretrained VLMs can be transferred into a lightweight online memory module through representation distillation. Our experimental results provide evidence supporting the effectiveness of this approach. Exploring alternative teacher models and representation targets would be a natural extension of our framework, but a systematic comparison of possible supervision sources is beyond the scope of the present work.

why multi-layer VLM representations are particularly suitable for memory learning.

  1. When training a memory module solely through downstream action supervision, the model must learn what historical information to preserve from the action prediction loss. This requires propagating the action loss through a sequence of memory updates, which can be computationally expensive for long interaction histories and uses of truncated backpropagation through time (TBPTT). In contrast, use representation distillation allows the memory module to learn representations of the accumulated history without relying on the downstream action loss. Although memory distillation still involves BPTT through the fast-weight updates, it avoids the need to backpropagate the downstream VLA action loss through these updates, simplifying the memory learning process.
  2. Several existing approaches leverage pretrained VLMs to provide historical information or high-level reasoning for VLA policies. However, these approaches typically require invoking the VLM during deployment, introducing substantial computational overhead and inference latency. Our approach instead transfers history-conditioned representations from the pretrained VLM into a lightweight online memory module during training. The teacher VLM is then removed entirely at inference time. Our efficiency experiments demonstrate that Memento achieves substantial improvements on history-dependent tasks while maintaining inference latency close to that of the original VLA backbone.
  3. Different VLM layers may encode complementary information about the interaction history. By distilling representations from multiple layers, we provide the memory module with supervision at different levels of abstraction, rather than relying on a single final-layer target. Our ablation results support this design choice.
Reviewer_LFbnOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Memento, a method for distilling history-dependent representations from a pretrained Vision-Language Model (VLM) into a compact online memory module for Vision-Language-Action (VLA) policies. The proposed method uses Test-Time Training (TTT) to maintain interaction history through online fast-weight updates. During training, the memory module is supervised to reconstruct multi-layer hidden representations of a frozen Qwen3.5-4B teacher conditioned on the interaction history. The resulting memory representations are converted into memory tokens through a lightweight connector and integrated into a pi0.5 policy using a three-stage training procedure. At inference time, the teacher VLM is removed, and the memory is updated online. Experiments on RoboMME and RMBench demonstrate improvements over the memory-free VLA baseline (19.33% to 41.04% and 10.56% to 51.67% success rate, respectively), while maintaining comparable performance on LIBERO. The authors also report low inference overhead and conduct ablations on memory probing, multi-layer distillation, and staged training.

Critical strengths
  1. An interesting idea to distill history-dependent representations from a pretrained VLM into a lightweight memory module, although it is not entirely clear how well the teacher VLM captures long-term history.
  2. Using Test-Time Training to maintain a compact, continuously updated memory is an interesting approach, especially in combination with VLM distillation.
Critical weaknesses
  1. Missing comparisons with existing RoboMME baselines. Unfortunately, the paper does not include several strong baselines from the original RoboMME paper, even though their results are already publicly available and would not require additional experiments to report. Some of these methods outperform Memento. For example, FrameSamp+Modul achieves 44.51% overall SR compared to 41.04% for Memento, and 51.39% vs. 33.83% on Imitation tasks. Although the memory budgets differ, these results should at least be reported and discussed. Their omission makes the evaluation look incomplete and potentially misleading.

  2. Insufficient implementation details. Several important parts of the method are poorly explained, including the connector architecture, BPTT length, selected VLM layers, and construction of the teacher's visual context. These details are essential for understanding and reproducing the proposed approach.

  3. Limited analysis of memory and TTT. For a paper focused on learning memory through TTT, there is little analysis of memory retention, fast-weight update dynamics, or performance across different memory horizons. The paper also lacks a proper failure analysis, making it difficult to understand why the method works on some memory tasks but performs poorly on others.

  4. Insufficient evidence that the VLA actually uses the memory. Linear probing shows that historical information can be decoded from memory representations, but this does not necessarily mean the VLA uses that information for action prediction. The paper lacks controlled experiments with oracle memory or counterfactual memory interventions. Moreover, the large improvement attributed to staged connector training may be confounded by the additional optimization steps.

Questions for authors
  1. Memory capability of the teacher VLM. You state that the method distills "the memory capability acquired through large-scale pretraining". I do not fully understand what memory capability you are referring to. A pretrained VLM does not have persistent memory, it can only reason over information available in its context window. Could you clarify this claim? What is the context length of Qwen3.5-4B, and how many images can it process?

  2. Layer-specific representations. Why do you introduce layer-specific heads hlh_l to project the shared memory readout rtr_t into the representation space of each VLM layer? Why not directly predict layer-specific memory readouts?

  3. BPTT length. What BPTT unroll length do you use during memory distillation? How does it compare to the TBPTT length used in the Action-Supervised TTT baseline?

  4. Writing quality. There are many repetitions throughout the paper. For example, lines 260-262 explain that the connector maps memory representations into a fixed number of VLA memory tokens, and lines 263-267 immediately repeat essentially the same explanation. I suggest revising the text to remove such redundancies (not only for this particular example, but for all such cases).

  5. Memory token placement. Why do you prepend memory tokens to the current observation tokens rather than append them? Is there any empirical or architectural motivation for this choice? I think this should disrupt the initial inductive bias of π0.5\pi_{0.5}, in which the first tokens of the context are reserved for current observations.

  6. Connector architecture. How exactly does the connector map the layer-specific representations zt(l)z_t^(l) into memory tokens mtm_t? This is an important component of the proposed method, but its architecture is not sufficiently explained.

  7. Figure 2 and oracle memory baseline. What exactly is the main purpose of Figure 2? Why are "with task prompt" and "prompt removed" included in the memory probing experiment rather than presented separately as an ablation? More importantly, why not include an oracle memory baseline, where an observation containing the task-relevant cue is manually placed into the VLA context? This would provide a more informative comparison.

  8. Current-observation-only probing. In Figure 2, why does the "current obs only" baseline achieve accuracy substantially above chance? If the target information is supposed to be unavailable in the current observation, what explains this result? Could there be correlations or shortcuts in the benchmark?

  9. Experimental narrative. Why does Section 4 start with memory probing rather than the main results on memory-intensive and non-memory-intensive benchmarks? I believe presenting the downstream performance first and using probing to explain the results would make the experimental narrative clearer.

  10. Missing RoboMME baselines. Why do you compare against only one existing external method (MemER), while the other two baselines (VLM-Subtask and Action-Supervised TTT) are your own implementations? The original RoboMME paper already provides strong memory-augmented baselines, such as FrameSamp+Modul and TokenDrop+Modul, which should be included in Table 1. For example, I expected TTT-based memory distillation to be particularly effective on the Imitation category, but Memento achieves only 33.83%, compared with 51.39% for FrameSamp+Modul and 38.28% for TokenDrop+Modul. Memento also underperforms FrameSamp+Modul on Reference and in overall average success rate. These comparisons are important for understanding the actual advantages of the method.

  11. Failure analysis. Could you provide a more detailed failure analysis to explain where and why Memento stops working? For example, memory-intensive benchmarks such as LIBERO-Mem (Chung et al., 2026) and MIKASA-Robo-VLA (Cherepanov et al., 2026) include tasks with different memory horizons and could help evaluate how performance changes with increasing memory requirements.

  12. Efficiency comparison. Why does Table 4 not include the existing lightweight memory baselines from the RoboMME paper? Comparing TTT-based memory against methods that repeatedly invoke a large VLM is useful, but I would also like to see how it compares with other efficient memory mechanisms.

  13. Table ordering. Tables 2, 3, and 4 appear in the wrong order relative to their discussion in the text. Please fix their placement.

  14. Missing RMBench baselines. Why does Table 2 omit VLM-Subtask and Action-Supervised TTT? These baselines are evaluated on RoboMME, and including them on RMBench would provide a more complete assessment of the contribution of VLM distillation.

  15. Teacher context and sequence lengths. What are the actual sequence lengths processed by the teacher VLM, both in terms of video duration and number of input tokens? How many frames are provided to Qwen at each timestep? Similarly, how many timesteps are included in the BPTT unroll for the student memory module?

  16. TTT training dynamics. For a paper focused on TTT, there is very little analysis of the TTT mechanism itself. What are the magnitudes of the gradients used to update the fast weights? How do these gradients and the fast weights evolve during an episode? I would like to see training dynamics, gradient statistics, and an analysis of whether memory updates remain stable over long trajectories.

  17. Video frame sampling. How is the frame sampling frequency selected when constructing the video input for the teacher VLM? What happens if an important event is visible for fewer timesteps than the sampling interval? In that case, the teacher may never observe the event, so the corresponding information cannot be distilled into the student memory. This seems particularly important for tasks involving counting or short-duration events.

5 comments

We would like to sincerely thank the reviewer for their detailed questions. We will try our best to address these concerns. As the reviewer raised a number of questions regarding our work, we will first address the weaknesses and their corresponding questions, followed by responses to the remaining questions individually.

Weakness 1, Question 10, and Question 14. Benchmark and Baseline. RoboMME indeed explores a wide range of different methods. However, we only include baselines that are closely related to our approach. For example, we report results for several VLM-VLA methods because our approach specifically aims to remove the VLM component from VLM-VLA frameworks. We also compare against action-supervised TTT methods that are closely related to our work. Our intention is to present comparisons with methods that are most relevant to our approach, rather than to provide an evaluation of all existing methods.

Furthermore, we cannot agree that it is appropriate to directly present results from other researchers’ papers without explicitly acknowledging their sources. We would like to clarify that evaluating all methods requires additional experiments. Rather than directly reporting results from existing papers, we reimplement the baselines ourselves to ensure a fair comparison. Therefore, we focus our experiments on methods that are most relevant to our approach. For instance, although FrameSample and TokenDrop are memory-augmented VLA methods, they are less directly related to our proposed approach, and thus we do not include them in our comparisons.

Some methods were reimplemented by us based on the original paper because some of these papers did not release the code, or because they used different VLA models with different architectures. In these cases, we made our best effort to ensure fair comparisons. Regarding the comparisons on RMBench, we would like to clarify that some methods could not be directly applied due to limitations in the dataset. For example, Memer requires annotations of important frames and subtasks, whereas RMBench primarily provides annotations of subtask boundaries. Therefore, we did not evaluate all baseline methods on RMBench, as their implementation would require additional annotations that are not available in the benchmark.

Weakness 2, Question 3, Question 6, Question 15. Implementation Details. We have added further implementation details in the appendix and highlighted in blue, which we hope will help address the reviewer's concerns regarding implementation details.

The connector is an MLP with LayerNorm; the projection weights are sampled from a normal distribution with a standard deviation of 0.002, while the scale and bias of the LayerNorm are both initialized to zero, ensuring the initial memory tokens are all zeros. The connector gets layer-specific representations (8 ×\times 2560) and projects the representation of each layer into 8 memory tokens (2560 -> 8 ×\times 2048). So we have 64 total memory tokens.

The BPTT unrolling length is determined by the length of each trajectory rather than a predefined window. Specifically, on RoboMME, we use a temporal stride of 16 frames, meaning that the memory is read and updated once every 16 action steps. The resulting BPTT length therefore varies across trajectories. Unlike TBPTT, which truncates gradient propagation to a fixed-length window. We use a TBPTT truncation length of 64 recurrent steps. Each recurrent step corresponds to one frame sampled every 16 action steps, covering approximately 1024 action steps. These 16 action steps correspond to the execution of a single action chunk generated by the VLA. In Memento, TBPTT is unnecessary since the frozen VLM provides history-conditioned representations at each timestep as direct supervision targets. This avoids the need to backpropagate the downstream action loss through time when learning the memory module, which is one of the advantages of our distillation approach.

Both the teacher context length and the input sequence length are determined by the trajectory data rather than manually specified. In our experiments, the longest video trajectory exceeds two minutes. At each timestep, the input to Qwen contains only the interaction history up to that timestep, without access to any future observations. Therefore, the input sequence length varies across timesteps and grows as more historical observations become available.

Weakness 3, Question 11, Question 16. Memory module dynamics and failure analysis. For failure case analysis, we have already discussed it in our paper. Our method is particularly effective for tasks that require continuously updating and utilizing memory throughout an episode. However, it shows relatively limited performance on tasks where success primarily depends on retaining important visual anchors observed early in the episode. This suggests that our approach is better suited to tasks requiring continuous accumulation and integration of historical information. We further analyze the relationship between horizon and performance across 16 tasks in RoboMME. We find that the horizon has almost no correlation with success rate (spearman correlation: 0.05).

Regarding the additional benchmarks suggested by the reviewer, we acknowledge that there are excellent works closely related to our method. However, LIBERO-Mem primarily focuses on evaluating different aspects of object-level memory and reasoning, rather than systematically analyzing memory retention horizons. Additionally, MIKASA-Robo-VLA was made publicly available after the submission deadline. We will discuss this work in our paper.

We further analyze the dynamics of our memory module by replaying tasks in RoboMME. We find that the memory module fast-weight updates with learned step size around 0.015 (gradient magnitude around 1-2.7). Each memory write changes the fast-weights by approximately 6% of their norm at the median, indicating that the memory state is continuously updated throughout the episode. To further examine stability, we replay the episodes with horizons longer than the training set, which have about 270 memory writes, approximately three times the longest training horizon. The fast-weight norms remain stable throughout the extended sequence. These results suggest that the memory updates remain stable even over longer sequences.

Weakness 4, Whether VLA uses memory. Connector training is an important stage in our training pipeline. However, we do not think that the performance improvements primarily stem from the additional training steps. The connector training stage plays a crucial role in aligning the distilled memory representations with the VLA’s representation space, enabling the downstream policy to effectively utilize the historical information encoded in the memory tokens. In addition, we allocated more optimization steps to train the memory module of the action-supervised TTT baseline, and it still performed worse than the distilled memory module. This further suggests that the performance gain arises not simply from additional optimization steps.

Regarding the Oracle baseline, directly providing the entire interaction history to the VLA is not feasible. For episodic memory tasks, the oracle would need to provide all historical observations preceding the current action, resulting in an input sequence that far exceeds the VLA’s practical context length and computational budget. While providing a single visual anchor may be feasible for tasks that depend on one specific historical frame, such an approach is not suitable for tasks requiring the continuous accumulation and integration of historical information. Moreover, incorporating additional historical observations into the VLA directly would require additional training to enable the policy to effectively process and utilize them. Therefore, we do not believe that such an oracle baseline would reveal whether our method effectively leverages historical information for action prediction. Regarding counterfactual memory interventions, our understanding is that replacing the fast-weight state of the memory module during VLA execution. We have conducted preliminary experiments following this approach. However, we observed that replacing the fast-weight disrupts the VLA’s ability to continue executing the task effectively. The reason may be that the substituted memory state is inconsistent with the current execution state.

We believe that our existing experiments provide evidence that the VLA effectively utilizes the historical information encoded by the memory module. For example, in our ablation study, replacing the memory distillation objective while keeping the overall training pipeline unchanged leads to a substantial performance drop. Since the training procedure remains consistent, we believe this result suggests that the performance gains are due to the task-relevant historical information learned through VLM representation distillation and subsequently utilized by the VLA for action prediction.

Question 1, VLM memory capability. Qwen3.5-4B supports a context length of over 200k tokens, allowing it to process several thousand image frames at the image resolution used in our experiments. This capacity is more than sufficient for our memory distillation. Regarding whether VLMs have memory capabilities, we consider a model to have memory capabilities if it can leverage information from past observations or events that are no longer available in the current observation to inform its present decision. Therefore, we believe that VLMs do have memory capabilities, as they can leverage historical context to retrieve and reason about past information. Furthermore, since VLMs can attend to historical information through the KV cache, their current hidden representations are influenced by preceding observations. These history-conditioned representations are precisely what we use as supervision targets in our distillation process, allowing the memory module to learn to preserve and utilize historical information.

Question 2, why not layer-specific readout. The primary motivation is to control the computational cost of the memory module, Since the fast-weight state is maintained and updated throughout the episode, directly predicting layer-specific readouts using separate memory states would significantly increase both memory size and per-step computational overhead. Instead, we use a shared memory readout followed by lightweight layer-specific projection heads. This design allows the memory module to maintain a compact fast-weight state while predicting representations corresponding to different VLM layers.

Question 4, writing quality. We will continue to improve the writing of our manuscript to ensure that our ideas are presented as concisely, accurately, and clearly as possible.

Question 5, memory token placement. Inserting memory tokens at any position would inevitably disrupt the original input structure of pi0.5, as the model was not originally designed with a dedicated position for memory tokens. We choose to prepend memory tokens because prior studies have suggested that tokens placed at the beginning of the input sequence can have greater influence on the transformer. Moreover, several existing works adopt a similar strategy of prepending additional tokens. Therefore, we follow this established design choice in our implementation.

Question 7, figure 2 and oracle baseline. The primary purpose of figure 2 is to demonstrate that the distilled memory module preserves historical information that is no longer available in the current observation. The comparison between representations distilled with and without the task prompt is indeed an ablation study, which we have separately discussed in the ablation section. We present both settings in figure 2 simply to consolidate all representation probing results into a single figure and avoid introducing multiple figures with similar experimental setups and redundant visualization.

For oracle baseline, as discussed in our response to Weakness 4, directly providing the entire interaction history to the VLA is not feasible. Furthermore, we would like to clarify that the probing experiments in figure 2 are designed to evaluate the representations of the memory module itself, rather than those of the downstream VLA. Therefore, directly providing historical observations to the VLA would not serve as a relevant baseline for this representation probing experiment.

Question 8, current observation only baseline. In our probing experiments, we extract representations from intermediate frames of each task. Intuitively, these frames should not contain information about earlier events or observations. However, we cannot guarantee that every selected frame is perfectly neutral with respect to the task history. For example, after the robot places a cube and returns toward its initial position, the selected frame may still capture the robot arm slightly shifted toward the direction from which it approached the cube. Such subtle visual cues may reveal some historical information, potentially explaining why the current observation only probe performs slightly above chance. Additionally, the duration for which the memory module is involved during an ongoing episode could also serve as another signal. This is because certain parameters, such as the forgetting factor, may provide additional useful signals.

Nevertheless, this does not affect our main conclusion, as the probe using the representations with memory substantially outperforms the current-observation-only probe. This large performance gap provides evidence that our memory module preserves historical information that cannot be reliably inferred from the current observation alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.