- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Memento trains an online fast-weight memory for a VLA by distilling history representations from a frozen VLM. Its read-before-write updates use momentum and forgetting. Layer-specific heads predict the teacher's centered final-token representations, which a connector maps to memory tokens. The policy is then fine-tuned with the memory's slow weights frozen, and deployment no longer needs the teacher. The experiments probe history recall, test control on RoboMME and RMBench, and check whether performance is preserved on LIBERO. RoboMME success rises from 19.33% to 41.04%. Latency is 44.36 ms per action chunk, compared with 40.41 ms for the base policy.
Critical strengths
The accuracy/latency tradeoff is what convinces me most. Memento roughly matches MemER's 40.88% success on RoboMME without the repeated VLM calls, and Counting improves from 48.67% to 68.50%. On RMBench, the larger improvement over Mem-0 is in tasks needing multiple observations: 41.25% against 25.00%. LIBERO performance is 97.45%.
Learning the memory content separately from teaching the VLA to use it is an interesting choice. The paper explains the read-before-write ordering clearly. I also liked the frozen-memory probes: StopCube count accuracy is 85%, versus 25% using the current observation, so there is evidence of retained information beyond the task-success numbers. Table 5 shows that the connector-only stage has a substantial effect.
Critical weaknesses
I lean weak reject at present because the comparison that should isolate distillation leaves too much uncertainty about optimization. Appendix A.4 does not give the Action-Supervised TTT truncation length or training budget, and says too little about its connector initialization and schedule. Memento itself drops from 68.50% to 26.84% on Counting when connector-only training is skipped. Given that sensitivity, difficulty learning to access memory could explain part of the action-supervised baseline's failure. It is hard to tell how much to attribute to the absence of VLM supervision. Distillation is the main distinction from existing fast-weight memory approaches, so this comparison needs tighter controls. The final-layer and training-stage ablations only cover Counting, which also leaves their explanation untested on the other suites.
I had a few concrete difficulties reconstructing the method from the appendix. I could not find the selected teacher layer indices. A.1 gives queries with D tokens, then a single 2560-dimensional readout, without saying how the tokenwise outputs are aggregated. A.5 mainly gives the RMBench specialist setup; the exact RoboMME training protocol remains unclear. For the probes, both dataset sizes and the unit of the train/test split are missing. Near-perfect state recall is hard to interpret without knowing whether complete trajectories were kept separate. The three evaluation seeds help assess rollout variability, but do not establish variation across independently trained memories. With these details missing, I cannot judge how reproducible the large differences in Table 5 are.
Questions for authors
-
Please specify the Action-Supervised TTT truncation length, connector warm-up and total training budget. Are the latter two matched to Memento? Can you compare the supervision sources while controlling these choices?
-
Which teacher layers do you distill, and how do the D tokenwise queries/readouts become a single memory vector? Please also give the corresponding RoboMME training settings.
-
How many trajectories are used for the probes, and are train/test sets disjoint by complete trajectory? Could you check the main distillation ablation on a non-Counting suite and report variation across training runs?
3 comments
- Understanding the performance gap between the two supervision sources.
To better understand the difference between these supervision signals, we also investigated why action-supervised TTT did not achieve the expected performance. Specifically, we analyzed the memory tokens produced during the training of the memory module and connector. We found that, with our distillation method, the average pairwise cos similarity of memory tokens across different interaction histories gradually decreases during training and eventually stabilizes around 0.4. In contrast, with action-supervised TTT, the similarity remains high, converging to approximately 0.9. One possible explanation is that, when trained solely through the action loss, the memory module may learn relatively input-insensitive representations that help minimize the downstream action loss, rather than distinguishing different interaction histories. In contrast, through the preceding VLM distillation stage, our memory module has already learned to encode semantic information from interaction histories into meaningful representations, enabling it to better capture and preserve histories.
These observations suggest that the performance gap reflects differences in the difficulty of learning meaningful memory representations under different supervision signals, rather than simply whether the VLA can access the memory module’s output. Furthermore, our distillation method offers additional advantages in terms of training efficiency and simplicity, such as avoiding the need for backpropagation through time during VLA fine tuning.
Question 2 & Weakness related to implementation details.
We thank the reviewer for pointing out these missing implementation details. We are happy to clarify the teacher layer selection and the aggregation of tokenwise memory readouts.
Teacher layer selection: Qwen3.5-4B consists of 32 decoder layers, with each group of four layers containing three linear attention layers followed by one full attention layer. We select the outputs of full attention layers (4,8,12,16,20,24 and 28, together with the final layer).
Tokenwise readout aggregation: At each timestep, the memory module generates a query for each of the D input tokens in each of the four heads. Each query retrieves a 640-dim vector from the corresponding fast weight memory. Within each head, we average the D tokenwise readouts to obtain a single 640-dim vector. We then concatenate the four head vectors to form a 2560-dim memory representation, which is passed through a readout MLP to get the eight layer-specific predictions. The resulting 8 2560 dim predictions are then passed through the connector to produce memory tokens in the VLA’s input space.
Training setting in RoboMME. RoboMME follows the same training pipeline, with several implementation details chosen to align with the benchmark’s own evaluation protocol. For RoboMME, the memory module takes SigLIP tokens from two camera views (front and wrist), together with the instruction embeddings. Demonstration video frames are written into memory before task execution (if the task has demonstration video) and memory is subsequently updated every 16 steps. This follows RoboMME’s evaluation protocol, in which the policy is queried every 16 environment steps and executes the first 16 actions of its predicted action chunk. We have updated Appendix A.5 of the revised manuscript to include more detailed implementation settings and training hyperparameters for each stage of the RoboMME pipeline.
Question 3 & Weakness related to probe experiments
We thank the reviewer for recognizing the value of our probing experiments and for highlighting the need for additional details. We are happy to provide further clarification on the probe datasets and train/test split.
For each task, we use 50 complete trajectories and conduct 5-fold cross-validation. In each fold, we train the probe on 40 trajectories and evaluate it on the remaining 10 trajectories, with no overlap in trajectories between the training and test sets. Each trajectory is held out for testing exactly once, and the reported results are averaged across all five folds. Importantly, each trajectory contributes only a single sample to the probing dataset, ensuring that the near-perfect state recall results cannot be attributed to leakage from correlated frames within the same trajectory.
Weakness related to ablation study.
We thank the reviewer for this valuable suggestion. We have indeed conducted the ablation experiments across all suites, although we only reported the Counting results in the original manuscript. We are happy to include the additional results in the revised manuscript. Overall, the results consistently show that multi-layer distillation contributes substantially to the final performance across different suites. We report the average performance and variation across two independent training runs, each evaluated three times.
| Suite | pi0.5 | Final Layer | w/o stage 2 | Memento |
|---|---|---|---|---|
| Counting | ||||
| Performance | ||||
| Reference | ||||
| Imitation |
We hope that our responses have addressed the reviewer's concerns and clarified the questions raised. In response to the reviewer's valuable feedback, we have updated the manuscript, with all revisions highlighted in blue for clarity. These updates include more detailed descriptions of our architecture and implementation, additional ablation results across different evaluation suites, and further details on our probing experiments. We sincerely appreciate the reviewer's constructive feedback and welcome any further questions or suggestions, which we would be happy to address.
We sincerely thank the reviewer for recognizing the strengths of our work and for raising thoughtful and constructive questions, which provide us with an opportunity to clarify points in the manuscript that were not sufficiently explained. We will address the questions and the corresponding weaknesses.
Question 1 & Weakness related to action-supervised TTT and connector training.
We thank the reviewer for raising this question and providing us with the opportunity to clarify the action-supervised TTT baseline.
To ensure a fair comparison, Action-Supervised TTT adopts the same memory module architecture and size as Memento. We also maintain comparable training budgets whenever possible. Specifically, both methods use 80K gradient steps for downstream VLA training. However, due to the inherent differences in their supervision signals, their memory training procedures are not entirely identical.
In Memento, the training procedure consists of three stages:
In contrast, action-supervised TTT follows a two-stage training:
For connector initialization, the projection weights are sampled from a normal distribution with a standard deviation of 0.002, while the scale and bias of the LayerNorm are both initialized to zero, ensuring the initial memory tokens are all zeros.
For action-supervised TTT, we use a TBPTT truncation length of 64 recurrent steps. Each recurrent step corresponds to one frame sampled every 16 action steps, covering approximately 1024 action steps. These 16 action steps correspond to the execution of a single action chunk generated by the VLA. In contrast, Memento does not require TBPTT during memory distillation, as the frozen VLM provides history-conditioned representations at each timestep as direct supervision targets. This avoids the need to backpropagate the downstream action loss through time when learning the memory module, which is one of the advantages of our distillation approach.
We would like to clarify that the connector serves different roles in Memento and action-supervised TTT. In Memento, the memory module is trained independently of the VLA to produce Qwen representations, which the VLA cannot directly interpret. Therefore, a separate connector training stage is necessary to bridge the representation gap.
In contrast, under action supervision, the memory module and connector are directly connected to the VLA during memory module training. Since gradients from the action loss are backpropagated through the VLA to optimize the memory module and connector, they learn to produce representations compatible with the VLA’s input space during memory training itself. Thus, action-supervised TTT does not require an additional connector-only alignment stage. Nevertheless, we retain the connector in action-supervised TTT to maintain the same model architecture and size as Memento.
Therefore, the performance degradation observed when removing connector-only training from Memento (68.50% to 26.84%) does not imply that action-supervised TTT suffers from missing the connector-only training stage and cannot access memory. The connector-only training stage in Memento addresses a representation mismatch specifically introduced by independently distilling Qwen representations, whereas action-supervised TTT learns the memory representations and their alignment with the VLA jointly.