- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper adds a fast-weight TTT memory to π0.5. Learned query tokens cross-attend to VLM features to form a summary U_t. U_t is projected to K/V for a self-supervised associative write and to Q for a read. The gated readout is given to the action expert as extra keys and values. Training alternates between two phases: memory slow parameters are trained with the policy frozen, then the policy is trained with memory frozen. A heuristic scale β_t modulates write strength. On the 16 RoboMME tasks, average success rises from 17.9% (π0.5) to 56.8%. RoboMME's own TTT/RMT memory variants reach about 18–22%. Intervention experiments (disabling writes, swapping memory between episodes, restricting the write window) show that actions depend on what is stored in memory.
Critical strengths
- The empirical gain is large. On the same backbone and with the same memory family (fast-weight TTT), the method gains about 35 points over the recurrent-memory baselines. It does this with a constant-size state and no external VLM calls.
- Sec. 4.4 is the strongest part of the paper. Using conflicting memory drops VideoUnmask from 300/300 to 1/300. Empty memory falls to chance. Pre- vs. post-occlusion writes show where the useful information is stored. Few memory-VLA papers test whether the memory content is actually used.
Critical weaknesses
- The main gain is not attributed. All baseline numbers in Table 1 are copied from the RoboMME paper. Training steps, compute (32 H200 × 48 h here) and the multi-task setup are therefore not controlled. RoboMME's TTT variants share the backbone and memory type and score about 22%. The paper never shows, on the benchmark, how the 35-point gap splits between the interface, alternation, β_t and the training budget. The needed experiment: rerun RoboMME's TTT-{Context, Modul, Expert} under the authors' recipe and budget, then add components one at a time on all 16 tasks.
- The ablations are too thin for the claims they support. They use two tasks and single-task checkpoints, with no variance reported.
- The RoboTTT-style variant (30/26) is at the no-memory level (23/28). RoboTTT reports that memory capability appears during its frozen-backbone memory training stage. If the reimplementation skips that stage, this baseline is a straw man.
- "Non-alternating" means joint training from the start, and it also lands at the no-memory level. The baseline that would justify repeated alternation is a single round: train memory with the policy frozen, then train jointly. The number of alternation rounds is also not reported.
- Adaptive writing is evaluated only by the NMSE of early K/V bindings. Damping later writes reduces interference with early bindings almost by construction. There is no task-success result, and no comparison with existing forgetting or surprise gating (Titans, gated delta rules). As presented, β_t is an unvalidated heuristic.
- The swap tasks expose the limit of the memory claim. VideoUnmaskSwap and ButtonUnmaskSwap are the weakest tasks, at about 20%. These tasks require updating a stored binding, which is exactly where an adaptive associative write should help. L362–367 blame π0.5's temporal reasoning, but this is not tested. The evidence fits a memory that stores a cue seen once and retrieves it later, i.e., mainly the Permanence/Reference regime.
- Evaluation scope is narrow. There is one simulated benchmark, no real-robot experiment and no second memory benchmark (e.g., MemoryBench). The efficiency comparison only covers FrameSamp and MemER. The relevant comparison is against constant-cost recurrent memory (TTT, RMT). Being faster than explicit-history methods is expected.
Questions for authors
- What compute and how many steps did the RoboMME baselines use, compared with yours? Can you rerun the TTT baselines under your training recipe?
- Does the RoboTTT-style variant include RoboTTT's frozen-backbone memory training stage? Was it trained with the same alternating schedule?
- What is task success with β_t = 1 on the full benchmark?
- How many alternation rounds are used, and how sensitive are results to this number? How does one round compare with repeated alternation?
- During training, how are gradients propagated through the fast-weight writes (full BPTT or truncated), and over what sequence length?
9 comments
Response to Reviewer LW6g (Part 2/5)
Weakness 1 + Question 1 (continued)
Why the RoboMME TTT baselines are not explained by insufficient training. RoboMME does not report training curves, so we cannot directly verify their convergence, but three lines of evidence indicate that the issue is not training length:
- The RoboMME authors attribute the failure of recurrent memory to unstable training caused by fine-tuning π0.5 with shallow recurrent layers, and state that effective recurrent memory requires deeper architectural integration (Sec. 5.2). T²Mem's observation interface integrates with the backbone across 17 layers and stabilizes optimization through alternating training, directly targeting this issue.
- The TTT variants perform close to using no memory: their 16-task average is only about 4 percentage points above the memory-free π0.5; on VideoUnmask they score 27.22–31.78%, close to chance in a three-way choice; on StopCube they score 2.11–4.00%, even below the memory-free π0.5 (6.67%). This reflects not learning to use memory, rather than insufficient learning.
- Under our recipe, TTT memory without alternating training, when trained 4× longer (2,000 → 8,000 steps) on the pure memory-retrieval task VideoUnmask, only changes from 30% to 32.67%, remaining at chance level throughout.
How the 35-point gap is distributed (Weakness 1). Based on the results above, we analyze the sources of the gap as follows:
- Training recipe and compute: The evidence indicates a small contribution. On the one hand, the memory-free Stage1 under the same recipe uses only 1/4 of the training steps of the public π0.5 (20,000 vs. 80,000) and achieves 16.38%, comparable to its 17.93%, showing that the recipe itself does not raise policy capability. On the other hand, the TTT controls trained with the same recipe and compute in the table above all fall back to the level of RoboMME TTT, showing that compute alone cannot produce the gap.
- Observation interface and alternating training: These are the main source of the gap. Removing either one brings success back to the level of RoboMME TTT. Both the single switch (72.00%) and memory pretraining followed by joint training (75.33%) are below repeated alternation (91.33%), showing that the benefit of alternation comes from repeated alternation itself.
- Adaptive writing: On top of this, it brings a further gain of 20–23 percentage points.
In addition, FrameSamp-Modul, the strongest non-privileged baseline in RoboMME, is also trained for 80,000 steps (44.51%), and T²Mem still outperforms it by 12.32 percentage points. We agree that the full experiment suggested by the reviewer would be the most convincing, and we carefully assessed its feasibility: rerunning the three TTT variants and adding the three components one by one would require at least six full 16-task training runs, each taking about 1,536 H200 GPU hours, which exceeds the computational resources available to us during the rebuttal period. We therefore removed components one at a time under the same recipe and used single-task controlled comparisons to answer the same question. We acknowledge that this remains a limitation of our work, and the revision will clearly distinguish external reference baselines from controlled comparisons.
Weakness 2 (opening and first point) + Question 2
We thank the reviewer for the comments.
Scale and variance. We agree that the original ablations were small in scale. The new experiments therefore cover four tasks (VideoUnmask, MoveCube, StopCube, and PatternLock), and the standard deviation across evaluation seeds is at most 6.26 percentage points for every task. In addition, single-task and 16-task performance are similar (VideoUnmask 91.33% vs. 88.00%, MoveCube 77.33% vs. 80.67%), indicating that the single-task ablations reflect multi-task behavior.
RoboTTT-style variant (Question 2). Yes. This variant includes the frozen-backbone memory training phase and uses the same alternating schedule (Appendix D): memory and policy alternate every 500 steps, the policy is frozen during memory phases, and the only difference from our method is the source of writes and queries. To further rule out insufficient training, we trained both interfaces to 8,000 steps (4× the budget). On StopCube, which requires tracking how many times an event has occurred, the AE-side variant reaches 11.33% and the observation-grounded interface reaches 60.00%, so the gap remains large. We note that this control is intended to compare interface designs under controlled conditions; the full RoboTTT method additionally includes re-pretraining on the full VLA data.
Response to Reviewer LW6g (Part 3/5)
Weakness 2 (second point) + Question 4
We thank the reviewer for suggesting this more informative baseline.
Single-round baseline. We added this baseline on VideoUnmask (8,000 steps, 3 seeds × 50 episodes):
| Training schedule | Memory updates | Policy updates | Success rate |
|---|---|---|---|
| Repeated alternation, | 4,000 | 4,000 | 91.33% |
| Single switch, | 4,000 | 4,000 | 72.00% |
| Memory training with frozen policy, then joint training, | 8,000 | 4,000 | 75.33% |
| Joint training from initialization | — | — | 32.67% |
The third row is the single-round baseline suggested by the reviewer. Repeated alternation outperforms it by 16.00 percentage points, and outperforms the single switch, which uses exactly the same number of updates for each parameter group, by 19.33 percentage points.
Number of rounds. The main experiments use ten rounds (500 memory steps and 500 policy steps per round, Appendix Table 3), and the single-task experiments use eight rounds.
Sensitivity. Since each round has a fixed length, the change with the number of rounds is the change with training steps. MoveCube reaches a high success rate within the first few rounds, while PatternLock keeps improving with the number of rounds (Stage2 steps):
| Task | 1 round | 3 rounds | 6 rounds | 8 rounds |
|---|---|---|---|---|
| PatternLock | 11.33% | 25.33% | 42.67% | 51.33% |
Weakness 2 (third point) + Question 3
We thank the reviewer for the comments. We add task success rates and explain how adaptive writing differs from existing gating rules.
Task success (Question 3). 8,000 steps, 3 seeds × 50 episodes; both settings use the same tasks, model, and evaluation protocol:
| Task | Adaptive writing | Fixed |
|---|---|---|
| VideoUnmask | 91.33% | 71.33% |
| PatternLock | 51.33% | 28.67% |
Fixed disables the adaptive scaling and keeps the curvature calibration. These are single-task results. A full-benchmark control requires a complete 16-task training run (about 1,536 H200 GPU hours), which could not be completed during the rebuttal period; we acknowledge this limitation, and the revision will restrict this conclusion to the single-task setting.
Not merely suppressing later writes. We agree with the reviewer that suppressing later writes almost inevitably reduces interference with early bindings, and we therefore provide the following argument. The adaptive coefficient only attenuates repeated writes aligned with the accumulated change, while keeping strong writes for new information that conflicts with stored content and has a large reconstruction residual (Sec. 3.5, Appendix A). If the gain came only from suppressing writes, PatternLock, which requires continually writing new information, should get worse; instead, it improves by 22.67 percentage points. Both tasks require writing locations into memory: VideoUnmask is an object–location binding (evidence only at the beginning), while PatternLock is a temporal–location binding (watching a pattern path drawn step by step and then reproducing it in the same order, which requires writing at different times and indexing by progress). Adaptive writing is better on both.
Differences from Titans and gated delta rules. (a) Their forget gates proportionally decay the entire memory state; our coefficient only modulates the strength of new writes and does not decay stored content, because key evidence in RoboMME often appears early and must be retained for hundreds of frames. (b) Their gates are learned functions; our coefficient is computed analytically from the update geometry and the reconstruction residual, with no additional parameters. We have not yet conducted a matched comparison with these rules; the revision will position adaptive writing as an auxiliary design, with the core contributions being the memory architecture, the observation interface, and alternating training.
Response to Reviewer LW6g (Part 1/5)
We sincerely thank the reviewer for the thorough and constructive review. We are glad that the reviewer finds the empirical gain large: on the same backbone and with the same fast-weight TTT memory family, T²Mem gains about 35 points over the recurrent-memory baselines, with a constant-size state and no external VLM calls. We also appreciate that the reviewer considers Sec. 4.4 the strongest part of the paper, where conflicting memory drops VideoUnmask from 300/300 to 1/300, empty memory falls to chance, and pre- versus post-occlusion writes show where the useful information is stored. Below we respond to each weakness and question in turn.
Weakness 1 + Question 1
We thank the reviewer for raising this important question. We first answer Question 1 by comparing compute and training steps and by reporting results of training TTT memory under our recipe; we then explain why the RoboMME TTT baselines are not explained by insufficient training; finally, we answer Weakness 1 on how the 35-point gap is distributed.
Compute and training steps (Question 1). According to the RoboMME paper (Sec. 5.1, Appendix B.2.3):
The difference in hardware cost stems from the training procedure itself: each Stage2 sample in T²Mem is a complete episode (up to 1,104 time positions), and full BPTT is performed over the entire episode so that late action supervision can propagate back to early memory writes. We therefore use fewer optimization steps, but each step is more expensive; this cost is a structural property of the method rather than additional tuning or longer optimization. The real question is whether this cost explains the performance gap, and the controls below, run with the same recipe and compute, show that it does not.
Training TTT memory under our recipe (Question 1). We trained controlled TTT-memory variants under our own recipe and budget by removing T²Mem's components one at a time (same recipe, 8,000 steps), and compared them with the same-task scores of the RoboMME TTT variants in Table 1:
The AE-side TTT integration is similar to RoboMME's TTT-Expert, placing memory on the action-expert side. The results show that, even with exactly the same recipe and compute as T²Mem, TTT memory by itself does not bring improvement: once alternating training or the observation interface is removed, success falls back to the level of the RoboMME TTT variants (e.g., 32.67% vs. 27.22–31.78% on VideoUnmask).