- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper adds two per-frame gates to the uniform-frame memory of MME-VLA (the FrameSamp+Modul baseline from RoboMME). A write gate rescales each frame's value vectors (sigmoid MLP on the mean frame token). A read gate adds one scalar bias per frame to the attention logits (MLP on the mean frame token and the mean action hidden state). Both gates are zero-initialized and trained with the action loss only, after a warm-start phase that trains the ungated memory policy. Experiments cover RoboMME (pi0.5 and GR00T N1.6), RMBench, LIBERO and an SO-101 arm. The main result is 44.5 -> 47.2 average success on RoboMME with pi0.5.
Critical strengths
- The add-on is cheap (~1.3M parameters, ~8 ms per call), zero-initialized, and in principle applies to any VLA that reads memory through attention.
- The evaluation is broad: two backbones, two memory benchmarks, LIBERO as a no-regression check, and a real robot. Baseline reproduction is checked (MME-VLA harness, Mem-0 on RMBench), and the weakness on Permanence tasks is stated plainly.
Critical weaknesses
-
Limited novelty. Zero-initialized gates on cross-attention and learned, data-dependent additive biases on attention logits are standard tools (e.g., gated cross-attention in Flamingo, forget-gate logit biases in the Forgetting Transformer, zero-init adapters in general). The paper introduces no new memory representation, selection mechanism or training signal. It attaches two scalar gates to an existing benchmark baseline.
-
The framing overstates what the gates add. L193-194 says that in the ungated policy "nothing in the readout distinguishes one moment of the history as more relevant than another". This is incorrect: cross-attention from the action queries to the memory is already content- and query-dependent, and it is already trained with the action loss alone. "Memory priority learned from the action objective, without annotation" describes MME-VLA as well. The read gate pools all action tokens into a single vector and outputs one scalar per frame, shared across heads and queries, so it is a coarser version of what attention can already express. The paper's stated insight is that frame relevance depends on the current action, yet in Table 5 the query-blind write gate alone (50.3 +- 4.0), the query-aware read gate alone (51.2 +- 1.2) and both (52.6 +- 3.4) are within noise of each other.
-
The main gain is not separated from the training budget. On RoboMME/pi0.5, MME-VLA is the released checkpoint: 80k steps, batch 64, constant LR 5e-5. LATCH is 50k steps at batch 64 plus 30k gate steps at batch 128 with cosine decay to 2e-6. That is about 7.0M vs 5.1M training samples plus an extra annealing phase. L277 calls this "matched in compute", which it is not. Table 1 lacks the obvious control: the ungated memory policy trained with the same phase-2 schedule. The ablation has the same gap. B.1 says the budget arms were trained "without and with both gates", but the warm-started no-gate arm at 32 frames is missing from Table 5. The text only says that gated variants beat it on Counting. This is the one number that shows whether the gates matter.
-
Gains are small and concentrated in a few tasks. On pi0.5, the overall gain is +2.7, and 5 of 16 tasks get worse (VideoPlaceButton -6.0, VideoRepick -3.7). On GR00T N1.6, the overall gain is +2.1, and PickXtimes alone (23.3 -> 55.3) accounts for +2.0 of it, while Permanence and Reference drop. Each setting appears to be a single training run evaluated with three evaluation seeds. RMBench uses 10 episodes per task with per-task stds up to 17-26 points, and most of the gap over MME-VLA comes from one task (Put Back Block, 50 -> 100). The real-robot comparison is one training seed and 25 counting trials (13 vs 8).
-
There is no evidence that the gates learn meaningful frame priority. The paper's thesis is that the gates learn which frames matter, but it never shows or analyzes the gate values (e.g., against time index or against events like pick/release). Temporal embeddings are added before the frame summary, so the write gate could simply be encoding recency. The only test keeps the top- vs bottom-ranked third of frames at inference (38.1 vs 32.2), and both are far below the full model (52.6). It has no random-third control. It is also an out-of-distribution intervention, which is the reason the paper gives one paragraph later for not switching gates off at inference.
-
Most ablation results are not reported. B.1 lists 33 arms, including conditioning variants (memory-only read gate, action-aware write gate, per-token write gate), gate-off/inverted/shuffled interventions, data-fraction arms and a 4-frame budget. Table 5 shows about a dozen. The interventions and conditioning variants are the experiments that would justify the read/write design, so their results should be reported, whatever they show.
-
The paper is inconsistent about what drives the result. Table 5 attributes the largest effect (>20 points) to the warm start. But the from-scratch arms get 15k steps from base pi0.5, compared with ~95k in total for the warm-started arms, so this comparison measures training length. On the real robot, LATCH is trained jointly from base weights without a warm start and is still reported as the best method.
-
Model selection uses test episodes. A.3 says the first ten episodes per task "serve model selection", and those episodes are part of the 50 reported evaluation episodes. The ablation subset is evaluated on the same first ten episodes.
Questions for authors
- On the full 16-task RoboMME protocol, what does the ungated uniform-memory policy score when trained with exactly the LATCH schedule (50k + 30k steps, batch 128 in phase 2, cosine decay)? What does the warm-started no-gate 32-frame arm score in Table 5?
- Please report all arms listed in B.1, especially gate-off/shuffle/invert and the three conditioning variants.
- How are the learned g_i and b_i distributed over an episode? Do they track time index, visual change (the TokenDrop criterion) or task events? Please add a random-third control to the frame-selection experiment.
- How many training seeds does each reported number use?
- Why does joint training from base weights work on the real robot when it costs >20 points in simulation?
3 comments
Response to Reviewer RLYH (2/2)
W1. We agree the tools are standard and claim no new gating operator. New is where the gates act: Flamingo [1] has one scalar per layer, the Forgetting Transformer [2] a forget gate per token. LATCH has one gate per remembered frame, and its read gate depends on the action being produced (now in Sec. 2). This makes an annotation-free perceptual memory more reliable competitive with hybrid methods that need labels and an extra VLM: ahead of MemER [3] on RoboMME (47.2 vs 42.4), close to Mem-0 [4] on RMBench (36.3 vs 37.8).
W2. You are right about L193-194: attention over the memory is already content- and query-dependent, and we have adjusted the sentence (Sec. 3). The gates add one explicit term per frame on top of the token-level attention; they are coarse on purpose. Each gate has its strength: the write gate on Permanence (41.7 vs 36.7), the read gate on the other three families, and LATCH with both is best overall (Table 5).
W3. Fixed (Q1). "Matched in compute" is replaced by the exact steps and schedules (Sec. 4.1, Appendix A.3). Every ablation arm trains for the same 15k steps, and further ablation results follow (Q2).
W4. With matched training, LATCH is ahead of MME-VLA on 13 of the 16 tasks, equal on one and lower on two (Table 7); VideoPlaceButton and VideoRepick are now +1.3 and +2.7. On GR00T N1.6 (fixed, Q1), Permanence and Reference no longer drop, and PickXtimes gives 0.7 of the 2.1 points. On RMBench, Put Back Block gives about half of the gain; three more tasks gain 10 to 27 points (Table 8).
W5. Figure 5 now shows them (Q3): the read gate follows task events, not recency. The frame-selection test is about the ranking, so the comparison is relative: every selection removes the same number of frames, and the top-ranked third keeps more success than the bottom-ranked third (Table 5).
W6. Unreported ablation arms. See Q2.
W7. The warm start is LATCH's first stage, and every simulation result uses it. The two arms without it (Table 5) also have fewer steps in total, as you say (now in Sec. 4.2). Real robot: see Q5.
W8. No checkpoint is selected on evaluation episodes: every reported policy is its schedule's final checkpoint. The unclear A.3 sentence is rewritten: it refers to the ablation study, which uses the first 10 of the 50 episodes because a full evaluation is expensive (Q4). The RoboMME challenge at CVPR 2026 [5] uses the same first 10 episodes per task in its first phase.
One correction w.r.t S1: the gates add 23.6M parameters (0.7% of the policy π0.5), not 1.3M; 1.3M is the count for one of the 18 blocks (Appendix A.1).
[1] Alayrac et al. Flamingo: a Visual Language Model for Few-Shot Learning. NeurIPS, 2022.
[2] Lin, Nikishin, He, Courville. Forgetting Transformer: Softmax Attention with a Forget Gate. ICLR, 2025.
[3] Sridhar, Pan, Sharma, Finn. Scaling up Memory for Robotic Control via Experience Retrieval (MemER). ICLR, 2026.
[4] Chen et al. RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design (Mem-0). arXiv:2603.01229, 2026.
[5] RoboMME Challenge, CVPR 2026 (phase 1: 10 episodes per task): https://robomme.github.io/challenge.html
Response to Reviewer RLYH
Thank you again for the detailed review. We managed to complete the following requested ablations, now in Tables 11 and 12; all changes to the submission are in blue. For Q3, we added the random-third control (Tables 5 and 12) and the frame selection at 8 and 16 of 32 frames (Table 12). For Q2, we added the gates switched off and inverted at test time (Table 12) and the three conditioning variants (Table 11). For W6, we added the 4-frame budget (Table 11).
The purpose of this ablation is to assess the relative advantage of the gate choices. As pointed out earlier, the full RoboMME benchmark (Table 1) and the results on RMBench and the real robot (Tables 2 and 4) are still the stronger signals that establish the advantage of the gates.
Response to Reviewer RLYH (1/2)
Thank you for the detailed review. Questions first, then the weaknesses; table numbers refer to the revised manuscript (changes in blue).
Q1. On the full 16-task RoboMME protocol, what does the ungated uniform-memory policy score when trained with exactly the LATCH schedule (50k + 30k steps, batch 128 in phase 2, cosine decay)? What does the warm-started no-gate 32-frame arm score in Table 5?
We checked the training logs, fixed this and retrained. MME-VLA∗ now gets exactly the training of LATCH: the same 30k steps after the released checkpoint, batch size, cosine schedule and hardware, without the gates. Table 1 now has every MME-VLA version:
GR00T N1.6 is fixed in the same way (LATCH 40k + 20k steps, MME-VLA 60k), and its memory-free row is now the vanilla checkpoint, not HAMLET with memory off (blue cells).
We will add further ablation results to Appendix B.1 before the discussion closes (Q2); the full-benchmark comparison above, with ten times more episodes, is the stronger test.
Q2. Please report all arms listed in B.1, especially gate-off/shuffle/invert and the three conditioning variants.
Sorry, this was a reporting issue we have highlighted in yellow . These arms were initially planned as extra checks: gates switched off, inverted or shuffled at test time, and variants of the data and the memory budget. We found the reported ablation sufficient to show that the gates rank the frames and to test the other choices, and the other benchmarks show what the gates add, so we left them out. We will add further ablation results to Appendix B.1 before the discussion closes.
Q3. How are the learned g_i and b_i distributed over an episode? Do they track time index, visual change (the TokenDrop criterion) or task events? Please add a random-third control to the frame-selection experiment.
New Figure 5 (Appendix B.3) shows the read-gate bias b_i on three real-robot episodes. b_i follows the task events, not the time index: it is high between the first grasp and the last release (40 of 41 frames above the mean) and low both before and after (39 of 40 below). The write gate g_i changes little from frame to frame there. We will add the random-third control to Table 5 before the discussion closes.
Q4. How many training seeds does each reported number use?
One training seed, three evaluation seeds (now in Appendix A.3). More seeds are cheap for small policies on a few tasks, not for a 3.5B-parameter VLA on these benchmarks: one row of Table 1 takes about 56 GPU-hours of training and 43 of evaluation (2,400 episodes), so at these rates the whole study takes about 2,700 GPU-hours per training seed, and three training seeds would need about 8,000 GPU-hours in total. We spent the compute on more settings instead (second back bone, benchmark (dual-arm), real robot); LATCH leads in all those independent trainings
From Tables 1, 2 and 4; HAMLET, MME-VLA and LATCH get the same training steps in each setting (real robot: each to its loss plateau).
Q5. Why does joint training from base weights work on the real robot when it costs >20 points in simulation?
Because the budget differs, as you note in W7. In simulation, the arm without the warm start trains only 15k steps on 16 tasks, without the uniform-memory policy's 80k steps. On the real robot there are three tasks and 200 demonstrations, and every policy trains until its loss plateaus, so memory and gates can be learned together. Sec. 4.2 now states this.