acceptodds
Under review as a conference paper at ICLR 2027

LATCH: Action-Guided Perceptual Memory for Vision-Language-Action Models

Abstract

Non-Markovian manipulation requires a vision-language-action (VLA) policy to know which past observations still matter for the action being produced. Most VLAs act from the current observation or a short visual window; existing mem- ory systems either add symbolic task state through a separate high-level policy trained on subtask or keyframe annotation, or retain perceptual history with no task-general way to prioritize useful past observations. We introduce LATCH, which learns this priority inside the policy: lightweight read and write gates con- trol where the action queries attend within a uniformly sampled visual history and how strongly each frame contributes, optimized jointly with the VLA ac- tion objective. The action objective is the only supervision signal: no annotation pipeline, no retrieval policy, and a single differentiable policy at inference. We evaluate LATCH on the π0.5 and GR00T N1.6 backbones across RoboMME’s 16 memory-centric tasks, the LIBERO suites, dual-arm RMBench, and real-world memory-centric tasks on an SO-101 arm. On RoboMME, LATCH lifts π0.5 from 17.9% to 47.2% average success, higher than all other evaluated memory methods for VLAs. These results indicate that a bounded perceptual memory can be orga- nized by the action objective alone and can effectively provide historical context without keyframe or subtask annotation and without an external memory policy. Project page: https://latch-robomem.github.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, perceptual memory, differentiable gating, imitation learning, robot manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RLYHOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper adds two per-frame gates to the uniform-frame memory of MME-VLA (the FrameSamp+Modul baseline from RoboMME). A write gate rescales each frame's value vectors (sigmoid MLP on the mean frame token). A read gate adds one scalar bias per frame to the attention logits (MLP on the mean frame token and the mean action hidden state). Both gates are zero-initialized and trained with the action loss only, after a warm-start phase that trains the ungated memory policy. Experiments cover RoboMME (pi0.5 and GR00T N1.6), RMBench, LIBERO and an SO-101 arm. The main result is 44.5 -> 47.2 average success on RoboMME with pi0.5.

Critical strengths
  1. The add-on is cheap (~1.3M parameters, ~8 ms per call), zero-initialized, and in principle applies to any VLA that reads memory through attention.
  2. The evaluation is broad: two backbones, two memory benchmarks, LIBERO as a no-regression check, and a real robot. Baseline reproduction is checked (MME-VLA harness, Mem-0 on RMBench), and the weakness on Permanence tasks is stated plainly.
Critical weaknesses
  1. Limited novelty. Zero-initialized gates on cross-attention and learned, data-dependent additive biases on attention logits are standard tools (e.g., gated cross-attention in Flamingo, forget-gate logit biases in the Forgetting Transformer, zero-init adapters in general). The paper introduces no new memory representation, selection mechanism or training signal. It attaches two scalar gates to an existing benchmark baseline.

  2. The framing overstates what the gates add. L193-194 says that in the ungated policy "nothing in the readout distinguishes one moment of the history as more relevant than another". This is incorrect: cross-attention from the action queries to the memory is already content- and query-dependent, and it is already trained with the action loss alone. "Memory priority learned from the action objective, without annotation" describes MME-VLA as well. The read gate pools all action tokens into a single vector and outputs one scalar per frame, shared across heads and queries, so it is a coarser version of what attention can already express. The paper's stated insight is that frame relevance depends on the current action, yet in Table 5 the query-blind write gate alone (50.3 +- 4.0), the query-aware read gate alone (51.2 +- 1.2) and both (52.6 +- 3.4) are within noise of each other.

  3. The main gain is not separated from the training budget. On RoboMME/pi0.5, MME-VLA is the released checkpoint: 80k steps, batch 64, constant LR 5e-5. LATCH is 50k steps at batch 64 plus 30k gate steps at batch 128 with cosine decay to 2e-6. That is about 7.0M vs 5.1M training samples plus an extra annealing phase. L277 calls this "matched in compute", which it is not. Table 1 lacks the obvious control: the ungated memory policy trained with the same phase-2 schedule. The ablation has the same gap. B.1 says the budget arms were trained "without and with both gates", but the warm-started no-gate arm at 32 frames is missing from Table 5. The text only says that gated variants beat it on Counting. This is the one number that shows whether the gates matter.

  4. Gains are small and concentrated in a few tasks. On pi0.5, the overall gain is +2.7, and 5 of 16 tasks get worse (VideoPlaceButton -6.0, VideoRepick -3.7). On GR00T N1.6, the overall gain is +2.1, and PickXtimes alone (23.3 -> 55.3) accounts for +2.0 of it, while Permanence and Reference drop. Each setting appears to be a single training run evaluated with three evaluation seeds. RMBench uses 10 episodes per task with per-task stds up to 17-26 points, and most of the gap over MME-VLA comes from one task (Put Back Block, 50 -> 100). The real-robot comparison is one training seed and 25 counting trials (13 vs 8).

  5. There is no evidence that the gates learn meaningful frame priority. The paper's thesis is that the gates learn which frames matter, but it never shows or analyzes the gate values (e.g., against time index or against events like pick/release). Temporal embeddings are added before the frame summary, so the write gate could simply be encoding recency. The only test keeps the top- vs bottom-ranked third of frames at inference (38.1 vs 32.2), and both are far below the full model (52.6). It has no random-third control. It is also an out-of-distribution intervention, which is the reason the paper gives one paragraph later for not switching gates off at inference.

  6. Most ablation results are not reported. B.1 lists 33 arms, including conditioning variants (memory-only read gate, action-aware write gate, per-token write gate), gate-off/inverted/shuffled interventions, data-fraction arms and a 4-frame budget. Table 5 shows about a dozen. The interventions and conditioning variants are the experiments that would justify the read/write design, so their results should be reported, whatever they show.

  7. The paper is inconsistent about what drives the result. Table 5 attributes the largest effect (>20 points) to the warm start. But the from-scratch arms get 15k steps from base pi0.5, compared with ~95k in total for the warm-started arms, so this comparison measures training length. On the real robot, LATCH is trained jointly from base weights without a warm start and is still reported as the best method.

  8. Model selection uses test episodes. A.3 says the first ten episodes per task "serve model selection", and those episodes are part of the 50 reported evaluation episodes. The ablation subset is evaluated on the same first ten episodes.

Questions for authors
  1. On the full 16-task RoboMME protocol, what does the ungated uniform-memory policy score when trained with exactly the LATCH schedule (50k + 30k steps, batch 128 in phase 2, cosine decay)? What does the warm-started no-gate 32-frame arm score in Table 5?
  2. Please report all arms listed in B.1, especially gate-off/shuffle/invert and the three conditioning variants.
  3. How are the learned g_i and b_i distributed over an episode? Do they track time index, visual change (the TokenDrop criterion) or task events? Please add a random-third control to the frame-selection experiment.
  4. How many training seeds does each reported number use?
  5. Why does joint training from base weights work on the real robot when it costs >20 points in simulation?
3 comments
AuthorsOpenReview ↗

Response to Reviewer RLYH (1/2)

Thank you for the detailed review. Questions first, then the weaknesses; table numbers refer to the revised manuscript (changes in blue).

Q1. On the full 16-task RoboMME protocol, what does the ungated uniform-memory policy score when trained with exactly the LATCH schedule (50k + 30k steps, batch 128 in phase 2, cosine decay)? What does the warm-started no-gate 32-frame arm score in Table 5?

We checked the training logs, fixed this and retrained. MME-VLA∗ now gets exactly the training of LATCH: the same 30k steps after the released checkpoint, batch size, cosine schedule and hardware, without the gates. Table 1 now has every MME-VLA version:

RoboMME, π0.5CountingPermanenceReferenceImitationOverall
MME-VLA†, reported by RoboMME65.225.136.351.444.5
MME-VLA‡, released checkpoint, our harness68.525.036.749.544.9
MME-VLA∗, same training as LATCH68.027.234.752.845.7
LATCH69.828.237.353.747.2

GR00T N1.6 is fixed in the same way (LATCH 40k + 20k steps, MME-VLA 60k), and its memory-free row is now the vanilla checkpoint, not HAMLET with memory off (blue cells).

We will add further ablation results to Appendix B.1 before the discussion closes (Q2); the full-benchmark comparison above, with ten times more episodes, is the stronger test.

Q2. Please report all arms listed in B.1, especially gate-off/shuffle/invert and the three conditioning variants.

Sorry, this was a reporting issue we have highlighted in yellow . These arms were initially planned as extra checks: gates switched off, inverted or shuffled at test time, and variants of the data and the memory budget. We found the reported ablation sufficient to show that the gates rank the frames and to test the other choices, and the other benchmarks show what the gates add, so we left them out. We will add further ablation results to Appendix B.1 before the discussion closes.

Q3. How are the learned g_i and b_i distributed over an episode? Do they track time index, visual change (the TokenDrop criterion) or task events? Please add a random-third control to the frame-selection experiment.

New Figure 5 (Appendix B.3) shows the read-gate bias b_i on three real-robot episodes. b_i follows the task events, not the time index: it is high between the first grasp and the last release (40 of 41 frames above the mean) and low both before and after (39 of 40 below). The write gate g_i changes little from frame to frame there. We will add the random-third control to Table 5 before the discussion closes.

Q4. How many training seeds does each reported number use?

One training seed, three evaluation seeds (now in Appendix A.3). More seeds are cheap for small policies on a few tasks, not for a 3.5B-parameter VLA on these benchmarks: one row of Table 1 takes about 56 GPU-hours of training and 43 of evaluation (2,400 episodes), so at these rates the whole study takes about 2,700 GPU-hours per training seed, and three training seeds would need about 8,000 GPU-hours in total. We spent the compute on more settings instead (second back bone, benchmark (dual-arm), real robot); LATCH leads in all those independent trainings

From Tables 1, 2 and 4; HAMLET, MME-VLA and LATCH get the same training steps in each setting (real robot: each to its loss plateau).

Q5. Why does joint training from base weights work on the real robot when it costs >20 points in simulation?

Because the budget differs, as you note in W7. In simulation, the arm without the warm start trains only 15k steps on 16 tasks, without the uniform-memory policy's 80k steps. On the real robot there are three tasks and 200 demonstrations, and every policy trains until its loss plateaus, so memory and gates can be learned together. Sec. 4.2 now states this.

AuthorsOpenReview ↗

Response to Reviewer RLYH (2/2)

W1. We agree the tools are standard and claim no new gating operator. New is where the gates act: Flamingo [1] has one scalar per layer, the Forgetting Transformer [2] a forget gate per token. LATCH has one gate per remembered frame, and its read gate depends on the action being produced (now in Sec. 2). This makes an annotation-free perceptual memory more reliable competitive with hybrid methods that need labels and an extra VLM: ahead of MemER [3] on RoboMME (47.2 vs 42.4), close to Mem-0 [4] on RMBench (36.3 vs 37.8).

W2. You are right about L193-194: attention over the memory is already content- and query-dependent, and we have adjusted the sentence (Sec. 3). The gates add one explicit term per frame on top of the token-level attention; they are coarse on purpose. Each gate has its strength: the write gate on Permanence (41.7 vs 36.7), the read gate on the other three families, and LATCH with both is best overall (Table 5).

W3. Fixed (Q1). "Matched in compute" is replaced by the exact steps and schedules (Sec. 4.1, Appendix A.3). Every ablation arm trains for the same 15k steps, and further ablation results follow (Q2).

W4. With matched training, LATCH is ahead of MME-VLA on 13 of the 16 tasks, equal on one and lower on two (Table 7); VideoPlaceButton and VideoRepick are now +1.3 and +2.7. On GR00T N1.6 (fixed, Q1), Permanence and Reference no longer drop, and PickXtimes gives 0.7 of the 2.1 points. On RMBench, Put Back Block gives about half of the gain; three more tasks gain 10 to 27 points (Table 8).

W5. Figure 5 now shows them (Q3): the read gate follows task events, not recency. The frame-selection test is about the ranking, so the comparison is relative: every selection removes the same number of frames, and the top-ranked third keeps more success than the bottom-ranked third (Table 5).

W6. Unreported ablation arms. See Q2.

W7. The warm start is LATCH's first stage, and every simulation result uses it. The two arms without it (Table 5) also have fewer steps in total, as you say (now in Sec. 4.2). Real robot: see Q5.

W8. No checkpoint is selected on evaluation episodes: every reported policy is its schedule's final checkpoint. The unclear A.3 sentence is rewritten: it refers to the ablation study, which uses the first 10 of the 50 episodes because a full evaluation is expensive (Q4). The RoboMME challenge at CVPR 2026 [5] uses the same first 10 episodes per task in its first phase.

One correction w.r.t S1: the gates add 23.6M parameters (0.7% of the policy π0.5), not 1.3M; 1.3M is the count for one of the 18 blocks (Appendix A.1).

[1] Alayrac et al. Flamingo: a Visual Language Model for Few-Shot Learning. NeurIPS, 2022.

[2] Lin, Nikishin, He, Courville. Forgetting Transformer: Softmax Attention with a Forget Gate. ICLR, 2025.

[3] Sridhar, Pan, Sharma, Finn. Scaling up Memory for Robotic Control via Experience Retrieval (MemER). ICLR, 2026.

[4] Chen et al. RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design (Mem-0). arXiv:2603.01229, 2026.

[5] RoboMME Challenge, CVPR 2026 (phase 1: 10 episodes per task): https://robomme.github.io/challenge.html

AuthorsOpenReview ↗

Response to Reviewer RLYH

Thank you again for the detailed review. We managed to complete the following requested ablations, now in Tables 11 and 12; all changes to the submission are in blue. For Q3, we added the random-third control (Tables 5 and 12) and the frame selection at 8 and 16 of 32 frames (Table 12). For Q2, we added the gates switched off and inverted at test time (Table 12) and the three conditioning variants (Table 11). For W6, we added the 4-frame budget (Table 11).

The purpose of this ablation is to assess the relative advantage of the gate choices. As pointed out earlier, the full RoboMME benchmark (Table 1) and the results on RMBench and the real robot (Tables 2 and 4) are still the stronger signals that establish the advantage of the gates.

Reviewer_9uykOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to address the problem of robots’ heavy reliance on historical information in non-Markovian manipulation tasks. Existing VLAs typically predict actions based on current observations or a short visual history, making it difficult to handle tasks that require long-term memory. Building upon existing uniformly sampled visual memory architectures, the authors introduce two lightweight gating modules: Read Gate and Write Gate. These modules independently adjust the attention weights and feature contributions of historical visual information, enabling the model to utilize different historical observations based on current action requirements. Both modules are trained end-to-end directly via the action prediction loss, without requiring additional memory annotations or independent memory retrieval strategies.

The paper has a clear overall structure, and the method design is easy to understand. The experiments cover various model architectures, simulation environments, and real-world robot platforms. Compared to methods that rely on additional memory annotations or independent high-level strategies, LATCH offers a relatively simple and easy-to-integrate solution.

The memory problem in VLA has recently garnered significant attention, though it is not a new issue. The core design of this paper is primarily built upon existing perception-memory architectures. The performance gains resulting from the newly introduced gating mechanism are limited. Its independent contribution and advantages over other simple memory-weighted strategies still need to be further clarified. I think the experiments in this paper are well-conducted. However, it still lacks an innovative highlight, both in terms of its starting point and its methodological approach.

Critical strengths
  1. The LATCH design does not require additional keyframes, subtask annotations, or a separate high-level memory strategy. Two lightweight gating modules can be directly integrated into the existing VLA’s memory-attention architecture. This results in a minimal increase in the number of parameters, making it practically viable for engineering applications.

  2. The authors validated the method on two architectures: π0.5 and GR00T N1.6. They also conducted a comprehensive set of experimental evaluations. These experiments demonstrated the method’s applicability across different model architectures, robot configurations, and task types.

Critical weaknesses
  1. The method described in this paper primarily introduces two MLPs into the existing uniform-sampling historical memory and Cross-Attention architecture of MME-VLA, which are used to adjust the attention logits and values, respectively. This is mechanistically similar to existing attention-gating and feature-modulation methods. Overall innovation is relatively limited. Based on the experimental results, the improvement in π0.5 on RoboMME from 17.9% to 47.2% is primarily attributable to the introduction of historical memory itself. In contrast, LATCH improved over MME-VLA by only 2.7%. The author needs to further explain what makes this design unique compared to simple attention-weighted methods.

  2. In Table 5, the average success rate when using a single Read Gate is 51.2%, while using two Gates simultaneously yields a success rate of 52.6%. This is an increase of only 1.4%, on the same order of magnitude as the reported standard deviation. Therefore, the results of this ablation experiment are insufficient to fully demonstrate the complementarity of the Read Gate and Write Gate.

  3. In addition, LATCH always uses uniformly sampled historical frames. Its gating module cannot recover critical historical events that were missed during sampling. As shown in Table 5, when the number of history frames increased from 32 to 64 and 128, the average success rate decreased from 52.6% to 45.0% and 37.9%. This result shows that increasing the number of historical frames actually reduces the model's performance. It is unclear whether this method is applicable to complex operations involving longer time series.

  4. Another issue with the experiment is that LATCH differs from some Baseline models in terms of batch size and learning rate schedule. The authors need to provide a more thorough explanation of whether these training settings affect the performance comparison.

Questions for authors

LATCH and MME-VLA use different training phases and learning rate settings. It would be helpful if the authors could control the experimental variables to rule out the influence of differences in training strategies on performance improvements.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 9uyk (1/2)

Thank you for the clear summary and the positive words on the experiments. We answer your question first and then the weaknesses. Table numbers refer to the revised manuscript, where changes are in blue.

Q1. LATCH and MME-VLA use different training phases and learning rate settings. It would be helpful if the authors could control the experimental variables to rule out the influence of differences in training strategies on performance improvements.

Yes, done. We retrained the baselines with exactly the training of LATCH: the same extra 30k steps, batch size, learning-rate schedule and hardware, just without the gates (Table 1, rows marked ∗). LATCH stays ahead in every task family:

RoboMME, π0.5 (∗: same training as LATCH)CountingPermanenceReferenceImitationOverall
π0.5∗, no memory28.819.316.210.818.8
HAMLET∗35.225.024.821.226.5
MME-VLA†, reported by RoboMME65.225.136.351.444.5
MME-VLA‡, released checkpoint, our harness68.525.036.749.544.9
MME-VLA∗, uniform memory68.027.234.752.845.7
LATCH69.828.237.353.747.2

LATCH also stays ahead on a second backbone, a second benchmark and a real robot (Tables 1, 2 and 4).

AuthorsOpenReview ↗

Response to Reviewer 9uyk (2/2)

W1 and summary. Novelty, and what is unique compared to simple attention weighting. The gates are simple on purpose. What is new is where they act and what they see. Existing gating works per layer, per token or per feature channel. LATCH has one gate per remembered frame, the read gate depends on the action being produced, and both are learned from the action loss alone, with no labels and no second model (now stated in Sec. 2). Simple attention over the same memory is exactly MME-VLA, and LATCH is ahead of it on both memory benchmarks and both backbones (Q1), most clearly on RMBench (36.3 vs 25.6). Other memory designs are far behind (HAMLET 26.5, SAM2Act+ 21.4, MemoryVLA 7.1; Table 1). Most of the gain from 17.9 is indeed the memory itself. Our highlight is that this perceptual memory, with no annotation, is competitive with hybrid methods that need manual labels and an extra VLM: ahead of MemER [1] on RoboMME (47.2 vs 42.4) and close to Mem-0 [2] on RMBench (36.3 vs 37.8).

W2. One gate versus two. We should have stated more clearly what each gate is good at. The write gate is better on Permanence (41.7 vs 36.7), the read gate on the other three families, and LATCH with both is the best overall (Table 5; per task in new Table 12). The ablation uses a small and simpler subset (8 tasks, 10 episodes each). On the full benchmark, LATCH with both gates is ahead of the ungated memory in every family (table in Q1).

Trained from the warm startCountingPermanenceReferenceImitationOverall
Write gate only50.041.735.074.650.3
Read gate only51.736.736.780.051.2
LATCH, both gates55.040.035.680.052.6

W3. Uniform sampling and longer horizons. Yes. Our focus is perceptual memory: the goal is to make the recent history reliable. For very long horizons and single events, methods like MemER [1] and Mem-0 [2] work well, but they need manually annotated data and a separately trained high-level policy. How to do this without labels, and so extend what perceptual memory can do, is the holy grail and the key focus of our research and future work. Adding more context directly to the policy is not free: it starts to form random relations between frames [3, 4, 5, 6], which hurts tasks like counting, and the problem grows with more frames (Table 5). HAMLET [7] also reports that naive multi-frame input lowers a VLA on LIBERO. We plan to combine perceptual memory and the gates with routing and test-time training [8] to tackle this. All of this is now stated in the limitations (Sec. 4.2). Still, from a plain π0.5 to LATCH the gain is already large: 18.8 to 47.2 on RoboMME.

W4. Different batch size and learning-rate schedule. Fixed. We retrained the baselines with the settings of LATCH and updated the manuscript (Table 1, rows marked ∗; see Q1).

[1] Sridhar, Pan, Sharma, Finn. Scaling up Memory for Robotic Control via Experience Retrieval (MemER). ICLR, 2026.

[2] Chen et al. RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design (Mem-0). arXiv:2603.01229, 2026.

[3] de Haan, Jayaraman, Levine. Causal Confusion in Imitation Learning. NeurIPS, 2019.

[4] Wen, Lin, Darrell, Jayaraman, Gao. Fighting Copycat Agents in Behavioral Cloning from Observation Histories. NeurIPS, 2020.

[5] Seo, Hwang, Yang, Kim. Regularized Behavior Cloning for Blocking the Leakage of Past Action Information. NeurIPS, 2023.

[6] Torne, Tang, Liu, Finn. Learning Long-Context Diffusion Policies via Past-Token Prediction. CoRL, 2025.

[7] Koo et al. HAMLET: Switch your Vision-Language-Action Model into a History-Aware Policy. ICLR, 2026.

[8] Behrouz, Zhong, Mirrokni. Titans: Learning to Memorize at Test Time. arXiv:2501.00663, 2024.

Thank you for the detailed rebuttal and additional experiments. I appreciate that the authors have addressed the training fairness concern by retraining the baselines under matched settings.

However, my main concern regarding the limited methodological novelty remains. The improvement over MME-VLA is still modest at 1.5%, and the advantages of the proposed gating mechanism are not sufficiently established. The limitations regarding longer-horizon memory also remain unaddressed.

While I appreciate the authors' efforts, these responses do not substantially change my overall assessment. Therefore, I maintain my original rating.

Thank you for the follow-up and for acknowledging the additional controlled experiments. We would like to clarify the evidence for the gates and the scope of our contribution.

The advantage of the gates becomes evident when we move to a different benchmark: on RMBench, LATCH reaches 36.3 against 25.6 for MME-VLA (Table 2).

Additionally, as we explained in W3, perceptual memory is an annotation-free, whose goal is to make the immediate memory buffer more reliable. It is interesting that it is competitive with Mem-0 and MemER, which are designed for long horizons and need manual labels (36.3 vs 37.8 for Mem-0 on RMBench, Table 2 and 47.2 vs 42.4 for MemER on RoboMME, Table 1).

However, both have their advantages: on RoboMME, perceptual memory is ahead on Counting and Imitation (69.8 vs 48.8 and 53.7 vs 29.5) where reliability in recent memory matters, and the hybrid is ahead on Permanence (53.2 vs 28.2) where long-horizon tracking matters. We acknowledge this in the limitations (Sec. 4.2), and our future work is to combine both.

We hope this clarifies the advantages of the gates and the scope of our work.

Reviewer_MhXbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LATCH adds two lightweight gates to a bounded perceptual memory inside a VLA model. A uniform sample of past frames conditions the action expert by cross-attention and feature-wise modulation. A query-aware read gate biases the attention logits, and a query-blind write gate scales each frame’s values. On RoboMME with pi0.5, LATCH has 47.2% average success, compared with 44.5% for MME-VLA and 17.9% for memory-free pi0.5. The paper also evaluates dual-arm RMBench, LIBERO, and three tasks on a physical SO-101 arm.

Critical strengths
  1. The proposed mechanism is simple and effective. The gates start as the uniform-memory policy, use only the action objective, and add about 1.3M parameters.

  2. The experiments cover pi0.5, GR00T, dual-arm RMBench, LIBERO, and the SO-101 physical robot. The experiments are thoroughly reliable.

Critical weaknesses

Regarding the 17.9% to 47.2% range mentioned in the abstract, the "uniform memory" baseline already reaches 44.5%; adding the two gates yields an improvement of only 2.7 percentage points, with performance actually declining on some tasks. Ablation studies reveal that the bulk of this ~20-point gain stems from "warm-starting" with uniform memory, while the difference between using two gates versus a single gate falls within the range of random fluctuation. In real-world deployment, the model with two gates performs on par with the uniform memory baseline across two ranking tasks, achieving scores of 8/10 and 10/10 in both cases.

The proposed mechanism does not appear to offer significant advantage. Could it be that the evaluation failed to utilize appropriate benchmarks that incorporate memory mechanisms?

Questions for authors
  1. Uniform sampling reaches 44.5% memory utilization, while LATCH has 47.2%. Does this gap stem from the two "gates," or is it simply the result of further training on the same memory set?
  2. The gates partition historical frames into "more useful" and "less useful" subsets. Yet, when only the top-ranked frames are retained, the success rate drops from 52.6% to 38.1%. Are these gates actually selecting the evidence truly required for the current action?
  3. Object metrics like "Permanence" and "Reference" show little improvement (in some tasks, they even decline). Could it be that the benefits are limited to tasks involving counting or those where the necessary evidence is distributed across the entire sequence of actions?
1 comment
AuthorsOpenReview ↗

Response to Reviewer MhXb

Thank you for the careful reading and the encouraging comments. We answer the three questions first and then the weaknesses. Table numbers refer to the revised manuscript, where changes are in blue.

Q1. Uniform sampling reaches 44.5% memory utilization, while LATCH has 47.2%. Does this gap stem from the two "gates," or is it simply the result of further training on the same memory set?

Further training alone does not close the gap. We trained MME-VLA for the same extra 30k steps as LATCH (same schedule, batch size and hardware, just without the gates): it reaches 45.7, LATCH 47.2 (Table 1, rows marked ∗). LATCH also stays ahead on a second backbone (GR00T N1.6, 23.6 vs 21.5), on the dual-arm RMBench (36.3 vs 25.6) and on real-robot counting (13 vs 8 exact counts in 25 trials).

RoboMME, π0.5CountingPermanenceReferenceImitationOverall
MME-VLA†, reported by RoboMME65.225.136.351.444.5
MME-VLA‡, released checkpoint, our harness68.525.036.749.544.9
MME-VLA∗, same training as LATCH68.027.234.752.845.7
LATCH69.828.237.353.747.2

LATCH also stays ahead on a second backbone, a second benchmark (dual arms) and a real robot:

Q2. The gates partition historical frames into "more useful" and "less useful" subsets. Yet, when only the top-ranked frames are retained, the success rate drops from 52.6% to 38.1%. Are these gates actually selecting the evidence truly required for the current action?

The gates rank the frames, they do not remove any. The policy is trained with all 32 frames, so it drops when it gets only a third of them, whichever third. The test is about the ranking: the top-ranked third keeps clearly more success than the bottom-ranked third (38.1 vs 32.2, in every task family; Table 5). On real robot episodes, the read gate gives higher values to the frames between the first grasp and the last release (new Figure 5, Appendix B.3).

Q3. Object metrics like "Permanence" and "Reference" show little improvement (in some tasks, they even decline). Could it be that the benefits are limited to tasks involving counting or those where the necessary evidence is distributed across the entire sequence of actions?

W1. + 2.7 points, and some tasks decline. With matched training, LATCH is ahead of MME-VLA on 13 of the 16 RoboMME tasks (Table 7). The advantage holds on a second backbone (GR00T N1.6) and grows on the dual-arm RMBench (36.3 vs 25.6), a different embodiment and benchmark that tests a different aspect of memory. See Q1.

W2. Warm start, and one gate versus two. Yes, the warm start comes first: it teaches the policy to use the memory frames. The gates then refine where the policy puts importance. The ablations use a small subset of the evaluation, to show how the gates work: the write gate is better on Permanence, the read gate on the other families, and LATCH with both is the best overall (Table 5; bold marks the better single gate, and where LATCH is best).

Trained from the warm startCountingPermanenceReferenceImitationOverall
Write gate only50.041.735.074.650.3
Read gate only51.736.736.780.051.2
LATCH, both gates55.040.035.680.052.6

LATCH also stays ahead on a second backbone, a second benchmark and a real robot (Tables 1, 2 and 4).

W3. Real robot: on par on two tasks. Each task tests a different aspect of perceptual memory (Table 4). The two ordering tasks show the limit of HAMLET (1/10 and 3/10, against 8/10 and 10/10 for LATCH and MME-VLA). The counting task shows what the gates add over MME-VLA: 13 against 8 exact counts in 25 trials. Real-robot time is limited, so we also evaluate on two memory benchmarks with different embodiments, RoboMME and RMBench.

W4. Is there an advantage, and are the benchmarks right? RoboMME and RMBench are both memory benchmarks for robot manipulation. We evaluate perceptual memory, the ability to extend the immediate memory reliably, which counting shows best. LATCH is the best perceptual memory method on both, across two backbones and two embodiments. It is also competitive with hybrid methods that need manual labels and an additional VLM: ahead of MemER on RoboMME (47.2 vs 42.4) and close to Mem-0 on RMBench (36.3 vs 37.8).

One correction w.r.t S1: the gates add 23.6M parameters (0.7% of the policy π0.5), not 1.3M; 1.3M is the count for one of the 18 blocks (Appendix A.1).

Comments on the paper

1 comment
AuthorsOpenReview ↗

LATCH Rebuttal Summary

Dear Area Chair,

Thank you for overseeing the review. We also thank the reviewers for their valuable time and feedback. We would like to summarize LATCH’s contribution and the additional evidence provided during the discussion. We would like to summarize LATCH’s contribution and the additional evidence provided during the discussion.

  • LATCH adds explicit frame-level control to the perceptual memory of a VLA. The read gate adjusts a frame’s attention priority based on the action being produced, and the write gate controls its contribution to memory. Both gates are learned through the action objective, without additional memory annotations or a separate high-level policy. Our contribution is this specific way of organizing perceptual history inside the policy, which we evaluate across two backbones, two memory benchmarks and a physical robot.

  • The main experimental concern raised by reviewers MhXb, 9uyk and RLYH was whether the improvement came from the gates or from additional training. We addressed this by retraining the baselines with the same additional steps, batch size, learning-rate schedule and hardware. Under this matched training, LATCH reaches 47.2 against 45.7 for MME-VLA on RoboMME, improving every task family and 13 of the 16 tasks. Reviewer 9uyk acknowledged that the training fairness concern had been addressed.

  • The advantage of the gates becomes more evident on RMBench, where LATCH reaches 36.3 against 25.6 for MME-VLA. The improvement also holds with GR00T N1.6 (23.6 vs 21.5) and in real-robot counting (13 vs 8 exact counts in 25 trials). These results provide evidence for the contribution of the gates across benchmarks, backbones and real-robot execution.

  • We provided additional ablations requested by reviewer RLYH, including random frame selection, gates switched off or inverted at inference, three gate-conditioning variants, and different frame-selection and memory budgets (Tables 5, 11 and 12). We also added gate-value visualizations over real-robot episodes (Figure 5). These experiments and questions helped us clarify the method and the training and evaluation settings, including final-checkpoint evaluation and the use of one training seed with three evaluation seeds.

  • As we explained to reviewer 9uyk, our focus is perceptual memory: making the immediate memory buffer more reliable without additional memory annotations. Within this scope, LATCH not only outperforms the other evaluated perceptual memory methods but is also competitive with Mem-0 and MemER, which are designed for longer horizons and require manual labels (36.3 vs 37.8 for Mem-0 on RMBench, Table 2; 47.2 vs 42.4 for MemER on RoboMME, Table 1). Both approaches have their advantages. On RoboMME, perceptual memory is ahead on Counting and Imitation (69.8 vs 48.8 and 53.7 vs 29.5), while hybrid memory is ahead on Permanence (53.2 vs 28.2). We acknowledge this in the limitations (Sec. 4.2), and our future work is to combine both.

Thank you for considering our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.