- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors present a hierarchical framework, MomentMem, for memory-based tasks in robotics, along with a benchmark, VMBench, to evaluate their method and other memory policies on tasks that require visual moment memory. These are tasks where the information required to solve them have left the current observation. MomentMem trains a VLM to select relevant keyframes using segment-level supervision, and then feeds these keyframes to a VLA policy, which act as a low-level executor.
Critical strengths
- VMBench is well-motivated and introduces an interesting experiment regarding generalization for memory-based tasks. Most previous works focus on just performance for memory-based tasks for the tasks the train, but Section 5.4 shows transfer results for an unseen task with few annotations, illustrating the importance of memory-based pretraining for robotics tasks.
- Training with multiple valid keyframes within a segment is well-motivated with ablations that compare to a "Point Annotation" baseline.
- VMBench also evaluates robustness to visual out-of-distribution settings, which I have not seen in other memory-based benchmarks.
Critical weaknesses
- MomentMem is missing a comparison to BPP [1], which uses the same keyframe selection and feeding to a robot policy mechanism.
- The baseline for the real-world experiment is just pi0.5, which is weak since it does not use memory.
- The argument about the limitations of textual memory summaries do not address using grounded (2D coordinate) subgoals as demonstrated in RoboMME [2].
[1] Max Sobol Mark et al. “BPP: Long-Context Robot Imitation Learning by Focusing on Key History Frames.” arXiv:2602.15010, 2026. https://arxiv.org/abs/2602.15010
[2] Yinpei Dai et al. “RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies.” arXiv:2603.04639, 2026. https://arxiv.org/abs/2603.04639
Questions for authors
- How does a BPP-style selection perform with the pi05 architecture and same demonstration data?
- Do the benefits of MomentMem hold with stronger memory-based baselines for the real-world experiments?
- How does a grounded subgoal baseline (see point 3 in the weaknesses section) perform on the VMBench tasks?
- What tasks best distinguish VMBench from RoboMME tasks's that also require visual moment memory (e.g. the tasks in the "Permanence" and "Reference" suite)?
3 comments
Response to Reviewer DR7Q (2/2)
Weakness 3. The argument about the limitations of textual memory summaries do not address using grounded (2D coordinate) subgoals as demonstrated in RoboMME.
We agree that 2D coordinates / grounded subgoals were not discussed in our original submission. We have now added this discussion and the experiment below to the paper (Appendix K).
However, from the conceptual perspective, 2D coordinates can just address parts of our designed tasks, such as the tasks involving positions, yet not the others, such as the tasks involving appearance. RoboMME designs the tasks that leverage 2D coordinates to refer individual objects and it is good enough for the robot to capture these referred objects. Unlike RoboMME, our benchmark further requires the robot to put the objects to exact locations, which is more challenging for a 2D coordinate approach.
Further, as requested by the reviewer, we perform additional experiments to test the 2D coordinate approach in our benchmark. As shown in the following table, 2D coordinate approach gets better performance on Spring Compression and Flash Beacon, yet its average success rate is lower than keyframe approach (ours). These results validate our conceptual arguments in the above paragraph. Solutions that can keep more visual details are needed to address visual moment memory problems.
| Memory | Explore Cube | Mine Sweeper | Hidden Pattern | Seal Checking | Spring Compression | Drawer Match | Flash Beacon | Scroll Discovery | Placement Imitation | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| Coordinates | 0% | 46% | 34% | 13% | 68% | 43% | 48% | 0% | 85% | 37.4% |
| Images (MomentMem) | 2% | 93% | 32% | 64% | 52% | 47% | 38% | 22% | 99% | 49.9% |
Implementation details: Following the grounded subgoals in RoboMME, we fine-tune Qwen3-VL-4B to predict 2D image coordinates as memory. A π0.5 executor is then trained on the same demonstrations to act on this memory. Since textures cannot be written as coordinates, for the Appearance tasks the VLM outputs the coordinates of the object to manipulate.
Question 1. How does a BPP-style selection perform with the pi05 architecture and same demonstration data?
See Weakness 1.
Question 2. Do the benefits of MomentMem hold with stronger memory-based baselines for the real-world experiments?
See Weakness 2.
Question 3. How does a grounded subgoal baseline (see point 3 in the weaknesses section) perform on the VMBench tasks?
See Weakness 3.
Question 4. What tasks best distinguish VMBench from RoboMME tasks's that also require visual moment memory (e.g. the tasks in the "Permanence" and "Reference" suite)?
The Appearance tasks distinguish VMBench from RoboMME most clearly, and most of the other tasks also differ from RoboMME in how they depend on positions.
In the RoboMME "Permanence" and "Reference" suites, the targets are mostly discrete objects, and memory serves to track their identity. In VMBench, the targets are more often continuous positions. For example, returning an object to its original place requires the exact target position.
Specifically, Table 1 groups the tasks into three categories.
- In the Appearance tasks, the robot must compare textures observed at different times, and the instruction never names the pattern. These tasks differ most from RoboMME, and they are also where our method is weakest.
- Explore Cube is the task most similar to RoboMME, since its target can be referred to as a discrete object (the cup that hides the cube).
- In the remaining tasks, the robot needs precise positions, which are continuous. We further evaluate a coordinate-based memory baseline and find that it underperforms MomentMem.
VMBench also offers two additional features. First, evidence segments are annotated, so keyframe-based memory can be evaluated on the selection itself. Second, every task has a visual OOD variant. VMBench complements RoboMME rather than replacing it.
If our responses have addressed your concerns, we hope that you could consider raising the score. Thank you for your time!
Dear Reviewer,
Following the AC’s reminder, please let us know if any concerns remain or further clarification would help. If our responses have addressed your concerns, we hope you might reconsider your assessment. Thank you for your time!
Response to Reviewer DR7Q (1/2)
MomentMem and BPP differ in how key events are specified. BPP specifies the key events of each task with a hand-written prompt. MomentMem learns them from interval annotations. The trade-off is between per-task prompt engineering and a small number of per-task interval annotations. We now cite BPP in the paper.
As for the comparison of performance, the conclusion is that our fine-tuned VLM is better than using a pretrained VLM in BPP-style. This is attributed to our training design (segment-level annotation and scheduled sampling), which teaches the model to understand what are the key moments for a task execution. More details can be found as follows.
We prompt a strong frozen VLM, GPT-6-Astra, to decide whether each frame shows a key event. Because of the cost, we did not run closed-loop evaluation. Instead, we evaluate keyframe selection on the same validation split and with the same metrics as Table 10 (20 demonstrations per task; Lines 956-957).
We identify two reasons for BPP's failures.
Short context. Each BPP query sees only two images. Spring Compression and Placement Imitation require the moment when an object reaches its farthest position, but two frames cannot show whether it has started to return.
Transition detection. BPP marks a keyframe at each "no"-to-"yes" transition. After a correct detection, any later error creates a new transition and thus a false keyframe. Such errors accumulate over long trajectories.
The frozen VLM captures only 69.5% of key segments, compared with 95.8% for MomentMem. It fails on Spring Compression and Placement Imitation (0%) and is weak on Scroll Discovery (15%). We also count how often it wrongly selects a non-keyframe: about 4.2% of non-keyframes are marked as keyframes (our memorizer is 0.14%). Per 100 queried frames, this gives about four false keyframes, more than the number of true key events in a trajectory. These results show that a stronger base model alone does not solve the selection problem. We have added this comparison to the paper (Section 5.7 and Appendix J.1).
We add EventVLA as a memory-based baseline in the real-world experiments. Each method is evaluated with 30 trials per task. We have added these results to Table 5 of the paper.
MomentMem outperforms EventVLA on both tasks. The failures of EventVLA differ between the two tasks.
Placement Imitation. The robot first watches a human move the cube to a target position and back, and then places the cube at that position. The key moment is therefore produced by the human, not by the robot. EventVLA predicts whether an upcoming frame is a keyframe from the hidden state of its own actions. As a result, it struggles to locate key moments driven by external agents. MomentMem selects keyframes directly from visual observations, so it does not depend on who causes the event.
Explore Cube. We observe no failure mode specific to memory. Most EventVLA failures are failed grasps, which reflect imprecise action execution rather than incorrect retrieval.