acceptodds
Withdrawn from ICLR 2027

MomentMem: Memorizing Key Moments for Robot Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Many manipulation tasks require recalling visual evidence that has left the robot's view. Yet most vision-language-action (VLA) policies see only the current observation or a short history. Such tasks call for visual moment memory, which captures the scene at the moments that matter. To evaluate this capability, we introduce VMBench, whose tasks require recalling patterns, locations, or both. We then propose MomentMem, which instantiates this memory with keyframes. A VLM selects keyframes, and a VLA executor predicts actions from them and the current observation. Both are trained with segment-level annotations and representative-frame sampling. MomentMem achieves state-of-the-art results on VMBench and RMBench, and improves success on a real robot. Its memory also transfers zero-shot to unseen visual appearances and adapts to a held-out task with few annotations.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Memory for Robotic Manipulation, Vision-language Models, Generalization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DR7QOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors present a hierarchical framework, MomentMem, for memory-based tasks in robotics, along with a benchmark, VMBench, to evaluate their method and other memory policies on tasks that require visual moment memory. These are tasks where the information required to solve them have left the current observation. MomentMem trains a VLM to select relevant keyframes using segment-level supervision, and then feeds these keyframes to a VLA policy, which act as a low-level executor.

Critical strengths
  1. VMBench is well-motivated and introduces an interesting experiment regarding generalization for memory-based tasks. Most previous works focus on just performance for memory-based tasks for the tasks the train, but Section 5.4 shows transfer results for an unseen task with few annotations, illustrating the importance of memory-based pretraining for robotics tasks.
  2. Training with multiple valid keyframes within a segment is well-motivated with ablations that compare to a "Point Annotation" baseline.
  3. VMBench also evaluates robustness to visual out-of-distribution settings, which I have not seen in other memory-based benchmarks.
Critical weaknesses
  1. MomentMem is missing a comparison to BPP [1], which uses the same keyframe selection and feeding to a robot policy mechanism.
  2. The baseline for the real-world experiment is just pi0.5, which is weak since it does not use memory.
  3. The argument about the limitations of textual memory summaries do not address using grounded (2D coordinate) subgoals as demonstrated in RoboMME [2].

[1] Max Sobol Mark et al. “BPP: Long-Context Robot Imitation Learning by Focusing on Key History Frames.” arXiv:2602.15010, 2026. https://arxiv.org/abs/2602.15010

[2] Yinpei Dai et al. “RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies.” arXiv:2603.04639, 2026. https://arxiv.org/abs/2603.04639

Questions for authors
  1. How does a BPP-style selection perform with the pi05 architecture and same demonstration data?
  2. Do the benefits of MomentMem hold with stronger memory-based baselines for the real-world experiments?
  3. How does a grounded subgoal baseline (see point 3 in the weaknesses section) perform on the VMBench tasks?
  4. What tasks best distinguish VMBench from RoboMME tasks's that also require visual moment memory (e.g. the tasks in the "Permanence" and "Reference" suite)?
3 comments
AuthorsOpenReview ↗

Response to Reviewer DR7Q (1/2)

Weakness 1. MomentMem is missing a comparison to BPP, which uses the same keyframe selection and feeding to a robot policy mechanism.

MomentMem and BPP differ in how key events are specified. BPP specifies the key events of each task with a hand-written prompt. MomentMem learns them from interval annotations. The trade-off is between per-task prompt engineering and a small number of per-task interval annotations. We now cite BPP in the paper.

As for the comparison of performance, the conclusion is that our fine-tuned VLM is better than using a pretrained VLM in BPP-style. This is attributed to our training design (segment-level annotation and scheduled sampling), which teaches the model to understand what are the key moments for a task execution. More details can be found as follows.

TaskSegment hit rate (GPT-6-Astra / Ours)Segment accuracy (GPT-6-Astra / Ours)Non-keyframe accuracy (GPT-6-Astra / Ours)
Explore Cube100 / 98.3100 / 96.70 / 90.0
Mine Sweeper72.5 / 10067.5 / 1000 / 100
Hidden Pattern100 / 10085 / 10055 / 100
Seal Checking100 / 97.5100 / 95.010 / 80.0
Spring Compression0 / 1000 / 90.035 / 100
Drawer Match100 / 100100 / 1000 / 100
Flash Beacon100 / 97.5100 / 87.5100 / 80.0
Scroll Discovery15 / 86.715 / 83.390 / 65.0
Placement Imitation0 / 85.00 / 82.50 / 45.0
Pooled69.5 / 95.867.5 / 93.032.2 / 84.4

We prompt a strong frozen VLM, GPT-6-Astra, to decide whether each frame shows a key event. Because of the cost, we did not run closed-loop evaluation. Instead, we evaluate keyframe selection on the same validation split and with the same metrics as Table 10 (20 demonstrations per task; Lines 956-957).

We identify two reasons for BPP's failures.

  1. Short context. Each BPP query sees only two images. Spring Compression and Placement Imitation require the moment when an object reaches its farthest position, but two frames cannot show whether it has started to return.

  2. Transition detection. BPP marks a keyframe at each "no"-to-"yes" transition. After a correct detection, any later error creates a new transition and thus a false keyframe. Such errors accumulate over long trajectories.

The frozen VLM captures only 69.5% of key segments, compared with 95.8% for MomentMem. It fails on Spring Compression and Placement Imitation (0%) and is weak on Scroll Discovery (15%). We also count how often it wrongly selects a non-keyframe: about 4.2% of non-keyframes are marked as keyframes (our memorizer is 0.14%). Per 100 queried frames, this gives about four false keyframes, more than the number of true key events in a trajectory. These results show that a stronger base model alone does not solve the selection problem. We have added this comparison to the paper (Section 5.7 and Appendix J.1).

Weakness 2. The baseline for the real-world experiment is just pi0.5, which is weak since it does not use memory.

We add EventVLA as a memory-based baseline in the real-world experiments. Each method is evaluated with 30 trials per task. We have added these results to Table 5 of the paper.

π0.5EventVLAMomentMem
Placement Imitation3 / 306 / 3014 / 30
Explore Cube11 / 3010 / 3017 / 30

MomentMem outperforms EventVLA on both tasks. The failures of EventVLA differ between the two tasks.

  1. Placement Imitation. The robot first watches a human move the cube to a target position and back, and then places the cube at that position. The key moment is therefore produced by the human, not by the robot. EventVLA predicts whether an upcoming frame is a keyframe from the hidden state of its own actions. As a result, it struggles to locate key moments driven by external agents. MomentMem selects keyframes directly from visual observations, so it does not depend on who causes the event.

  2. Explore Cube. We observe no failure mode specific to memory. Most EventVLA failures are failed grasps, which reflect imprecise action execution rather than incorrect retrieval.

AuthorsOpenReview ↗

Response to Reviewer DR7Q (2/2)

Weakness 3. The argument about the limitations of textual memory summaries do not address using grounded (2D coordinate) subgoals as demonstrated in RoboMME.

We agree that 2D coordinates / grounded subgoals were not discussed in our original submission. We have now added this discussion and the experiment below to the paper (Appendix K).

However, from the conceptual perspective, 2D coordinates can just address parts of our designed tasks, such as the tasks involving positions, yet not the others, such as the tasks involving appearance. RoboMME designs the tasks that leverage 2D coordinates to refer individual objects and it is good enough for the robot to capture these referred objects. Unlike RoboMME, our benchmark further requires the robot to put the objects to exact locations, which is more challenging for a 2D coordinate approach.

Further, as requested by the reviewer, we perform additional experiments to test the 2D coordinate approach in our benchmark. As shown in the following table, 2D coordinate approach gets better performance on Spring Compression and Flash Beacon, yet its average success rate is lower than keyframe approach (ours). These results validate our conceptual arguments in the above paragraph. Solutions that can keep more visual details are needed to address visual moment memory problems.

MemoryExplore CubeMine SweeperHidden PatternSeal CheckingSpring CompressionDrawer MatchFlash BeaconScroll DiscoveryPlacement ImitationAvg.
Coordinates0%46%34%13%68%43%48%0%85%37.4%
Images (MomentMem)2%93%32%64%52%47%38%22%99%49.9%

Implementation details: Following the grounded subgoals in RoboMME, we fine-tune Qwen3-VL-4B to predict 2D image coordinates as memory. A π0.5 executor is then trained on the same demonstrations to act on this memory. Since textures cannot be written as coordinates, for the Appearance tasks the VLM outputs the coordinates of the object to manipulate.

Question 1. How does a BPP-style selection perform with the pi05 architecture and same demonstration data?

See Weakness 1.

Question 2. Do the benefits of MomentMem hold with stronger memory-based baselines for the real-world experiments?

See Weakness 2.

Question 3. How does a grounded subgoal baseline (see point 3 in the weaknesses section) perform on the VMBench tasks?

See Weakness 3.

Question 4. What tasks best distinguish VMBench from RoboMME tasks's that also require visual moment memory (e.g. the tasks in the "Permanence" and "Reference" suite)?

The Appearance tasks distinguish VMBench from RoboMME most clearly, and most of the other tasks also differ from RoboMME in how they depend on positions.

In the RoboMME "Permanence" and "Reference" suites, the targets are mostly discrete objects, and memory serves to track their identity. In VMBench, the targets are more often continuous positions. For example, returning an object to its original place requires the exact target position.

Specifically, Table 1 groups the tasks into three categories.

  1. In the Appearance tasks, the robot must compare textures observed at different times, and the instruction never names the pattern. These tasks differ most from RoboMME, and they are also where our method is weakest.
  2. Explore Cube is the task most similar to RoboMME, since its target can be referred to as a discrete object (the cup that hides the cube).
  3. In the remaining tasks, the robot needs precise positions, which are continuous. We further evaluate a coordinate-based memory baseline and find that it underperforms MomentMem.

VMBench also offers two additional features. First, evidence segments are annotated, so keyframe-based memory can be evaluated on the selection itself. Second, every task has a visual OOD variant. VMBench complements RoboMME rather than replacing it.


If our responses have addressed your concerns, we hope that you could consider raising the score. Thank you for your time!

Dear Reviewer,

Following the AC’s reminder, please let us know if any concerns remain or further clarification would help. If our responses have addressed your concerns, we hope you might reconsider your assessment. Thank you for your time!

Reviewer_VZhmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a timely problem: current VLA policies struggle to remember historical key moments, and benchmarks for this problem are also scarce. It first introduces a novel benchmark VMBench for analysing difficult memory problems, and it also proposes a novel method of historical key moment memorising for VLA policies. Instead of directly using text or fixed keyframes, it uses a separate keyframe identifier to memorise the keyframe. Experiments cover VMBench, RMBench, real-world, which show good performance.

Critical strengths
  1. This paper proposes a useful method to memorise task-relevant visual evidence for VLA policies. The motivation is clear, and the paper is well-structured.
  2. Using segment-level supervision rather than specific keyframes is a good trick. Also, anchor frame sampling can provide multiple valid selections and even recover from missed selections.
  3. Among VMBench, MomentMem achieves significant improvement compared with other baselines. Specifically, the gains on appearance-location tasks are promising.
Critical weaknesses
  1. The major performance gain is not fully analysed. The system consists of Qwen3-VL-4B keyframe identifier, segment-level supervision, and a memory-conditioned policy. The results don’t provide enough information about whether this advantage comes from the proposed selection strategy or from retaining visual observations rather than compressed representations. Adding experiments on using the same policy with learned keyframe selection would strengthen the paper.
  2. The evidence for fine-grained visual memory is limited. MomentMem doesn’t outperform the strongest baselines on the three Appearance tasks. Although this limitation is acknowledged, it constrains the conclusion about visual memory capabilities. A deeper analysis of the failure cases could help clarify.
Questions for authors
  1. Could you specify the memory supervision and computational resources used by each baseline?
  2. Could you provide controlled experiments that keep the executor, memory format, and memory budget fixed while varying the keyframe selection method?
  3. How does anchor-frame sampling compare with selecting a fixed representative frame from each annotated segment?
4 comments
AuthorsOpenReview ↗

Response to Reviewer VZhm (1/2)

Weakness 1. The major performance gain is not fully analysed. The system consists of Qwen3-VL-4B keyframe identifier, segment-level supervision, and a memory-conditioned policy. The results don’t provide enough information about whether this advantage comes from the proposed selection strategy or from retaining visual observations rather than compressed representations. Adding experiments on using the same policy with learned keyframe selection would strengthen the paper.

Both factors contribute, and the selection strategy matters more. We test this with three variants, each changing one component of MomentMem. In each variant, the executor uses the same π0.5 architecture and is retrained on the same demonstrations with the new memory.

1. Memory representation: keyframes vs. coordinates. We keep the Qwen3-VL-4B memorizer but fine-tune it to output 2D image coordinates instead of keyframes, following the grounded subgoals in RoboMME [1]. Since textures cannot be written as coordinates, for the Appearance tasks the VLM outputs the coordinates of the object to manipulate.

MemoryExplore CubeMine SweeperHidden PatternSeal CheckingSpring CompressionDrawer MatchFlash BeaconScroll DiscoveryPlacement ImitationAvg.
Coordinates0%46%34%13%68%43%48%0%85%37.4%
Images (MomentMem)2%93%32%64%52%47%38%22%99%49.9%

Coordinates work well when only a position is needed, but fall behind when each position must be paired with a color. Images keep both.

2. Selector training: fine-tuned VLM vs. frozen VLM. Following BPP [2], we prompt a frozen VLM, GPT-6-Astra, to detect key events online. Because of the cost of querying it, we evaluate keyframe selection instead of closed-loop success (full table in our response to Reviewer DR7Q, Weakness 1). The frozen VLM captures 69.5% of key segments, compared with 95.8% for MomentMem. It misses all key segments on Spring Compression and Placement Imitation, and selects more false keyframes. So a stronger base model alone does not solve selection; our training is needed.

3. Selector architecture: VLM vs. RNN. Due to limited resources and rebuttal time, we only test a basic alternative: a recurrent selector (CLIP features with a two-layer GRU), trained with the same segment annotations, anchor sampling, and scheduled sampling. The selected frames are still given to the executor as raw images.

SettingSelectorExplore CubeMine SweeperHidden PatternSeal CheckingSpring CompressionDrawer MatchFlash BeaconScroll DiscoveryPlacement ImitationAvg.
StandardRNN0%3%32%2%48%42%6%25%6%18.2%
StandardMomentMem2%93%32%64%52%47%38%22%99%49.9%
UnseenRNN0%0%25%0%36%27%0%26%1%12.8%
UnseenMomentMem12%79%35%55%79%51%59%28%99%55.2%

Success drops sharply with the RNN selector, most on tasks where the evidence appears only briefly, such as Mine Sweeper and Seal Checking. Once the selector misses that moment, the memory holds nothing useful. On the Appearance tasks, the two selectors are close; as discussed in Weakness 2, the bottleneck there is matching textures in the executor. Under unseen appearances, the gap becomes larger.

In summary, keeping image memory but using a weaker selector (18.2%) hurts more than keeping the memorizer but using coordinates (37.4%). So raw images alone are not enough; the gain relies on selecting the right frames. We have added all three variants to the paper (Section 5.7, Appendices J and K).

[1] Yinpei Dai et al. "RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies." arXiv:2603.04639, 2026. https://arxiv.org/abs/2603.04639

[2] Max Sobol Mark et al. "BPP: Long-Context Robot Imitation Learning by Focusing on Key History Frames." arXiv:2602.15010, 2026. https://arxiv.org/abs/2602.15010

AuthorsOpenReview ↗

Response to Reviewer VZhm (2/2)

Weakness 2. The evidence for fine-grained visual memory is limited. MomentMem doesn’t outperform the strongest baselines on the three Appearance tasks. Although this limitation is acknowledged, it constrains the conclusion about visual memory capabilities. A deeper analysis of the failure cases could help clarify.

We organize in-depth analysis on task Hidden Pattern as follows. The short conclusion is that fine-grained visual information is already captured, understood, yet not well utilized by the executor:

  1. The keyframes already capture the key information. As in Line 1048, recall (segment hit rate) reaches 100%.

  2. The executor can read and understand the texture details from keyframes. As in Lines 380-381, we did a probing. A linear probe reads “what the texture is” from the executor's features with 99% accuracy.

  3. The executor fails to match the texture details from keyframes. As in Lines 381-382, we did a probing on pattern matching across images. The probe finds the matching box only 40% of the time, just slightly higher than 33% (random).

So raw-image memory does keep the fine details, as we argued. The bottleneck is the next step: the executor cannot compare two textures. We now state this in the Limitations (Lines 537-539).

Question 1. Could you specify the memory supervision and computational resources used by each baseline?

All methods use the same 100 demonstrations per task and are trained jointly on the nine tasks (Lines 351-352). The table below lists the memory supervision and training resources of each method.

MethodMemory supervisionComputational resources
π0.5No memory labels.8 × NVIDIA A800, ~2 days
Mem-0Textual summaries as memory. We write textual summaries for Mem-0.VLM: 8 × NVIDIA A800, ~2 days, VLA: 8 × NVIDIA A800, ~3 days
EventVLAOur segment-level annotations. A keyframe sampled from each segment serves as the label.8 × NVIDIA A800, ~4 days
NativeMEMNo memory labels; trained end to end.8 × NVIDIA A800, ~3 days
MomentMemOur segment-level annotations. A keyframe sampled from each segment serves as the anchor.VLM: 8 × NVIDIA A800, ~2 days, VLA: 8 × NVIDIA A800, ~4 days

Question 2. Could you provide controlled experiments that keep the executor, memory format, and memory budget fixed while varying the keyframe selection method?

See Weakness 1.

Question 3. How does anchor-frame sampling compare with selecting a fixed representative frame from each annotated segment?

"Fixed representative frame" cannot work well. This is the Point Annotation ablation in Table 6. It labels one fixed frame per segment as positive, following prior work. Only the memorizer labels change. The average success drops from 49.9% to 6.9% (Lines 472-473). The drop is large on Mine Sweeper (93% → 0%), Seal Checking (64% → 0%), and Placement Imitation (99% → 5%).

The reason is that evidence often stays visible for several frames. A fixed frame turns its neighbors into negatives, even though they show the same evidence (Lines 297-298). Anchor sampling picks a new anchor each epoch, and earlier frames in the segment are left out of the loss (Lines 289-291). So the memorizer learns that any frame showing the evidence is a valid choice.


If our responses have addressed your concerns, we hope that you could consider raising the score. Thank you for your time!

Dear Reviewer,

Following the AC’s reminder, please let us know if any concerns remain or further clarification would help. If our responses have addressed your concerns, we hope you might reconsider your assessment. Thank you for your time!

Thank you for the detailed response and additional experiments. The coordinate-memory and RNN-selector comparisons address my concern about the sources of the performance gains, while the probing results help clarify the limitations on Appearance tasks.

The response strengthens my confidence in the paper, and I maintain my weak-accept recommendation.

Reviewer_mM23OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new method in managing visual memory in long-horizon robotic manipulation, and created a benchmark in which robots have to recall previously observed information that is no longer visible. The benchmark consists of 3 suite and 9 memory-dependent tasks in simulation. The authors proposed MomentMem, a two-module framework consisting of a keyframe identifier, fine-tuned from a VLM, and a Memory conditioned Executor, finetuned from a VLA. The key innovations are segment-level keyframe annotations and scheduled sampling which improve memory selection and reduce training-inference discrepancies. MomentMem substantially outperforms the other baselines, provided both simulation experiments and real robot experiments and demonstrated cross-scene and cross task memory selection transfer.

Critical strengths

Here are the strength of the paper in my opinion:

  1. the paper proposed a meaningful research problem and well-targeted benchmark. MomentMem addresses visual memory in partially observable robotic tasks. VMBench evaluates Spatial, Appearance, and combined memory requirements, with visual OOD settings to assess memory capabilities beyond general long-horizon task completion.

  2. the paper has a clear framework and effective training strategies. MomentMem adopts a modular design that separates keyframe selections from action execution. Ablation results validate their effectiveness.

  3. The paper conducts insightful analysis of memory mechanisms. The authors evaluate keyframe selection quality and use counterfactual interventions to examine how memory influences action decisions, helping distinguish failures in memory selection, utilization, and physical execution.

Critical weaknesses

Thank you authors for your awesome work. i have a few concerns hope that you may address.

  1. The strength of VMBench. The paper argues that the main reason for designing this benchmark is to capture memory that cannot be summarized as text or other structured output. However, many tasks inside VMBench can be summarised as structured output. For example, the location where the different beacon blinks can be transformed into image coordinates. And for the task on appearances, all three tasks share the same set of artificially designed appearances, which may limit the diversity and representativeness of real-life tasks.

  2. Limited evidence of effectiveness in appearance reasoning. the paper argues that raw visual memroy better preserves fine-grained details than text-based or compreseed representations. but MomentMem underperforms existing baselines on all three appearance tasks. This suggests that the executor struggles to utilize fine-grained visual information which contradicts the original motivation.

Questions for authors

Thank you authors for your awesome work. i have a few questions hope that you may answer.

  1. Do the tasks in VMBench genuinely require retaining raw visual information, or can they be solved using structured text, object attributes, or precise spatial coordinates instead?

  2. Could the authors evaluate zero-shot or few-shot prompting with a strong frozen pretrained VLM for keyframe selection? this can better clarify the improvement from proposed training strategies rather than the base model's inherent visual understanding.

  3. How robust is the executor to imperfect memory stacks during inference?

  4. Why does MomentMem underperform existing baselines on all three appearances tasks, despite its emphasis on preserving fine-grained visual details?

3 comments
AuthorsOpenReview ↗

Response to Reviewer mM23 (1/2)

Weakness 1. The strength of VMBench. The paper argues that the main reason for designing this benchmark is to capture memory that cannot be summarized as text or other structured output. However, many tasks inside VMBench can be summarised as structured output. For example, the location where the different beacon blinks can be transformed into image coordinates. And for the task on appearances, all three tasks share the same set of artificially designed appearances, which may limit the diversity and representativeness of real-life tasks.

We would like to clarify that we did not claim that the information "cannot be written as text or coordinates". Instead, while previous benchmarks only consider the information well represented by the text, we build a new benchmark that requires visual moment memory: recalling precise, briefly visible evidence for later actions. This differs from task progress, such as "I have pressed the button twice" in Figure 1 of main paper, which a short text summary captures well.

The above is the motivation of our benchmark. As for the modeling to address visual moment memory, text or coordinates are feasible solutions, and our keyframe solution is also an alternative. However, from both the theoretical and empirical perspectives, keyframes would work better. As requested by the reviewer, we perform additional experiments that replace keyframes with coordinates. Please refer to our response to reviewer DR7Q Weakness 3. The conclusion validates our assumption above, text and coordinates are feasible yet not as effective as keyframes (or other visual modeling in the future).

Weakness 2. Limited evidence of effectiveness in appearance reasoning. the paper argues that raw visual memory better preserves fine-grained details than text-based or compressed representations. but MomentMem underperforms existing baselines on all three appearance tasks. This suggests that the executor struggles to utilize fine-grained visual information which contradicts the original motivation.

Our motivation is to highlight the importance of visual moment memory. No method performs well on appearance-related tasks, which exactly validates that our benchmark design requires more attention from the community.

As for modeling, to better help the reviewer to understand these tasks, we organize in-depth analysis on task Hidden Pattern as follows. The short conclusion is that fine-grained visual information is already captured, understood, yet not well utilized by the executor:

  1. The keyframes already capture the key information. As in Line 1048, recall (segment hit rate) reaches 100%.
  2. The executor can read and understand the texture details from keyframes. As in Lines 380-381, we did a probing. A linear probe reads “what the texture is” from the executor's features with 99% accuracy.
  3. The executor fails to match the texture details from keyframes. As in Lines 381-382, we did a probing on pattern matching across images. The probe finds the matching box only 40% of the time, just slightly higher than 33% (random).

So raw-image memory does keep the fine details, as we argued. The bottleneck is the next step: the executor cannot compare two textures. We now state this in the Limitations (Lines 537-539).

AuthorsOpenReview ↗

Response to Reviewer mM23 (2/2)

Question 1. Do the tasks in VMBench genuinely require retaining raw visual information, or can they be solved using structured text, object attributes, or precise spatial coordinates instead?

See Weakness 1.

Question 2. Could the authors evaluate zero-shot or few-shot prompting with a strong frozen pretrained VLM for keyframe selection? This can better clarify the improvement from proposed training strategies rather than the base model's inherent visual understanding.

As requested by the reviewer, we perform additional experiments by using frozen pretrained VLM. The conclusion is that our fine-tuned VLM is better than the frozen ones at selecting the keyframes. This is attributed to our training design (segment-level annotation and scheduled sampling), which teaches the model to understand what are the key moments for a task execution. More details can be found as follows.

TaskSegment hit rate (GPT-6-Astra / Ours)Segment accuracy (GPT-6-Astra / Ours)Non-keyframe accuracy (GPT-6-Astra / Ours)
Explore Cube100 / 98.3100 / 96.70 / 90.0
Mine Sweeper72.5 / 10067.5 / 1000 / 100
Hidden Pattern100 / 10085 / 10055 / 100
Seal Checking100 / 97.5100 / 95.010 / 80.0
Spring Compression0 / 1000 / 90.035 / 100
Drawer Match100 / 100100 / 1000 / 100
Flash Beacon100 / 97.5100 / 87.5100 / 80.0
Scroll Discovery15 / 86.715 / 83.390 / 65.0
Placement Imitation0 / 85.00 / 82.50 / 45.0
Pooled69.5 / 95.867.5 / 93.032.2 / 84.4

Our implementation follows BPP [1]. We prompt a strong frozen VLM, GPT-6-Astra, to decide whether each frame shows a key event. Because of the cost, we did not run closed-loop evaluation. Instead, we evaluate keyframe selection on the same validation split and with the same metrics as Table 10 (20 demonstrations per task; Lines 956-957).

We identify two reasons for BPP's failures.

  1. Short context. Each BPP query sees only two images. Spring Compression and Placement Imitation require the moment when an object reaches its farthest position, but two frames cannot show whether it has started to return.

  2. Transition detection. BPP marks a keyframe at each "no"-to-"yes" transition. After a correct detection, any later error creates a new transition and thus a false keyframe. Such errors accumulate over long trajectories.

The frozen VLM captures only 69.5% of key segments, compared with 95.8% for MomentMem. It fails on Spring Compression and Placement Imitation (0%) and is weak on Scroll Discovery (15%). We also count how often it wrongly selects a non-keyframe: about 4.2% of non-keyframes are marked as keyframes (our memorizer is 0.14%). Per 100 queried frames, this gives about four false keyframes, more than the number of true key events in a trajectory. These results show that a stronger base model alone does not solve the selection problem. We have added this comparison to the paper (Section 5.7 and Appendix J.1).

[1] Max Sobol Mark et al. "BPP: Long-Context Robot Imitation Learning by Focusing on Key History Frames." arXiv:2602.15010, 2026. https://arxiv.org/abs/2602.15010

Question 3. How robust is the executor to imperfect memory stacks during inference?

The executor is fairly robust to small selection errors. This is by design: it is trained on keyframes predicted by the memorizer, which may be early, late, missing, or extra (Lines 323-324). The paper shows two cases.

  1. Early frames. Under unseen appearances, Spring Compression has a strict hit rate of only 10.0%. Most misses are just 0.10 s early and still show the right position (Lines 1080-1083). Success is still 79%.

  2. Extra frames. On Placement Imitation, about half of the demonstrations contain extra frames (Line 1053). Success is still 99%.

However, the executor cannot fix wrong content. When we swap the stored colors in Seal Checking, all seven modified scenes follow the new colors.

Question 4. Why does MomentMem underperform existing baselines on all three appearances tasks, despite its emphasis on preserving fine-grained visual details?

See Weakness 2.


If our responses have addressed your concerns, we hope that you could consider raising the score. Thank you for your time!

Dear Reviewer,

Following the AC’s reminder, please let us know if any concerns remain or further clarification would help. If our responses have addressed your concerns, we hope you might reconsider your assessment. Thank you for your time!

Comments on the paper

1 comment
Area_Chair_zqUgOpenReview ↗

Reminder: Author–Reviewer Discussion Deadline Approaching

Dear Reviewers and Authors,

Thank you to the reviewers for your time, effort, and dedication in reviewing this paper. This AC also appreciates the authors' detailed rebuttal, which addresses the reviewers' comments and includes additional clarifications and experimental results.

To the reviewers: As the author–reviewer discussion deadline approaches, please carefully review the authors' responses and assess whether they adequately address your concerns, if you haven't already. If any concerns have not been fully addressed, please let the authors know what further clarification or evidence would be helpful.

Thank you again for your valuable contributions and constructive discussion.

Best,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.