- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies memory for long-horizon robotic manipulation in a hierarchical VLM–VLA framework, where a high-level VLM predicts language subgoals and a low-level VLA executes them. The paper first presents an interesting diagnostic study showing that replacing predicted keyframes with ground-truth keyframes does not improve average success, while doubling the keyframe output frequency provides only modest gains. Motivated by this observation, the authors propose R2Mem with Structured Keyframe Memory (SKM), which pairs visual keyframes with structured textual records describing task-relevant object changes and completed operations. Persistent object identifiers are used to maintain entity correspondence over time, and the high-level VLM jointly updates memory and predicts subsequent subgoals. R2Mem substantially improves over MemER on RoboMME (50.72% to 70.62%) and also performs strongly on RoboMemArena, with additional evaluation on three real-world manipulation tasks.
Critical strengths
- Strong diagnostic motivation. The controlled comparison with ground-truth keyframes and increased keyframe frequency is a meaningful empirical contribution. Instead of assuming that memory failures arise from imperfect frame selection, the paper directly tests this hypothesis and shows that better visual evidence alone provides limited benefit. This makes the subsequent investigation of memory representation well motivated.
- Simple, interpretable, and empirically effective representation. SKM provides an intuitive way to maintain object history and execution progress through persistent object references, task-relevant event records, and associated visual evidence. The large improvements on RoboMME, especially on occlusion and object-swap tasks, are consistent with the proposed mechanism rather than appearing purely incidental.
Critical weaknesses
- The current experiments do not isolate the value of the proposed structured representation from the more general benefit of explicit semantic/task-progress memory. The key retrained ablation compares full R2Mem (70.62%) against a version without reasoning text (53.54%), which convincingly shows that explicit textual task-state information is useful, but does not establish that SKM's particular event structure, persistent object identifiers, or keyframe grounding are necessary. A stronger control would be a matched free-form semantic-memory baseline using the same backbone, visual inputs, and comparable memory/token budget. Without such a comparison, a simpler semantic summary of object history and completed operations could potentially explain a large fraction of the reported gain.
- The novelty claims appear broader than the demonstrated methodological contribution. The general ideas that visual observations leave semantic information implicit, that sparse snapshots may lose intervening temporal events, and that long-term memory can explicitly preserve semantic events or task progress have substantial precedent in long-video and embodied-memory research. Moreover, R2Mem operates entirely at the high-level VLM planner and does not use VLA-specific signals such as low-level action histories, policy states, proprioception, or action-conditioned representations; the low-level VLA remains frozen throughout the experiments. I therefore view the main novelty more narrowly as a task-conditioned, visually grounded, entity-consistent event representation for embodied subgoal planning, rather than a fundamentally new principle for VLA memory.
- The richer privileged supervision and the current ablation protocol make causal attribution difficult. SKM training targets are constructed using simulator object identities, visibility, image-space and 3D positions, discrete task states, and subgoal annotations. Using such privileged information during training is reasonable, but the comparison with visual-memory baselines therefore changes both the representation and the supervision available to the model. In addition, most component ablations are performed by removing records only at test time from a model trained with the complete representation; for example, removing subgoal-completion records reduces success from 70.62% to 24.31%. Such results demonstrate that the trained model relies on these records, but conflate information removal with train–test distribution shift. Matched-supervision controls and retrained ablations, especially for completion records and object identifiers, would provide much cleaner evidence for the proposed design.
- The real-world evaluation is promising but currently too underspecified to support the stated generalization claim. The paper reports three tasks with only 10 trials per task, but does not clearly specify the robot platform, low-level controller/VLA, whether the high-level R2Mem model is transferred directly from simulation or fine-tuned on real-world data, how real-world SKM supervision is obtained if fine-tuning is used, or whether the reported numbers correspond to fully closed-loop physical execution. These distinctions materially affect how strong the result is. If the evaluation is genuinely simulation-trained, zero-shot, closed-loop transfer, this would be a meaningful strength and should be stated explicitly; otherwise, the current real-world section is better viewed as a small proof-of-concept study.
Questions for authors
- Can the authors provide a matched free-form semantic-memory baseline that uses the same backbone, observations, supervision information, and comparable memory/token budget, but does not impose the proposed structured event representation? This would directly test whether SKM provides benefits beyond semantic memory in general.
- Given the very large performance drop after removing subgoal-completion records, can the authors report a simpler progress-only baseline (e.g., MemER + completed-subgoal history) and, ideally, retrained ablations without completion records and without persistent object identifiers?
- Training uses ground-truth previous memory entries, whereas deployment recursively conditions on predicted memory. Can the authors report memory-generation quality, such as object-ID consistency, completion-detection accuracy, or an oracle GT-memory result, to quantify error accumulation?
- Please clarify the complete real-world evaluation protocol, including the robot platform, low-level policy, high-level training/fine-tuning procedure, whether transfer from simulation is zero-shot, how memory labels are obtained if real-world fine-tuning is used, and whether evaluation consists of fully closed-loop physical rollouts.
3 comments
Response to Reviewer LLnM (Part 2/3)
Progress-Only Memory and Retrained Controls (W3, Q2)
… can the authors report a simpler progress-only baseline … and, ideally, retrained ablations without completion records and without persistent object identifiers?
We now retrain the ablated variants with the same memory format used at evaluation, addressing the distribution shift introduced by test-time removal. The variant without observation memory retains keyframes and completed-subgoal records, providing a progress-only textual memory control within our framework. We also retrain variants without completion records or object identifiers.
| Variant | SwingXtimes | BinFill | VideoUnmaskSwap | ButtonUnmaskSwap | AVG |
|---|---|---|---|---|---|
| R2Mem | 72.67 | 66.67 | 85.33 | 74.67 | 70.62 |
| w/o reasoning text | 54.00 | 52.67 | 27.33 | 20.00 | 53.54 |
| w/o action memory | 59.33 | 55.33 | 78.00 | 56.67 | 60.62 |
| w/o observation memory (progress-only text) | 64.67 | 62.00 | 32.67 | 44.00 | 64.31 |
| w/o object identifiers | 70.00 | 65.33 | 46.67 | 58.67 | 67.95 |
Evaluation uses 150 episodes per task; AVG covers all 13 RoboMME tasks. Removing reasoning text leaves visual memory; removing action or observation memory deletes completion or object-history records, respectively.
Progress records help, but do not account for the full benefit of SKM. The progress-only variant falls behind on both swap tasks, where object histories are needed to track hidden targets. Conversely, removing completion records reduces success on repetition and collection tasks. Removing identifiers particularly affects the swap tasks, supporting consistent links between events and objects. These retrained results demonstrate complementary contributions after each model has adapted to its available memory.
Simulator metadata supplies training labels, while deployment relies on predicted memory. Our simulation data construction also follows RoboMME's noise settings. The real-world protocol below demonstrates how corresponding labels are obtained without simulator metadata. We will include the retrained controls and clarify the supervision used by each variant.
Predicted Memory Quality and Oracle Control (Q3)
Can the authors report memory-generation quality, … or an oracle GT-memory result, to quantify error accumulation?
We assess generated SKM against GT records from the same rollout history. HunYuan-4 separately checks text and keyframes for task-critical errors or omissions: each receives full credit when it passes and zero otherwise, with 6 points for text and 4 for keyframes. Scores and task success are measured on the same episodes.
| Metric | Counting | Permanence | Reference | Imitation |
|---|---|---|---|---|
| Text score / 6 | 5.28 | 5.60 | 4.52 | 4.64 |
| Image score / 4 | 3.76 | 3.81 | 3.92 | 3.97 |
| SKM score / 10 | 9.04 | 9.41 | 8.44 | 8.61 |
| Success (%) | 77.78 | 86.67 | 53.67 | 61.67 |
Predicted SKM preserves task-relevant information across all four categories. Keyframe scores remain near full marks, and categories with higher text-memory quality also have higher task success. This correspondence supports memory quality as an indicator of downstream performance, consistent with the rationale of reasoning to remember.
To examine changes over time, we sort rollout chunks by timestamp and average their scores within successive sample percentile intervals.
| Sample Percentile Range | Mean SKM Score / 10 |
|---|---|
| 0–25% | 9.79 |
| 25–50% | 9.71 |
| 50–75% | 9.19 |
| 75–100% | 8.20 |
Quality decreases later in execution but remains high in the final interval. SKM therefore retains useful historical information despite some degradation. Efficient memory correction is a direction for future work.
For the oracle control, we replace predicted SKM with GT memory available up to the current query. The planner and low-level policy are unchanged, and the planner still predicts its own subgoals.
| Memory | Counting | Permanence | Reference | Imitation | AVG |
|---|---|---|---|---|---|
| Predicted SKM | 77.78 | 86.67 | 53.67 | 61.67 | 70.62 |
| GT SKM | 81.56 | 87.67 | 77.83 | 78.00 | 81.74 |
| (pp) | +3.78 | +1.00 | +24.16 | +16.33 | +11.12 |
Values are success rates (%); AVG covers all 13 RoboMME tasks.
The oracle intervention confirms that memory quality affects task completion with the rest of the system fixed. Predicted SKM performs close to the GT reference on Counting and Permanence, while Reference and Imitation benefit more from corrected memory. This separates the effect of memory quality from changes to the planner or executor. We will include these evaluations and the scoring protocol in the revision.
Response to Reviewer LLnM (Part 3/3)
Real-World Evaluation Protocol (W4, Q4)
Please clarify the complete real-world evaluation protocol, … and whether evaluation consists of fully closed-loop physical rollouts.
Our real-world experiments follow RoboMME's official task design and evaluation protocol. They use real-world fine-tuning, rather than zero-shot transfer from simulation.
Platform and training data. We use a UR5e robot with wrist and shoulder cameras and as the low-level policy. We collect 50 demonstrations per task and construct training labels primarily through manual annotation with VLM assistance, covering keyframes, object IDs, event descriptions, and completed subgoals.
Fine-tuning and evaluation. We continue fine-tuning the MemER and R2Mem Qwen3-VL-4B planners from their respective simulation-trained checkpoints on these demonstrations. Both learn keyframe selection and subgoal prediction. The low-level policy is initialized from its official pretrained checkpoint and fine-tuned on real-world data. Each method is evaluated over 10 closed-loop physical trials per task. We will include these protocol details in the revised manuscript.
Response to Reviewer LLnM (Part 1/3)
We thank the reviewer for recognizing the diagnostic study and the effectiveness of SKM. We address the representation controls, supervision, memory errors, and real-world protocol below.
Structured vs. Free-Form Memory (W1, Q1)
We have added two free-form memory baselines to examine whether unstructured text can provide the same benefit. Both use Qwen3-VL-4B at evaluation and learn keyframe selection and subgoal prediction. One uses self-generated text without direct text supervision; the other receives offline GPT-generated text supervision. The names in parentheses identify the models used to generate the free-form memory text.
Success rates are measured over 150 episodes per task; AVG covers all 13 RoboMME tasks.
SKM outperforms both free-form alternatives, including the variant with GPT-generated text supervision. The largest gains among the displayed tasks occur in hidden-object tracking through swaps, where the planner must retain and connect successive changes. These comparisons support the value of explicitly organizing task-relevant events beyond providing free-form textual memory. We will add these baselines and their training details to the revision.
Scope of the Contribution (W2)
Our contribution is SKM, which organizes historical reasoning outcomes around task-relevant key events for reuse in subgoal planning. Figure 1 shows limited gains from improving keyframe selection or density alone. Our analysis identifies two remaining issues: conclusions remain implicit in images, and frame selection can discard intervening changes. SKM pairs keyframes with explicit key event records to preserve this information. Its event types apply across tasks, while the task determines which object changes and completed operations to record, preserving object histories and execution progress beyond individual snapshots.
R2Mem enables an existing VLM to jointly learn memory construction and subgoal prediction through its standard image–text interface. With a common low-level VLA, the retrained ablations demonstrate the complementary value of object histories and execution progress, while comparisons with free-form memory support their explicit organization. Together, these results support SKM's design: retaining and organizing task-relevant historical information improves subsequent planning and task success.