acceptodds
Withdrawn from ICLR 2027

Reasoning to Remember: Structured Keyframe Memory for Long-Horizon Robotic Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-horizon robotic manipulation requires historical information to guide decisions beyond what current observations reveal. Keyframe-based memory provides visual context for a high-level vision-language model (VLM), which predicts subgoals for a low-level vision-language-action (VLA) policy. However, whether selecting better keyframes or retaining more frames effectively improves planning remains unclear. To investigate this, we replace predicted keyframes with ground-truth keyframes and separately double the keyframe output frequency. Ground-truth keyframes do not improve average success, while increased frequency yields only modest gains. Further analysis identifies two limitations that visual selection alone does not directly address. First, task-relevant conclusions remain implicit in visual content, requiring subsequent planning to reconstruct them even when the original supporting context has changed or become unavailable. Second, keyframe selection can discard intervening changes and interactions, leaving distinct histories indistinguishable from the retained snapshots. To address these limitations, we propose R2Mem (Reasoning to Remember), built around Structured Keyframe Memory (SKM). SKM pairs keyframes with reasoning text organized around task-relevant key events. These events specify which object changes and completed operations to record beyond what individual snapshots convey. The records form observation memory and action memory, explicitly preserving object histories and execution progress, respectively. The high-level VLM jointly constructs SKM and predicts subgoals using these explicit records. Experiments on RoboMME, RoboMemArena, and real-world manipulation tasks show that R2Mem improves average task success over memory-augmented baselines, highlighting the value of explicitly recording task-relevant events for subsequent subgoal planning. The code will be released.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLM, VLA, Memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LLnMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies memory for long-horizon robotic manipulation in a hierarchical VLM–VLA framework, where a high-level VLM predicts language subgoals and a low-level VLA executes them. The paper first presents an interesting diagnostic study showing that replacing predicted keyframes with ground-truth keyframes does not improve average success, while doubling the keyframe output frequency provides only modest gains. Motivated by this observation, the authors propose R2Mem with Structured Keyframe Memory (SKM), which pairs visual keyframes with structured textual records describing task-relevant object changes and completed operations. Persistent object identifiers are used to maintain entity correspondence over time, and the high-level VLM jointly updates memory and predicts subsequent subgoals. R2Mem substantially improves over MemER on RoboMME (50.72% to 70.62%) and also performs strongly on RoboMemArena, with additional evaluation on three real-world manipulation tasks.

Critical strengths
  1. Strong diagnostic motivation. The controlled comparison with ground-truth keyframes and increased keyframe frequency is a meaningful empirical contribution. Instead of assuming that memory failures arise from imperfect frame selection, the paper directly tests this hypothesis and shows that better visual evidence alone provides limited benefit. This makes the subsequent investigation of memory representation well motivated.
  2. Simple, interpretable, and empirically effective representation. SKM provides an intuitive way to maintain object history and execution progress through persistent object references, task-relevant event records, and associated visual evidence. The large improvements on RoboMME, especially on occlusion and object-swap tasks, are consistent with the proposed mechanism rather than appearing purely incidental.
Critical weaknesses
  1. The current experiments do not isolate the value of the proposed structured representation from the more general benefit of explicit semantic/task-progress memory. The key retrained ablation compares full R2Mem (70.62%) against a version without reasoning text (53.54%), which convincingly shows that explicit textual task-state information is useful, but does not establish that SKM's particular event structure, persistent object identifiers, or keyframe grounding are necessary. A stronger control would be a matched free-form semantic-memory baseline using the same backbone, visual inputs, and comparable memory/token budget. Without such a comparison, a simpler semantic summary of object history and completed operations could potentially explain a large fraction of the reported gain.
  2. The novelty claims appear broader than the demonstrated methodological contribution. The general ideas that visual observations leave semantic information implicit, that sparse snapshots may lose intervening temporal events, and that long-term memory can explicitly preserve semantic events or task progress have substantial precedent in long-video and embodied-memory research. Moreover, R2Mem operates entirely at the high-level VLM planner and does not use VLA-specific signals such as low-level action histories, policy states, proprioception, or action-conditioned representations; the low-level VLA remains frozen throughout the experiments. I therefore view the main novelty more narrowly as a task-conditioned, visually grounded, entity-consistent event representation for embodied subgoal planning, rather than a fundamentally new principle for VLA memory.
  3. The richer privileged supervision and the current ablation protocol make causal attribution difficult. SKM training targets are constructed using simulator object identities, visibility, image-space and 3D positions, discrete task states, and subgoal annotations. Using such privileged information during training is reasonable, but the comparison with visual-memory baselines therefore changes both the representation and the supervision available to the model. In addition, most component ablations are performed by removing records only at test time from a model trained with the complete representation; for example, removing subgoal-completion records reduces success from 70.62% to 24.31%. Such results demonstrate that the trained model relies on these records, but conflate information removal with train–test distribution shift. Matched-supervision controls and retrained ablations, especially for completion records and object identifiers, would provide much cleaner evidence for the proposed design.
  4. The real-world evaluation is promising but currently too underspecified to support the stated generalization claim. The paper reports three tasks with only 10 trials per task, but does not clearly specify the robot platform, low-level controller/VLA, whether the high-level R2Mem model is transferred directly from simulation or fine-tuned on real-world data, how real-world SKM supervision is obtained if fine-tuning is used, or whether the reported numbers correspond to fully closed-loop physical execution. These distinctions materially affect how strong the result is. If the evaluation is genuinely simulation-trained, zero-shot, closed-loop transfer, this would be a meaningful strength and should be stated explicitly; otherwise, the current real-world section is better viewed as a small proof-of-concept study.
Questions for authors
  1. Can the authors provide a matched free-form semantic-memory baseline that uses the same backbone, observations, supervision information, and comparable memory/token budget, but does not impose the proposed structured event representation? This would directly test whether SKM provides benefits beyond semantic memory in general.
  2. Given the very large performance drop after removing subgoal-completion records, can the authors report a simpler progress-only baseline (e.g., MemER + completed-subgoal history) and, ideally, retrained ablations without completion records and without persistent object identifiers?
  3. Training uses ground-truth previous memory entries, whereas deployment recursively conditions on predicted memory. Can the authors report memory-generation quality, such as object-ID consistency, completion-detection accuracy, or an oracle GT-memory result, to quantify error accumulation?
  4. Please clarify the complete real-world evaluation protocol, including the robot platform, low-level policy, high-level training/fine-tuning procedure, whether transfer from simulation is zero-shot, how memory labels are obtained if real-world fine-tuning is used, and whether evaluation consists of fully closed-loop physical rollouts.
3 comments
AuthorsOpenReview ↗

Response to Reviewer LLnM (Part 1/3)

We thank the reviewer for recognizing the diagnostic study and the effectiveness of SKM. We address the representation controls, supervision, memory errors, and real-world protocol below.

Structured vs. Free-Form Memory (W1, Q1)

Can the authors provide a matched free-form semantic-memory baseline …?

We have added two free-form memory baselines to examine whether unstructured text can provide the same benefit. Both use Qwen3-VL-4B at evaluation and learn keyframe selection and subgoal prediction. One uses self-generated text without direct text supervision; the other receives offline GPT-generated text supervision. The names in parentheses identify the models used to generate the free-form memory text.

Memory RepresentationSwingXtimesBinFillVideoUnmaskSwapButtonUnmaskSwapAVG
SKM (R2Mem)72.6766.6785.3374.6770.62
Free-form (Qwen3-VL-4B)56.0054.0030.6726.0055.69
Free-form (GPT-6-Sol-High)58.6754.0038.0029.3358.77

Success rates are measured over 150 episodes per task; AVG covers all 13 RoboMME tasks.

SKM outperforms both free-form alternatives, including the variant with GPT-generated text supervision. The largest gains among the displayed tasks occur in hidden-object tracking through swaps, where the planner must retain and connect successive changes. These comparisons support the value of explicitly organizing task-relevant events beyond providing free-form textual memory. We will add these baselines and their training details to the revision.

Scope of the Contribution (W2)

The novelty claims appear broader than the demonstrated methodological contribution.

Our contribution is SKM, which organizes historical reasoning outcomes around task-relevant key events for reuse in subgoal planning. Figure 1 shows limited gains from improving keyframe selection or density alone. Our analysis identifies two remaining issues: conclusions remain implicit in images, and frame selection can discard intervening changes. SKM pairs keyframes with explicit key event records to preserve this information. Its event types apply across tasks, while the task determines which object changes and completed operations to record, preserving object histories and execution progress beyond individual snapshots.

R2Mem enables an existing VLM to jointly learn memory construction and subgoal prediction through its standard image–text interface. With a common low-level VLA, the retrained ablations demonstrate the complementary value of object histories and execution progress, while comparisons with free-form memory support their explicit organization. Together, these results support SKM's design: retaining and organizing task-relevant historical information improves subsequent planning and task success.

AuthorsOpenReview ↗

Response to Reviewer LLnM (Part 2/3)

Progress-Only Memory and Retrained Controls (W3, Q2)

… can the authors report a simpler progress-only baseline … and, ideally, retrained ablations without completion records and without persistent object identifiers?

We now retrain the ablated variants with the same memory format used at evaluation, addressing the distribution shift introduced by test-time removal. The variant without observation memory retains keyframes and completed-subgoal records, providing a progress-only textual memory control within our framework. We also retrain variants without completion records or object identifiers.

VariantSwingXtimesBinFillVideoUnmaskSwapButtonUnmaskSwapAVG
R2Mem72.6766.6785.3374.6770.62
w/o reasoning text54.0052.6727.3320.0053.54
w/o action memory59.3355.3378.0056.6760.62
w/o observation memory (progress-only text)64.6762.0032.6744.0064.31
w/o object identifiers70.0065.3346.6758.6767.95

Evaluation uses 150 episodes per task; AVG covers all 13 RoboMME tasks. Removing reasoning text leaves visual memory; removing action or observation memory deletes completion or object-history records, respectively.

Progress records help, but do not account for the full benefit of SKM. The progress-only variant falls behind on both swap tasks, where object histories are needed to track hidden targets. Conversely, removing completion records reduces success on repetition and collection tasks. Removing identifiers particularly affects the swap tasks, supporting consistent links between events and objects. These retrained results demonstrate complementary contributions after each model has adapted to its available memory.

Simulator metadata supplies training labels, while deployment relies on predicted memory. Our simulation data construction also follows RoboMME's noise settings. The real-world protocol below demonstrates how corresponding labels are obtained without simulator metadata. We will include the retrained controls and clarify the supervision used by each variant.

Predicted Memory Quality and Oracle Control (Q3)

Can the authors report memory-generation quality, … or an oracle GT-memory result, to quantify error accumulation?

We assess generated SKM against GT records from the same rollout history. HunYuan-4 separately checks text and keyframes for task-critical errors or omissions: each receives full credit when it passes and zero otherwise, with 6 points for text and 4 for keyframes. Scores and task success are measured on the same episodes.

MetricCountingPermanenceReferenceImitation
Text score / 65.285.604.524.64
Image score / 43.763.813.923.97
SKM score / 109.049.418.448.61
Success (%)77.7886.6753.6761.67

Predicted SKM preserves task-relevant information across all four categories. Keyframe scores remain near full marks, and categories with higher text-memory quality also have higher task success. This correspondence supports memory quality as an indicator of downstream performance, consistent with the rationale of reasoning to remember.

To examine changes over time, we sort rollout chunks by timestamp and average their scores within successive sample percentile intervals.

Sample Percentile RangeMean SKM Score / 10
0–25%9.79
25–50%9.71
50–75%9.19
75–100%8.20

Quality decreases later in execution but remains high in the final interval. SKM therefore retains useful historical information despite some degradation. Efficient memory correction is a direction for future work.

For the oracle control, we replace predicted SKM with GT memory available up to the current query. The planner and low-level policy are unchanged, and the planner still predicts its own subgoals.

MemoryCountingPermanenceReferenceImitationAVG
Predicted SKM77.7886.6753.6761.6770.62
GT SKM81.5687.6777.8378.0081.74
Δ\Delta (pp)+3.78+1.00+24.16+16.33+11.12

Values are success rates (%); AVG covers all 13 RoboMME tasks.

The oracle intervention confirms that memory quality affects task completion with the rest of the system fixed. Predicted SKM performs close to the GT reference on Counting and Permanence, while Reference and Imitation benefit more from corrected memory. This separates the effect of memory quality from changes to the planner or executor. We will include these evaluations and the scoring protocol in the revision.

AuthorsOpenReview ↗

Response to Reviewer LLnM (Part 3/3)

Real-World Evaluation Protocol (W4, Q4)

Please clarify the complete real-world evaluation protocol, … and whether evaluation consists of fully closed-loop physical rollouts.

Our real-world experiments follow RoboMME's official task design and evaluation protocol. They use real-world fine-tuning, rather than zero-shot transfer from simulation.

Platform and training data. We use a UR5e robot with wrist and shoulder cameras and π0.5\pi_{0.5} as the low-level policy. We collect 50 demonstrations per task and construct training labels primarily through manual annotation with VLM assistance, covering keyframes, object IDs, event descriptions, and completed subgoals.

Fine-tuning and evaluation. We continue fine-tuning the MemER and R2Mem Qwen3-VL-4B planners from their respective simulation-trained checkpoints on these demonstrations. Both learn keyframe selection and subgoal prediction. The low-level π0.5\pi_{0.5} policy is initialized from its official pretrained checkpoint and fine-tuned on real-world data. Each method is evaluated over 10 closed-loop physical trials per task. We will include these protocol details in the revised manuscript.

Reviewer_ckTMOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses memory-dependent robotic manipulation by extending the hierarchical VLA framework of MemER. It combines historical keyframes with structured reasoning text to explicitly record object state changes and completed operations, enabling more accurate subgoal planning. Experiments on RoboMME, RoboMemArena, and real-world tasks demonstrate substantial improvements over existing visual memory methods.

Critical strengths
  1. Controlled experiments on MemER reveal the limitations of visual-only keyframe memory, providing clear motivation for combining keyframes with explicit reasoning records.

  2. The method is well-motivated, extending keyframe memory with reasoning text, object identifiers, and key-event records. It achieves substantial improvements on multiple memory-dependent simulation benchmarks and real-world tasks.

Critical weaknesses
  1. The ablation studies are not sufficiently rigorous. Only the "w/o Reasoning Text" variant is retrained, while other ablations, such as removing object identifiers, subgoal completion, or significant displacement records, are performed only at test time, introducing a train-test mismatch. Thus, their actual contributions remain unclear. Notably, removing reasoning text yields 53.54% success, close to MemER's 50.72%.

  2. Training labels are constructed using simulator-provided object identities, visibility, positions, discrete states, and subgoal completion signals, potentially limiting applicability to real-world data without such annotations. Moreover, training uses ground-truth memory, whereas inference relies on self-generated memory, potentially causing error accumulation. The paper does not analyze memory prediction accuracy or its impact on task success.

  3. Structured reasoning records introduce additional generation and context overhead, but inference latency, token cost, and GPU memory usage are not reported relative to MemER.

Questions for authors
  1. Can the authors retrain and evaluate the ablated variants to isolate the actual contributions of object identifiers, subgoal completion, and other key events?
  2. Can the authors quantify memory prediction errors, their accumulation over time, and their impact on task success?
  3. Can the authors report inference latency, token cost, and GPU memory usage compared with MemER?
2 comments
AuthorsOpenReview ↗

Response to Reviewer ckTM (Part 1/2)

We thank the reviewer for recognizing the motivation and empirical improvements. We address the concerns below.

Retrained Ablations (W1, Q1)

Can the authors retrain and evaluate the ablated variants to isolate the actual contributions of object identifiers, subgoal completion, and other key events?

We have added retrained ablations, using consistent memory representations during training and evaluation. Each variant removes one component: reasoning text (all textual memory records), action memory (textual records of completed subgoals), observation memory (textual records of object histories), or object identifiers (textual labels linking events to specific objects).

VariantSwingXtimesBinFillVideoUnmaskSwapButtonUnmaskSwapAVG
R2Mem72.6766.6785.3374.6770.62
w/o reasoning text54.0052.6727.3320.0053.54
w/o action memory59.3355.3378.0056.6760.62
w/o observation memory64.6762.0032.6744.0064.31
w/o object identifiers70.0065.3346.6758.6767.95
Free-form (Qwen3-VL-4B)56.0054.0030.6726.0055.69
Free-form (GPT-6-Sol-High)58.6754.0038.0029.3358.77

Results are success rates (%), with 3×50=1503\times50=150 episodes per task. AVG covers all 13 RoboMME tasks; the four displayed tasks illustrate counting and hidden-object tracking.

Object histories and identifiers support tracking through occlusion and swaps. Removing observation memory substantially reduces success on both important swap tasks. Removing object identifiers also causes large drops on these tasks, while having much smaller effects on counting. Together, these results support preserving both what happened and which object each event concerns, so the planner can track targets across successive changes.

Action memory preserves execution progress. Removing completion records reduces success on SwingXtimes and BinFill, which require counting repetitions and object placements, respectively. Explicitly recording completed operations helps the planner determine what remains to be done.

Free-form memory comparisons further support SKM's structured representation. Both variants use Qwen3-VL-4B at evaluation: one uses self-generated text without direct text supervision; the other uses offline GPT-generated text supervision. SKM outperforms both, exceeding the stronger GPT-supervised variant by 11.85 percentage points in average success, with particularly large gains on the swap tasks. These results support explicitly organizing object histories and completed operations for subsequent planning.

We will include these retrained ablations in the revision.

Training Annotations and Real-World Applicability (W2)

Training labels are constructed using simulator-provided object identities, visibility, positions, discrete states, and subgoal completion signals, potentially limiting applicability to real-world data without such annotations.

Simulator metadata is used to construct training supervision; at deployment, the VLM predicts memory records without externally supplied object IDs or completion signals. Following RoboMME, we also introduce noise when constructing the simulation training data.

For real-world training, we obtain the corresponding labels through manual annotation with VLM assistance. These cover keyframes, object IDs, event descriptions, and completed subgoals. With planners fine-tuned on these demonstrations, R2Mem outperforms MemER on all three real-world tasks in closed-loop evaluation (Fig. 5), demonstrating that SKM can be learned and used without simulator-provided annotations. We will clarify this training and annotation protocol in the revision.

AuthorsOpenReview ↗

Response to Reviewer ckTM (Part 2/2)

Memory Quality and Error Accumulation (W2, Q2)

Can the authors quantify memory prediction errors, their accumulation over time, and their impact on task success?

We evaluate task-critical memory quality against GT SKM from the same rollout prefix. HunYuan-4 checks text and keyframes for errors or omissions likely to prevent task completion. Each component receives full credit if it passes, and zero otherwise: 6 points for text and 4 for keyframes.

Memory quality and task success. We report category-level mean scores and success rates on the same episodes.

MetricCountingPermanenceReferenceImitation
Text score / 65.285.604.524.64
Image score / 43.763.813.923.97
SKM score / 109.049.418.448.61
Success (%)77.7886.6753.6761.67

SKM maintains high task-critical memory quality across categories. Keyframe selection scores remain near full marks, while higher text-memory scores consistently accompany higher task success across categories. This alignment supports memory quality as an indicator of downstream performance and reinforces the rationale behind “reasoning to remember”: more accurate retention of task-relevant conclusions is associated with more successful task execution.

Memory quality over time. We sort chunks across rollouts by timestamp and average scores within four successive sample percentile intervals.

Sample Percentile RangeMean SKM Score / 10
0–25%9.79
25–50%9.71
50–75%9.19
75–100%8.20

Memory quality declines over time but remains high in the final interval, with a mean score of 8.20/10. This indicates that SKM continues to preserve task-relevant information later in execution despite some degradation. Developing an efficient and effective memory correction mechanism is a direction for future work.

Oracle-memory evaluation. We replace predicted SKM with GT SKM containing only information available up to the current query. The planner and low-level VLA remain unchanged, and subgoals are still model-predicted.

MemoryCountingPermanenceReferenceImitationAVG
Predicted SKM77.7886.6753.6761.6770.62
GT SKM81.5687.6777.8378.0081.74
Δ\Delta (pp)+3.78+1.00+24.16+16.33+11.12

Values are success rates (%); Δ\Delta is computed from the displayed rates. AVG averages all 13 RoboMME tasks.

Replacing predicted memory with GT memory improves average success by 11.12 percentage points, demonstrating the impact of memory quality with the planner and executor fixed. Predicted SKM already approaches GT-memory performance on Counting and Permanence; the larger gains on Reference and Imitation show greater sensitivity to memory quality. We will include these analyses and evaluation details in the revision.

Computational Cost (W3, Q3)

Can the authors report inference latency, token cost, and GPU memory usage compared with MemER?

We quantify the additional cost of SKM relative to MemER using the same NVIDIA H20, Qwen3-VL-4B planner, and π0.5\pi_{0.5} policy. Both methods process 8 recent frames plus memory keyframes per call, with 64 visual tokens per frame. R2Mem jointly predicts memory updates and subgoals, so memory construction requires no additional VLM call. Keyframe counts are averages. The low-level policy uses 6.3 GB VRAM in all settings. Relative success is measured on Figure 1's diagnostic task subset and normalized to MemER.

Model ConfigurationVisual TokensMedian Latency (s)VLM VRAM (GB)Relative Success
MemER (8 recent + 7.14 keyframes)9690.8709.51.00
MemER, 2×2\times keyframes (8 recent + 14.28 keyframes)1,4260.9889.51.10
R2Mem (8 recent + 7.77 keyframes)1,0090.9629.51.32

R2Mem achieves substantially higher task success with modest increases in visual-token count and latency, while reported VLM memory usage remains unchanged. It also outperforms MemER with doubled keyframes while using fewer visual tokens and slightly less inference time, supporting the efficiency of explicit event records for planning. We will include these cost comparisons in the revision.

Reviewer_QtrkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper tackles the challenges of adding robot memory to long-horizon manipulation tasks. This paper found that directly adding key frames to the robot memory is inefficient and provides very little performance gain. Instead, the paper provides an alternative way that will pair keyframes with reasoning text to more efficiently manage the robot memory. The proposed approach is evaluated on 2 benchmarks and 3 real-world experiments.

Critical strengths

1: Robot memory is an important and unexplored problem in robot manipulation, especially in large-scale experiments.

2: The proposed approach builds upon a hierarchical structure that will use a VLM to do keyframe reasoning and generate subgoals while using VLAs to execute each behavior. This system overall decomposes the challenges of memory reasoning and motion generation.

3: The proposed approach is evaluated in both simulation and real-world experiments and outperforms several baselines.

4: The paper also includes an ablation study to test which component is most important for the overall system.

Critical weaknesses

1: The paper is overall difficult to understand. For example, in the introduction, the paper first introduces VLA, then long-horizon manipulation, then the hierarchical system, and finally memory. Since memory is the core problem the paper addresses, I would introduce the concept first.

2: The paper should also provide more literature on robot memory in manipulation. Here are two papers for reference: https://arxiv.org/abs/2602.15010 and https://arxiv.org/pdf/2309.15278.

3: The authors should also be careful when defining new concepts. For example, “a Markovian state” is neither defined nor cited properly.

4: The real-world experiments are overall too simple, with only quasi-static pick-and-place tasks. Studying robot memory will be more exciting in large-scale environments.

5: The figures also have significant room to improve. For example, in Figure 1, I get very little information about the paper’s core takeaway.

Questions for authors

1: What’s the key difference between the proposed approach and https://arxiv.org/abs/2602.15010 and https://arxiv.org/pdf/2309.15278?

2: What is the frequency at which you can deploy your hierarchical system in the real world? Would keyframe reasoning and subgoal decision be the bottleneck for deployment efficiency?

3: What are the failure cases of your proposed approach? Why does your proposed approach overall perform worse in RoboMemArena compared to RoboMME?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Qtrk (Part 1/2)

We thank the reviewer for the constructive feedback and address the concerns below.

Introduction and Terminology (W1, W3)

Since memory is the core problem the paper addresses, I would introduce the concept first.

Our intention in introducing the hierarchical VLA framework first was to help readers understand how memory influences robot behavior: SKM preserves historical information for the high-level VLM to select subgoals, which the low-level VLA then executes. Following your suggestion, we will foreground the memory problem and its motivation, then introduce the hierarchical framework to explain how memory supports subgoal planning.

“a Markovian state” is neither defined nor cited properly.

We use “Markovian” in its standard sense: given the current state and action, the next-state distribution is independent of earlier history. The current image may omit hidden-object locations or completed operations, making history necessary for planning. We will add this definition and a supporting reference.

Comparison to Prior Work (W2, Q1)

The paper should also provide more literature on robot memory in manipulation.

Thank you for suggesting these relevant works. We clarify below how the suggested methods differ in what they preserve and how their memories support planning:

  • BPP, like MemER, uses selected visual keyframes as memory. Our controlled experiments on MemER (Fig. 1) show limited gains from ground-truth keyframe selection or doubled keyframe density. This motivates R2Mem's Structured Keyframe Memory (SKM), which pairs keyframes with explicit event records so the planner can reuse historical conclusions rather than reconstruct them from snapshots.
  • DOOM/LOOM maintain geometric or latent object-state memories for planning. R2Mem explicitly records task-level knowledge, including object histories and completed operations, as text grounded in keyframes. This unified representation uses the standard image–text interface, enabling existing open-source VLMs to learn memory construction and use through fine-tuning, without architectural changes or dedicated tracking and dynamics networks.

We will include both references and clarify these distinctions.

Real-World Evaluation (W4)

The real-world experiments are overall too simple, with only quasi-static pick-and-place tasks.

Our three real-world tasks are based on RoboMME's real-robot task designs, which mirror memory-dependent tasks in its simulation benchmark:

  • TrackFruit adapts RoboMME's TrackCube task, using fruit as the hidden target. It corresponds to VideoUnmaskSwap in simulation and requires tracking the target through container swaps.
  • PickFruit follows RoboMME's PutFruits task, the real-world counterpart of BinFill, and requires remembering how many fruits have been collected.
  • RepickBlock follows RoboMME's real-world task of the same name, corresponding to VideoRepick in simulation. Our version requires identifying the object manipulated in a demonstration and repeating the operation a specified number of times.

We also follow RoboMME’s official evaluation protocol, with 10 trials per method per task. R2Mem outperforms MemER on all three tasks (Fig. 5). A key priority for our future work is to deploy R2Mem in large-scale real-world robotic applications.

AuthorsOpenReview ↗

Response to Reviewer Qtrk (Part 2/2)

Figure 1 (W5)

In Figure 1, I get very little information about the paper’s core takeaway.

Figure 1 tests whether better keyframe selection or greater density addresses the memory failures observed with MemER. Ground-truth keyframes do not improve average success, while doubling keyframes yields approximately 10% higher success. R2Mem achieves approximately 32% higher success with only 9% more keyframes, relative to MemER.

The limited gains from the keyframe-only interventions motivate examining what task-relevant information remains implicit in, or absent from, the retained images. As discussed in Section 3.2, task-relevant conclusions remain implicit, while keyframe selection can discard intervening changes and interactions. We appreciate this feedback and recognize that the current caption does not make the connection between these findings and SKM sufficiently clear. We will revise the caption to explain how the results motivate preserving explicit event records alongside keyframes.

Deployment and Compute Analysis (Q2)

What is the frequency at which you can deploy your hierarchical system in the real world? Would keyframe reasoning and subgoal decision be the bottleneck for deployment efficiency?

To compare methods under the same execution settings, we use our MemER baseline’s schedule: one VLM call per 16 low-level control steps, with a configured timestep of 0.05 s (20 Hz). R2Mem updates memory and predicts subgoals in the same call.

All configurations in the following table are benchmarked on NVIDIA H20, using Qwen3-VL-4B as the planner and π0.5\pi_{0.5} as the low-level policy. Each VLM call receives 8 frames sampled from the 16-step chunk plus memory keyframes, with 64 visual tokens per frame. The low-level policy uses 6.3 GB VRAM in all settings.

Model ConfigurationVisual TokensMedian Latency (s)VLM VRAM (GB)Relative Success
MemER (8 recent + 7.14 keyframes)9690.8709.51.00
MemER, 2×2\times keyframes (8 recent + 14.28 keyframes)1,4260.9889.51.10
R2Mem (8 recent + 7.77 keyframes)1,0090.9629.51.32

With comparable VLM latency and unchanged reported VRAM usage, R2Mem achieves approximately 32% higher task success than MemER in the Figure 1 comparison. It also achieves higher success at lower latency than MemER with doubled keyframes. We will include these measurements in the revision.

Failure Cases and RoboMemArena (Q3)

What are the failure cases of your proposed approach? Why does your proposed approach overall perform worse in RoboMemArena compared to RoboMME?

Failure cases. SKM is designed for decisions that depend on object histories and task progress. As noted in our conclusion, tasks readily solved through visual imitation may benefit less from additional reasoning records, since retained images already provide sufficient guidance. Adapting these records to each task’s memory demands is a direction for future work.

RoboMemArena performance. We use the pretrained VLA checkpoint provided by the RoboMemArena authors, which achieves only 46.1% success even with ground-truth subgoals, indicating a low-level execution bottleneck. With the same checkpoint, R2Mem achieves 43.8%, approximately 95% of the reference performance with ground-truth subgoals. We will clarify this execution constraint and comparison in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.