- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Recall & Roll (R&R-Bench), a benchmark for evaluating memory-dependent embodied tasks across navigation and manipulation. The benchmark contains 300 long-horizon experience contexts across 228 simulated houses and 1,772 queries spanning four task families: Finding, Restore, Resume, and Routine. To better understand the role of memory in embodied agents, the authors introduce three evaluation settings covering memory question answering, execution with ground-truth goals, and full-pipeline execution with recalled goals.
The paper further investigates the strengths and limitations of different memory strategies, including textual and visual memory. Based on these findings, the authors propose R&R Agent, a reference system combining textual records, visual evidence, and request-conditioned retrieval, which achieves promising results across the evaluated tasks.
Critical strengths
1. A well-designed benchmark for embodied memory evaluation.
The paper presents a meaningful and well-structured benchmark that covers different memory requirements through four task families. I particularly appreciate the separation of memory QA, execution with ground-truth goals, and full-pipeline execution, which helps distinguish memory retrieval failures from difficulties encountered during task execution. This provides a more informative evaluation than relying solely on memory QA accuracy or end-to-end task success.
2. Comprehensive experiments with useful insights into memory representations.
The paper provides extensive comparisons across different memory strategies and VLM readers. The findings reveal interesting complementary strengths of textual and visual memory, particularly for spatial recall, task progress tracking, and rule inference. The proposed R&R Agent further demonstrates how these insights can be incorporated into a practical memory system. Overall, the experimental analysis provides useful guidance for future memory-augmented embodied agents.
Critical weaknesses
1. Limited isolation of memory performance from low-level manipulation failures.
Although the benchmark introduces separate evaluation settings to analyze memory and execution, the full-pipeline results may still be heavily affected by the limited capabilities of the low-level manipulation policy. For example, Appendix E.2 reports relatively low success rates for several basic manipulation skills, particularly precise placement and opening operations. Consequently, it remains unclear to what extent full-pipeline failures reflect limitations in memory rather than low-level execution. A controlled evaluation with oracle skills or a more reliable manipulation policy would help better isolate the contribution of memory.
2. Limited evaluation of end-to-end memory-augmented policies.
The current experiments primarily compare memory strategies within a shared hierarchical framework, where a VLM handles memory and high-level planning while a VLA executes low-level actions. While this provides a controlled setting for comparing memory mechanisms, the benchmark does not evaluate representative end-to-end memory-augmented VLA approaches, such as MemoryVLA, MemoAct, or HAMLET. Including such methods would help establish the benchmark's applicability to a broader range of embodied memory architectures and provide insights into the differences between hierarchical and end-to-end approaches.
Questions for authors
Q1. Impact of low-level manipulation reliability on memory evaluation.
Given the relatively low success rates of some basic manipulation skills reported in Appendix E.2, could the authors further investigate how much of the full-pipeline failure is attributable to memory limitations versus low-level execution failures? An additional experiment using oracle skills or a more reliable manipulation policy, while keeping the memory and high-level planning components unchanged, would help clarify whether the benchmark can effectively distinguish different memory capabilities.
Q2. Evaluation of end-to-end memory-augmented policies.
The current evaluation focuses on different memory strategies within a two-stage VLM–VLA framework. Could the authors consider evaluating representative end-to-end memory-augmented VLA methods on compatible tasks in R&R-Bench? Such comparisons would help assess how different memory architectures perform on the proposed benchmark and provide a broader perspective on hierarchical versus end-to-end memory integration.
Q3. Statistical reliability of the reported improvements.
R&R Agent achieves the highest average full-pipeline success rates among the evaluated experience-based memory methods, but some improvements over the strongest baselines are relatively small. For example, the gains are only 1.5 and 2.0 percentage points on Finding and Restore, respectively. Although the paper reports averages over three independent runs, could the authors provide confidence intervals or statistical significance analysis for these comparisons? Since multiple queries share the same contexts and houses, estimating uncertainty at the context or house level would help establish whether the reported improvements are statistically reliable.
4 comments
W2 & Q2. Evaluation of end-to-end memory-augmented VLAs
Thank you for this suggestion. We agree that evaluating end-to-end memory VLAs is valuable for assessing how broadly the benchmark applies across memory architectures. We therefore evaluated two representative methods, HAMLET and MemoryVLA (Appendix C.10, Table 13). We could not include MemoAct because its code was not publicly available at the time of our experiments, and we will add it once it is released. Since these policies do not support navigation, we use only Single queries that require no navigation. All Restore and Routine queries require navigation and are therefore not applicable. We run full-pipeline execution: the recorded context is given to the policy as its observation history with its actions ignored, and the policy acts only during execution.
| Method | Finding (83) | Resume (13) |
|---|---|---|
| HAMLET | 2.4 (2/83) | 0.0 (0/13) |
| MemoryVLA | 0.0 (0/83) | 0.0 (0/13) |
Both policies achieve near-zero success. The recorded contexts in R&R-Bench are much longer than the history these policies are designed to retain, so the information needed for the request is no longer available at execution time. This result highlights the gap that R&R-Bench is designed to expose: current end-to-end memory VLAs cannot yet exploit long-horizon experience across navigation and manipulation. As navigation-capable memory VLAs become available, they can be evaluated under the same protocol and compared directly with two-stage methods, which demonstrates the extensibility of the benchmark.
Q3. Statistical reliability of the reported improvements
Summary (details in Common Response 2): we now report mean ± s.d. over three runs for Tables 2 to 4 (Appendix C.8, Figure 13) and the number of queries for every column.
Context-level uncertainty. As the reviewer notes, queries that share a context or house are not independent. We therefore estimated 95% confidence intervals for the full-pipeline margin of R&R Agent over the strongest baseline without the GT Action Log, using a paired bootstrap that resamples contexts or houses (10,000 resamples).
Table R5. Full-pipeline margin (points) of R&R Agent over the strongest baseline, with 95% bootstrap intervals. Margins are computed from unrounded values.
| Task (queries / contexts / houses) | R&R Agent | Strongest baseline | Margin | Context-level | House-level |
|---|---|---|---|---|---|
| Finding (722 / 100 / 84) | 24.5 | MemER$^R$ 23.0 | +1.5 | [−1.9, +4.9] | [−1.8, +5.0] |
| Restore (575 / 100 / 84) | 5.7 | MemER$^R$ 3.7 | +2.1 | [−0.2, +4.5] | [−0.2, +4.4] |
| Resume (161 / 100 / 100) | 11.2 | MEM$^R$ 8.7 | +2.5 | [−3.7, +8.9] | [−3.7, +8.8] |
| Routine (310 / 100 / 59) | 11.3 | MEM$^R$ 8.4 | +2.9 | [−1.0, +6.8] | [−1.0, +6.7] |
| Average | +2.2 | [+0.1, +4.3] | [+0.2, +4.4] |
The margin is positive on every task, and its average over the four tasks excludes zero at both levels. The per-task intervals individually include zero, so we do not claim significance for each task separately. Context- and house-level resampling give nearly identical intervals, indicating that the dependence among queries has little effect.
Effect of low-level noise. Part of the small margins comes from low-level execution failures shared by all methods. With the oracle manipulator (Table R4), the margins grow from 1.5 and 2.0 points to 3.0 and 4.0 points on Finding and Restore, the two examples raised by the reviewer, and to 4.3 and 9.7 points on Resume and Routine.
Thank you for the rebuttal and additional experiments. My concerns have been largely addressed, and I will raise my score to 3 (Weak acceptance).
Thank you for the positive reassessment and the constructive suggestions.
Response to Reviewer WYir
We thank the reviewer for the constructive feedback and for finding the benchmark well designed and the analysis useful.
W1 & Q1. Isolating memory from low-level manipulation failures
Thank you for this constructive suggestion. We agree that isolating memory from low-level manipulation failures is important to verify that the benchmark measures memory capability. To this end, we replaced the VLA with an oracle manipulator (Appendix C.11): a magic gripper that automatically grasp, place and open/close the object when the robot moves near the target and attempts a grasp (place or open/close), with a manipulation success rate of about 99%. Navigation, subgoal decisions, and all memory components are unchanged, so the remaining failures reflect the agent's choices rather than manipulation. We evaluated both execution with ground-truth instructions and full-pipeline execution.
Execution with ground-truth instructions. Finding and Restore use 30 queries for each of one, two, and three targets, and Resume and Routine use the same queries as Table 3 (mean over three runs).
Table R3. Execution success (%) with ground-truth instructions and the oracle manipulator (Table 14).
With reliable manipulation, memory-based methods clearly outperform No Memory, by up to 27.4 points on Routine, confirming that execution-time memory remains necessary even when low-level skills succeed. R&R Agent achieves the highest success on Finding and Routine and is within 0.6 points of MEM$^R$ on Restore and Resume. The strong results of the two methods with textual records are consistent with our finding that textual memory is effective for progress tracking. As shown in Table R2 of our response to Reviewer wbmu, the gap between memory-based methods and No Memory also widens as the number of targets increases.
Full-pipeline execution. We also ran the full pipeline with the oracle manipulator on all queries (Table 15). Due to the evaluation cost, these results are from a single run. Values in parentheses are the original results with the VLA (Table 4).
Table R4. Full-pipeline success (%) with the oracle manipulator. Bold: best among methods without the GT Action Log.
These results support that the benchmark distinguishes memory capabilities:
We will extend the full-pipeline evaluation with repeated runs in the final version.