acceptodds
Under review as a conference paper at ICLR 2027

RECALL & ROLL: Benchmarking How Embodied Agents Remember, Act, and Update Memory

Abstract

Embodied agents must recall past experience and track progress while acting. We introduce Recall & Roll Benchmark (R&R-Bench) to evaluate these requirements across navigation and manipulation. It comprises 300 contexts across 228 simulated houses, totaling 43 hours of experience and 1,772 queries. Four task families, Finding, Restore, Resume, and Routine, cover distinct memory requirements, with variations in request scope, evidence source, and target concealment. Three evaluation conditions examine memory QA, execution given correct goals, and the full pipeline, supporting analysis of retrospective recall and its relationship to task execution. Across the evaluated systems, visual memory better supports spatial recall and passively observed information, while textual memory better supports unfinished-work recovery, storage-rule inference, and progress tracking. Guided by these findings, we introduce R&R Agent as a reference system that combines textual records, tagged visual evidence, and request-conditioned retrieval. It achieves the highest task-average memory-QA accuracy and full-pipeline success across all four families among the evaluated methods that construct memory from experience. Separate placement experiments examine retrieved visual goals for precise manipulation. R&R-Bench provides a common testbed for characterizing embodied memory requirements, identifying limitations of existing approaches, and informing future agent design.

Primary area:
datasets and benchmarks
Keywords:
Robotics, Memory, Manipulation, Navigation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WYirOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Recall & Roll (R&R-Bench), a benchmark for evaluating memory-dependent embodied tasks across navigation and manipulation. The benchmark contains 300 long-horizon experience contexts across 228 simulated houses and 1,772 queries spanning four task families: Finding, Restore, Resume, and Routine. To better understand the role of memory in embodied agents, the authors introduce three evaluation settings covering memory question answering, execution with ground-truth goals, and full-pipeline execution with recalled goals.

The paper further investigates the strengths and limitations of different memory strategies, including textual and visual memory. Based on these findings, the authors propose R&R Agent, a reference system combining textual records, visual evidence, and request-conditioned retrieval, which achieves promising results across the evaluated tasks.

Critical strengths

1. A well-designed benchmark for embodied memory evaluation.

The paper presents a meaningful and well-structured benchmark that covers different memory requirements through four task families. I particularly appreciate the separation of memory QA, execution with ground-truth goals, and full-pipeline execution, which helps distinguish memory retrieval failures from difficulties encountered during task execution. This provides a more informative evaluation than relying solely on memory QA accuracy or end-to-end task success.

2. Comprehensive experiments with useful insights into memory representations.

The paper provides extensive comparisons across different memory strategies and VLM readers. The findings reveal interesting complementary strengths of textual and visual memory, particularly for spatial recall, task progress tracking, and rule inference. The proposed R&R Agent further demonstrates how these insights can be incorporated into a practical memory system. Overall, the experimental analysis provides useful guidance for future memory-augmented embodied agents.

Critical weaknesses

1. Limited isolation of memory performance from low-level manipulation failures.

Although the benchmark introduces separate evaluation settings to analyze memory and execution, the full-pipeline results may still be heavily affected by the limited capabilities of the low-level manipulation policy. For example, Appendix E.2 reports relatively low success rates for several basic manipulation skills, particularly precise placement and opening operations. Consequently, it remains unclear to what extent full-pipeline failures reflect limitations in memory rather than low-level execution. A controlled evaluation with oracle skills or a more reliable manipulation policy would help better isolate the contribution of memory.

2. Limited evaluation of end-to-end memory-augmented policies.

The current experiments primarily compare memory strategies within a shared hierarchical framework, where a VLM handles memory and high-level planning while a VLA executes low-level actions. While this provides a controlled setting for comparing memory mechanisms, the benchmark does not evaluate representative end-to-end memory-augmented VLA approaches, such as MemoryVLA, MemoAct, or HAMLET. Including such methods would help establish the benchmark's applicability to a broader range of embodied memory architectures and provide insights into the differences between hierarchical and end-to-end approaches.

Questions for authors

Q1. Impact of low-level manipulation reliability on memory evaluation.

Given the relatively low success rates of some basic manipulation skills reported in Appendix E.2, could the authors further investigate how much of the full-pipeline failure is attributable to memory limitations versus low-level execution failures? An additional experiment using oracle skills or a more reliable manipulation policy, while keeping the memory and high-level planning components unchanged, would help clarify whether the benchmark can effectively distinguish different memory capabilities.

Q2. Evaluation of end-to-end memory-augmented policies.

The current evaluation focuses on different memory strategies within a two-stage VLM–VLA framework. Could the authors consider evaluating representative end-to-end memory-augmented VLA methods on compatible tasks in R&R-Bench? Such comparisons would help assess how different memory architectures perform on the proposed benchmark and provide a broader perspective on hierarchical versus end-to-end memory integration.

Q3. Statistical reliability of the reported improvements.

R&R Agent achieves the highest average full-pipeline success rates among the evaluated experience-based memory methods, but some improvements over the strongest baselines are relatively small. For example, the gains are only 1.5 and 2.0 percentage points on Finding and Restore, respectively. Although the paper reports averages over three independent runs, could the authors provide confidence intervals or statistical significance analysis for these comparisons? Since multiple queries share the same contexts and houses, estimating uncertainty at the context or house level would help establish whether the reported improvements are statistically reliable.

4 comments

Response to Reviewer WYir

We thank the reviewer for the constructive feedback and for finding the benchmark well designed and the analysis useful.


W1 & Q1. Isolating memory from low-level manipulation failures

Thank you for this constructive suggestion. We agree that isolating memory from low-level manipulation failures is important to verify that the benchmark measures memory capability. To this end, we replaced the VLA with an oracle manipulator (Appendix C.11): a magic gripper that automatically grasp, place and open/close the object when the robot moves near the target and attempts a grasp (place or open/close), with a manipulation success rate of about 99%. Navigation, subgoal decisions, and all memory components are unchanged, so the remaining failures reflect the agent's choices rather than manipulation. We evaluated both execution with ground-truth instructions and full-pipeline execution.

Execution with ground-truth instructions. Finding and Restore use 30 queries for each of one, two, and three targets, and Resume and Routine use the same queries as Table 3 (mean over three runs).

Table R3. Execution success (%) with ground-truth instructions and the oracle manipulator (Table 14).

MethodFindingRestoreResumeRoutine
No Memory70.452.228.038.5
MEM$^R$81.575.241.462.3
MemER$^R$75.967.830.451.2
R&R Agent83.075.040.865.9

With reliable manipulation, memory-based methods clearly outperform No Memory, by up to 27.4 points on Routine, confirming that execution-time memory remains necessary even when low-level skills succeed. R&R Agent achieves the highest success on Finding and Routine and is within 0.6 points of MEM$^R$ on Restore and Resume. The strong results of the two methods with textual records are consistent with our finding that textual memory is effective for progress tracking. As shown in Table R2 of our response to Reviewer wbmu, the gap between memory-based methods and No Memory also widens as the number of targets increases.

Full-pipeline execution. We also ran the full pipeline with the oracle manipulator on all queries (Table 15). Due to the evaluation cost, these results are from a single run. Values in parentheses are the original results with the VLA (Table 4).

Table R4. Full-pipeline success (%) with the oracle manipulator. Bold: best among methods without the GT Action Log.

MethodFindingRestoreResumeRoutine
GT Action Log37.3 (20.6)23.7 (8.2)21.1 (20.5)37.7 (14.2)
No Memory10.9 (6.2)0.3 (0.2)0.0 (0.0)5.8 (1.6)
Uniform Frames31.9 (20.1)7.7 (2.1)5.0 (2.5)14.2 (5.5)
MEM$^R$29.4 (17.5)9.0 (2.8)11.8 (8.7)31.3 (8.4)
MemER$^R$39.5 (23.0)13.0 (3.7)6.8 (5.6)19.4 (6.5)
R&R Agent42.5 (24.5)17.0 (5.7)16.1 (11.2)41.0 (11.3)

These results support that the benchmark distinguishes memory capabilities:

  • Differences between memory methods become clearer. R&R Agent remains the best method without the GT Action Log on all four tasks, and its margin over the strongest such baseline grows from 1.5, 2.0, 2.5, and 2.9 points with the VLA to 3.0, 4.0, 4.3, and 9.7 points with the oracle manipulator on Finding, Restore, Resume, and Routine. Low-level failures therefore compress, rather than create, the differences between memory methods.
  • Remaining failures are memory failures. On Finding and Restore, full-pipeline success approaches Memory QA accuracy for every memory-based method (e.g., 42.5% vs. 46.6% and 17.0% vs. 19.0% for R&R Agent), so performance is largely bounded by recall. On Resume and Routine, success remains limited even with ground-truth instructions (Table R3), reflecting the demands on execution-time memory over many steps.

We will extend the full-pipeline evaluation with repeated runs in the final version.

W2 & Q2. Evaluation of end-to-end memory-augmented VLAs

Thank you for this suggestion. We agree that evaluating end-to-end memory VLAs is valuable for assessing how broadly the benchmark applies across memory architectures. We therefore evaluated two representative methods, HAMLET and MemoryVLA (Appendix C.10, Table 13). We could not include MemoAct because its code was not publicly available at the time of our experiments, and we will add it once it is released. Since these policies do not support navigation, we use only Single queries that require no navigation. All Restore and Routine queries require navigation and are therefore not applicable. We run full-pipeline execution: the recorded context is given to the policy as its observation history with its actions ignored, and the policy acts only during execution.

MethodFinding (83)Resume (13)
HAMLET2.4 (2/83)0.0 (0/13)
MemoryVLA0.0 (0/83)0.0 (0/13)

Both policies achieve near-zero success. The recorded contexts in R&R-Bench are much longer than the history these policies are designed to retain, so the information needed for the request is no longer available at execution time. This result highlights the gap that R&R-Bench is designed to expose: current end-to-end memory VLAs cannot yet exploit long-horizon experience across navigation and manipulation. As navigation-capable memory VLAs become available, they can be evaluated under the same protocol and compared directly with two-stage methods, which demonstrates the extensibility of the benchmark.


Q3. Statistical reliability of the reported improvements

Summary (details in Common Response 2): we now report mean ± s.d. over three runs for Tables 2 to 4 (Appendix C.8, Figure 13) and the number of queries for every column.

Context-level uncertainty. As the reviewer notes, queries that share a context or house are not independent. We therefore estimated 95% confidence intervals for the full-pipeline margin of R&R Agent over the strongest baseline without the GT Action Log, using a paired bootstrap that resamples contexts or houses (10,000 resamples).

Table R5. Full-pipeline margin (points) of R&R Agent over the strongest baseline, with 95% bootstrap intervals. Margins are computed from unrounded values.

Task (queries / contexts / houses)R&R AgentStrongest baselineMarginContext-levelHouse-level
Finding (722 / 100 / 84)24.5MemER$^R$ 23.0+1.5[−1.9, +4.9][−1.8, +5.0]
Restore (575 / 100 / 84)5.7MemER$^R$ 3.7+2.1[−0.2, +4.5][−0.2, +4.4]
Resume (161 / 100 / 100)11.2MEM$^R$ 8.7+2.5[−3.7, +8.9][−3.7, +8.8]
Routine (310 / 100 / 59)11.3MEM$^R$ 8.4+2.9[−1.0, +6.8][−1.0, +6.7]
Average+2.2[+0.1, +4.3][+0.2, +4.4]

The margin is positive on every task, and its average over the four tasks excludes zero at both levels. The per-task intervals individually include zero, so we do not claim significance for each task separately. Context- and house-level resampling give nearly identical intervals, indicating that the dependence among queries has little effect.

Effect of low-level noise. Part of the small margins comes from low-level execution failures shared by all methods. With the oracle manipulator (Table R4), the margins grow from 1.5 and 2.0 points to 3.0 and 4.0 points on Finding and Restore, the two examples raised by the reviewer, and to 4.3 and 9.7 points on Resume and Routine.

Thank you for the rebuttal and additional experiments. My concerns have been largely addressed, and I will raise my score to 3 (Weak acceptance).

Thank you for the positive reassessment and the constructive suggestions.

Reviewer_wbmuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents R&R-Bench, a mobile manipulation and navigation benchmark composed of 300 contexts across 288 rooms created using MolmoSpaces, totaling 43 hours of experience and 1,722 queries. The queries cover four task families:

  • Finding: Identify where an object was previously seen.
  • Restore: Return an object to its original location.
  • Resume: Continue an interrupted task.
  • Routine: Follow a storage rule inferred from examples.

Each task family is further divided according to:

  • Single or multiple objects
  • Manipulated or observed objects
  • Concealed or unconcealed objects

The benchmark evaluates models at three separate levels:

  • Memory QA, with no physical action involved.
  • Execution from ground-truth instructions, which isolates execution-state memory.
  • Execution from recalled instructions, which evaluates the end-to-end pipeline.

All evaluated methods share a two-stage design consisting of a VLM memory writer and reader built on top of a fine-tuned MolmoBot policy. The authors compare a no-memory policy, ground-truth action logs, uniform frame memory, and prompt-based reimplementations of MEM, which uses rewritten textual summaries, and MemER, which uses keyframe selection.

Consistent with findings from RoboMME, visual memory is more effective for spatial reasoning and recalling passively observed information, whereas textual memory is more effective for progress tracking and rule inference. Based on these findings, the authors design the R&R Agent, which maintains an append-only memory of frames and textual annotations and retrieves relevant entries based on the current request.

Critical strengths
  1. Filling an important benchmark gap. Most existing memory-related benchmarks focus on either tabletop manipulation or navigation. R&R-Bench instead combines long-horizon navigation, manipulation, and memory in a mobile manipulation setting. In particular, the Routine task, which requires inferring and applying storage rules from previous examples, appears to be missing from existing benchmarks.

  2. Strong diagnostic evaluation design. Separating evaluation into Memory QA, execution from ground-truth instructions, and execution from recalled instructions provides a useful diagnosis of different bottlenecks in the end-to-end system. The edit-distance-based metrics are particularly valuable because relying only on success rate may hide unexpected or incorrect behaviors that still result in task completion.

  3. Broad baseline and model evaluation. The authors reimplement two widely used memory methods and evaluate the approaches with three VLMs from different model families. This provides useful evidence about the relative strengths of visual and textual memory representations.

Critical weaknesses
  1. Limited comparison of memory writers. Both the MEM and MemER baselines use Gemini 3.7 Flash as the memory writer. It would be useful to evaluate additional VLM writers to determine whether the reported findings generalize across memory-generation models.

  2. Missing variance estimates. The paper reports averages over three independent runs but does not provide standard deviations or confidence intervals. These statistics are needed to assess the robustness of the reported differences. The paper should also report the number of queries represented in each cell of Tables 2 to 4.

  3. Memory and token budgets are not controlled. The paper should report the total memory budget for each method, including both textual tokens and visual inputs. A comparison under matched text, image, or total context budgets would help determine whether the R&R Agent gains come from a more effective memory architecture or simply from storing and retrieving more information.

  4. Important retrieval-augmented approaches are missing. The R&R Agent is closely related to retrieval-augmented embodied-memory approaches such as Embodied-RAG and ReMEmbR, but these methods are not discussed in the related-work section. Their relationship to the proposed approach should be clarified.

  5. Minor typographical issues. In Table 4, “M-/Obs.” in the Routine column should be “M-Obs.” In Figure 11, “R&R Bench” should be “R&R Agent.”

Questions for authors
  1. Could the authors report the standard deviation or confidence interval across the three runs, as well as the number of queries represented in each cell of Tables 2 to 4?

  2. What are the total memory and inference budgets of the evaluated methods, including textual tokens and visual inputs? Could the authors compare the methods under matched memory budgets?

  3. Could the authors discuss the relationship between the R&R Agent and retrieval-augmented embodied-memory approaches such as Embodied-RAG and ReMEmbR?

  4. In the Multi setting, how does performance change as the number nnn of target objects increases? For example, how does performance vary for instructions of the form: “Pick up nnn of the objects from the source and place them in or on the destination”?

3 comments

Response to Reviewer wbmu

We thank the reviewer for the thorough and encouraging review, and for recognizing that R&R-Bench fills an important gap in mobile manipulation memory evaluation, that the three-level evaluation provides a useful diagnosis of bottlenecks, and that the broad baselines offer useful evidence on visual and textual memory.


W1. Generalization across memory writers

Thank you for this helpful suggestion. We agree that evaluating additional memory writers provides useful evidence on whether our findings depend on a particular memory-generation model. We therefore repeated Memory QA using GPT-6.1 Terra, a model from a different family with a similar cost, as both the writer and the reader (Appendix C.9, Table 12). Gemini 3.7 Flash results are shown in parentheses.

MethodFindingRestoreResumeRoutine
MEM$^R$43.9 (34.0)22.8 (11.5)55.3 (46.0)61.5 (54.2)
MemER$^R$48.2 (44.1)21.8 (16.0)34.2 (30.4)32.5 (28.7)
R&R Agent50.5 (46.6)24.7 (19.0)54.0 (52.8)64.0 (60.6)

GPT-6.1 Terra achieves higher accuracy for every method and task, but the relative patterns remain the same: MemER$^R$ leads MEM$^R$ on Finding, MEM$^R$ leads MemER$^R$ on Resume and Routine, and R&R Agent is best on Finding, Restore, and Routine while staying close to MEM$^R$ on Resume (54.0% vs. 55.3%). This is consistent with our main finding that visual memory is more effective for recalling observed information and textual memory for progress tracking and rule inference, indicating that the finding is not specific to one memory-generation model. We will extend this analysis to additional writers in the final version.


W2 & Q1. Variance estimates and query counts

Summary (details in Common Response 2):

  • We now report the number of queries for every column of Tables 2 to 4.
  • Mean ± s.d. over three runs for Tables 2 to 4 is shown in Figure 13 (Appendix C.8).
  • We ran additional full-pipeline Routine experiments so that Table 4 uses the same queries as Table 2 for all tasks.

W3 & Q2. Memory and inference budgets

Summary (details in Common Response 1): All methods share the same reader budget of at most 1,200 text tokens and eight frames, and R&R Agent uses fewer text tokens than MEM$^R$. Under matched or smaller budgets, its gains hold: the text-only variant matches or exceeds MEM$^R$ (Table 5), and with four frames it outperforms frame-based baselines given eight (Appendix C.6).

W4 & Q3. Relationship to retrieval-augmented embodied memory

Thank you for pointing out these closely related works. We agree that clarifying this relationship better positions R&R Agent, and we have added Embodied-RAG and ReMEmbR to the related work (p. 3). R&R Agent shares their core idea of storing long experience and retrieving it on demand, and we do not claim retrieval itself as our contribution. R&R Agent is a reference design that incorporates the findings of our benchmark analysis. The main difference lies in what the memory must support. These methods are designed for navigation, where memory mainly records where things were observed. In mobile manipulation (our setting), the agent also changes the environment, so memory must also capture when each object state held and what the agent itself did. R&R Agent therefore stores timestamped textual notes on actions and state changes together with visual evidence, which Table 5 shows to be complementary, and keeps updating its memory during execution to track completed targets and subtasks in Multi, Resume, and Routine requests.


Q4. Performance by the number of target objects

Thank you for this insightful question. We agree that analyzing performance by the number of targets offers a useful view of how memory demands scale in Multi requests. We therefore broke down execution with ground-truth instructions (Table 3) by the number of targets n. With the fine-tuned VLA, however, multi-object execution approaches zero success for every method, which makes it difficult to see how memory itself scales with n. We therefore also report the same breakdown with an oracle manipulator (Appendix C.11, Table 14): a magic gripper that automatically manipulates the target when the robot attempts a grasp (place or open/close) near it (about 99% manipulation success), while navigation and subgoal decisions remain unchanged.

Table R2. Execution success (%) with ground-truth instructions by number of targets, with the fine-tuned VLA.

TasknMEM$^R$ (VLA)MemER$^R$ (VLA)R&R (VLA)
Finding150.045.648.9
221.118.423.7
30.04.54.7
Restore131.826.127.3
22.92.85.7
30.00.00.0
ResumeSingle41.930.141.1
Multi11.56.611.6
RoutineSingle25.023.827.5
Multi0.00.00.0

Table R3. Execution success (%) with ground-truth instructions by number of targets, with the oracle manipulator. Finding and Restore use 30 queries per n, sampled separately from Table 3.

TasknNo Mem. (Oracle)MEM$^R$ (Oracle)MemER$^R$ (Oracle)R&R (Oracle)
Finding1100.0100.0100.0100.0
265.675.672.274.4
345.668.955.674.4
Restore174.487.278.990.0
253.378.973.376.7
328.959.150.057.1
ResumeSingle35.752.035.749.7
Multi15.324.021.926.2
RoutineSingle50.075.860.478.3
Multi17.137.234.142.6

The results reveal three trends:

  • Low-level failures compound as n grows. With the VLA, success at n = 3 is at most 4.7% on Finding and 0% on Restore, and no method completes any Multi Routine request. With the oracle manipulator, success remains substantial at n = 3 (45.6% to 74.4% on Finding).
  • Execution-time memory becomes more important as n grows. With the oracle manipulator, the gap between memory-based methods and No Memory widens with n. This supports our claim that Multi requests require online memory updates to track completed and remaining work (Section 3.2).
  • Textual progress records scale best. At n = 3, the two methods that keep textual records, MEM$^R$ and R&R Agent, retain more success than frame-based MemER$^R$ on Finding (68.9% and 74.4% vs. 55.6%) and Restore (59.1% and 57.1% vs. 50.0%), and the same holds for Multi Resume and Routine. This matches our main finding that textual memory is more effective for progress tracking. Since this setting provides ground-truth instructions, it isolates execution-time memory, where MEM$^R$ and R&R Agent perform comparably. The advantage of R&R Agent in recalling the goal is reflected in Tables 2 and 4.

W5. Typographical issues

Thank you for catching these. We have corrected "M-/Obs." to "M-Obs." in Table 4 and "R&R Bench" to "R&R Agent" in Figure 11.

Thank you for your responses. I remain positive and I will keep my rating.

Reviewer_EjzUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

R&R-Bench contains 300 recorded contexts across 228 simulated houses and 1,772 queries covering object finding, restoration, task resumption, and household rules. It evaluates memory QA, execution with correct goals, and the full pipeline, with a reference agent combining text and visual memory.

Critical strengths
  1. Validated histories support several initially unknown requests, including incidental observations, unfinished work, and rules applied to held-out objects. This makes the benchmark a useful evaluation resource (pp.15–18).

  2. The breakdowns reveal failures hidden by aggregate success. For R&R with Gemini3.7Flash, current and past locations are correct in 41.7% and 35.0% of 575 Restore queries, but both in only 19.0% (Table 8).

Critical weaknesses
  1. Retention, storage, retrieval, and action budgets are not fully specified. Small gains are therefore hard to separate from differences in available information (pp.18–21,26–27).

  2. The link between recall and execution remains unclear. Routine uses different QA and execution query sets, and per-method execution rates conditioned on correct QA are not reported. These aggregate comparisons do not isolate memory failures (pp.6,20,22).

Questions for authors
  1. Please provide per-method memory and action budgets. Do the main comparisons hold with comparable retained evidence and reader access?

  2. Can existing logs provide execution success conditioned on correct QA for matched query sets, with sample counts and uncertainty?

3 comments

Response to Reviewer EjzU

We thank the reviewer for the careful review, and for recognizing that the validated histories make R&R-Bench a useful evaluation resource and that our breakdowns reveal failures hidden by aggregate success.


W1 & Q1. Memory and action budgets

Summary (details in Common Response 1): All methods share the same reader budget and action budget. Retention differs by design, but reader access is matched, and under matched access the main comparisons hold: the text-only R&R Agent matches or exceeds MEM$^R$, and R&R Agent with four frames outperforms frame-based baselines given eight.


W2 & Q2. Linking recall and execution

Thank you for this insightful question. We agree that linking recall and execution more directly helps isolate memory failures.

Matched query sets. We ran additional full-pipeline Routine experiments, so Table 4 now uses the same queries as Memory QA (Table 2) for all four tasks.

Execution success conditioned on correct QA. Using the existing logs on these matched sets, we computed execution success among correctly answered queries for each method, with sample counts and 95% Wilson intervals (Table R1).

Table R1. Full-pipeline success (%) among queries with correct QA answers (Gemini 3.7 Flash, first of three runs). Cells: success rate [95% Wilson interval] (number of correctly answered queries).

TaskGT Action LogNo MemoryUniform FramesMEM$^R$MemER$^R$R&R Agent
Finding44.7 [39.5, 50.1] (333)52.3 [41.9, 62.6] (86)56.6 [50.5, 62.6] (256)51.6 [45.4, 57.8] (244)52.5 [47.0, 58.0] (316)53.0 [47.6, 58.3] (334)
Restore17.7 [13.6, 22.7] (266)50.0 [9.5, 90.5] (2)21.8 [12.9, 34.4] (55)24.2 [15.5, 35.8] (66)22.8 [15.4, 32.4] (92)30.3 [22.4, 39.5] (109)
Resume20.5 [15.0, 27.4] (161)0.0 [0.0, 25.9] (11)9.5 [3.8, 22.1] (42)18.9 [11.6, 29.3] (74)18.4 [10.0, 31.4] (49)21.2 [13.8, 31.0] (85)
Routine18.7 [14.3, 24.2] (235)21.7 [9.7, 41.9] (23)24.6 [16.0, 36.0] (69)15.5 [10.8, 21.7] (168)22.5 [15.0, 32.2] (89)18.6 [13.7, 24.8] (188)

Since each method answers a different subset of queries correctly, these rates are not fully comparable across methods. For example, GT Action Log has the best recall but the lowest rate on Finding (44.7%), as it also answers harder queries. Queries answered correctly by every method are too few for a reliable comparison (85, 37, 11, and 9 for Finding, Routine, Restore, and Resume).

Nevertheless, two patterns stand out. First, an incorrect answer rarely leads to success (Appendix C.7), so correct recall is a prerequisite for execution. Second, even after correct recall, Restore and Resume, which require tracking states and remaining work during execution, still show differences between methods, with R&R Agent highest among experience-based methods (30.3% and 21.2%). This is consistent with the role of execution-time memory shown in Table 3.

Controlled comparison. For a fully controlled comparison of execution, Table 3 gives all methods the same ground-truth instructions on shared queries, which removes recall by design.

Thank you for the additional experiments and clarifications. Matching the QA and full-pipeline query sets, specifying the reader and action budgets, and reporting execution success conditioned on correct QA address the main issues in my review. I remain supportive of acceptance and will retain my score of 3 (Weak acceptance).

Table R1 is particularly useful for showing how much execution difficulty remains after correct recall. I agree with your qualification that the conditional rates use different correctly answered subsets across methods. The shared ground-truth-instruction setting therefore remains important for comparing execution. The reader-budget results also strengthen the comparison, while retention remains part of the memory designs being evaluated.

Two reporting details would benefit from clarification. Does the set of queries answered correctly by “every method” exclude No Memory? The reported intersections contain 11 Restore and 37 Routine queries, whereas No Memory answers only 2 and 23 correctly, respectively. Please specify which methods enter this calculation. Also, please explain the difference between the 314 Routine queries in the benchmark description and the 310 used in Tables 2 and 4.

These clarifications would make the new analysis easier to interpret. The revision strengthens my positive assessment of the benchmark and its diagnostic value.

Thank you for the follow-up and for your continued support. We apologize for the confusion caused by these two reporting details.

Intersection. The intersection excludes No Memory and GT Action Log. It is computed over the four experience-based methods: Uniform Frames, MEM$^R$, MemER$^R$, and R&R Agent.

Routine query count. The query-time visibility check (Section 3.3) removed four Routine queries whose answers were directly visible in the query-time observation, so they could be answered without memory. All results use the remaining 310 queries, but the benchmark description was not updated after this filtering. We have corrected the counts to 310 Routine queries and 1,768 queries in total throughout the revised manuscript.

We are happy to address any further questions.

Comments on the paper

1 comment

General Response to All Reviewers

We thank all reviewers for their constructive feedback. We are encouraged that the reviewers found that R&R-Bench fills an important gap in mobile manipulation memory evaluation (wbmu), serves as a useful evaluation resource (EjzU, WYir), and provides a strong diagnostic design that separates recall from execution (wbmu, WYir).

All revisions are highlighted in blue:

  • Related work: Embodied-RAG and ReMEmbR (p. 3).
  • Statistics: Query counts for all columns of Tables 2 to 4; mean ± s.d. over three runs (Appendix C.8, Figure 13).
  • Matched query sets: Additional full-pipeline Routine runs, so Table 4 now uses the same queries as Table 2.
  • Budgets: Memory and action budgets (Appendix B.5, Table 8).
  • New experiments: A different memory writer (Appendix C.9), success by number of targets (Appendix [C.11]), a magic gripper that removes manipulation failures (Appendix [C.11]), and end-to-end memory VLAs (Appendix C.10).
  • Typos: Table 4 and Figure 11.

Shared questions are answered below, and reviewer-specific questions are answered in each thread.


CR1. Memory and action budgets (EjzU, wbmu)

Memory budget. All methods share the same reader budget of at most 1,200 text tokens and eight frames per query (Appendix B.5). In practice, R&R Agent uses fewer text tokens than MEM$^R$ (149 vs. 208 on average). Retention differs by design: R&R Agent keeps an append-only database, MEM$^R$ rewrites a summary, and MemER$^R$ replaces frames. Since what to retain is part of the memory design being compared, we keep this difference and match what the reader can access instead. Under matched reader access, the gains of R&R Agent hold:

  • Its text-only variant matches or exceeds MEM$^R$ on every task (Table 5).
  • With only four frames, it outperforms Uniform Frames and MemER$^R$ given eight frames on Finding (45.0% vs. 35.7% and 44.1%) and Restore (17.2% vs. 9.6% and 16.0%) (Appendix C.6).

Action budget. All methods share the same budget in simulator steps (Table 8): 1,500 per target for Finding, 5,000 plus 3,500 per additional object for Restore and Routine, and max(400, 1.5 × oracle steps) for Resume. It leaves enough steps to complete feasible requests but not enough to succeed only after many repeated failures, and outcomes are usually settled before the limit.


CR2. Statistical reporting (wbmu, WYir, EjzU)

We agree that uncertainty estimates and sample sizes are needed to assess the reported differences, and we have made the following revisions.

  • Query counts. The number of queries is now reported for every column of Tables 2 to 4.
  • Variability. Figure 13 (Appendix C.8) reports mean ± s.d. over three runs for the task averages of Tables 2 to 4. We present them as a figure to keep the main tables readable. The s.d. is small for Memory QA and larger for the two execution settings, which also include variation from VLA execution. With the oracle manipulator (Appendix C.11), the s.d. reported in Figure 14 is smaller than with the VLA. Under this lower-variance setting, the conclusions hold and become clearer. With ground-truth instructions, which isolate execution-time memory, memory-based methods outperform No Memory by up to 27.4 points, and R&R Agent achieves the highest success on Finding and Routine while staying within 0.6 points of MEM$^R$ on Restore and Resume (Table 14). In the full pipeline, R&R Agent achieves the highest success among methods without the GT Action Log on all four tasks (Table 15).
  • Query sets. We ran additional full-pipeline Routine experiments, so Table 4 now uses exactly the same queries as Memory QA (Table 2) for all tasks, allowing recall and execution to be compared on identical queries. Execution with ground-truth instructions (Table 3) does not depend on recalled answers, so its queries are sampled independently: 100 Single and 50 Multi queries for Finding and Restore, all 161 Resume queries, and 123 Routine queries (80 Single and 43 Multi).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.