acceptodds
Under review as a conference paper at ICLR 2027

What Should Embodied Agents Remember? Evaluating and Benchmarking Memory Modality for Embodied Agents

96 views

Abstract

Memory allows large language model (LLM) agents to use past experience, often stored as textual records, when responding to requests. For embodied agents that control robots, visual and spatial observations introduce a broader choice of memory modalities, affecting both the information available for subsequent tasks and the model inference costs of constructing and using memory. We introduce MeMoREval, a benchmark for studying this problem through a controlled experience-request protocol. Agents first experience a sequence of events without knowing the subsequent tasks, and subsequently they use memory to respond to information and action requests without receiving new observations. The benchmark comprises 53 tasks across four simulated indoor environments and measures task success alongside the inference costs of memory construction and use. Our reference evaluations reveal that storing frames outperforms storing generated descriptions at lower write-time inference cost for the primary model, while storing frames with scene graphs achieves the highest aggregate task success for every tested model. Extensions to accumulated experience, selective frame retention, and visual input to memory writers demonstrate how the benchmark accommodates further memory-design investigations. Two physical-robot demonstrations establish the feasibility of applying its protocol beyond simulation.

Primary area:
datasets and benchmarks
Keywords:
Embodied agent, agent memory, multimodal memory, benchmark, LLM agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xQ5gOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MeMoREval, a benchmark for studying what forms of past experience embodied agents should retain. A fixed experience phase precedes previously unknown information and action requests. Each request starts from a fresh context, and action plans execute without new observations, reducing the opportunity to recover missing memory through renewed exploration. The benchmark comprises 53 tasks in four simulated environments and compares textual memory, three description budgets, stored frames, and frames plus scene graphs under shared memory interfaces. The primary-model experiments find that frames outperform descriptions, while frames plus graphs achieve the highest average success among the tested conditions. Separate write/read inference costs, stacked histories, writer variants, and two physical-robot demonstrations complement the main comparison.

Critical strengths
  1. The protocol and controls are well aligned with the research question. Fixed experiences, unknown future requests, fresh contexts, and the absence of new request-time observations help separate memory evaluation from exploration. The no-memory and full-information controls providing useful checks of memory dependence and solvability for the primary model .
  2. The comparison between stored frames and generated descriptions is carefully controlled.
Critical weaknesses
  1. The contribution beyond existing embodied-memory evaluations is limited. FindingDory already separates scripted experience collection from subsequent tasks and includes visual-history and text-summary agents . GraphEQA combines images and scene graphs as memory, while WorldLines evaluates memory QA and downstream planning . The submission cites these works. Its more distinctive contribution is the tighter control of writer inputs and retrieval interfaces, together with observation-free requests and separate inference-cost accounting. These are useful refinements, but I am not yet convinced that the resulting evaluation capability constitutes a sufficiently substantial advance.
  2. The empirical findings provide limited explanation beyond configuration rankings. Retaining visual details that task-agnostic descriptions may discard, and supplying explicit spatial records, have clear reasons to help. Quantifying these benefits is useful, but the category breakdowns do not sufficiently distinguish evidence loss, retrieval failure, reader interpretation, and planning errors . The reversal between frames and descriptions on GLM, and the inconsistent benefit of visual writer input, could yield more informative conclusions if their causes were analyzed. Similarly, lower write-time inference cost for frames follows in part from omitting description-generation calls while excluding storage costs. The reduction in read cost per successful request is a meaningful empirical result, but it does not establish the preferred modality under storage constraints or mapping costs. The stated cost scope is clear; the concern is how much new design guidance these results provide.
Questions for authors
  1. Does the frame–description gap mainly reflect information loss during captioning or differences in how the reader uses evidence?
  2. Do frames and scene graphs provide complementary information, or largely encode the same facts differently?
  3. How dependent are the findings on unconstrained storage and the prohibition on new observations?
  4. What explains the GLM ranking reversal, and when should descriptions be preferred over frames?
4 comments

Thank you for the detailed review. We appreciate your insight on classifying different failure modes, and performing such an analysis was the main focus of our revision. With additional failure attribution analyses, we find that our controlled evaluation protocol lends itself to useful conclusions that support design guidance for memory systems. In the following responses, intervals are 95% paired bootstrap intervals over tasks.

W2 / Q1 / Q4. New analysis: where each memory type fails, and resulting design guidance.

We assigned each failed run to the first stage it failed:

  1. Evidence loss: the stored information does not contain any fact required by the task.
  2. Missed retrieval: a piece of task-necessary information is stored, but the agent does not retrieve it.
  3. Reader error: a piece of task-necessary information is retrieved, but the agent cannot produce the correct response to the request.
  4. Planning/execution: the information request succeeds, but the action plan fails.
  5. Other: any failure not captured through this process.

The result of the analysis is shown in Fig. 5 in the revised paper and discussed in §5.2.1. We randomly examined 5% of all rule-based classifications and found 100% agreement. The results reveal some recurring patterns across all three models. Descriptions cause evidence loss failures, and frames have minimal evidence loss failures but cause more reader errors. Scene graphs make frames easier to interpret and thus reduce downstream errors, which is why frames+graph have the highest success on every model. The table below summarizes Fig. 5 with error counts by category across three models and the three backends run on all models. Gemini values are means across three seeds, and GLM and GPT use seed 0.

Failed runs, desc@1024 → frames → frames+graphGeminiGLMGPT
Evidence loss23.7 → 2.3 → 2.736 → 13 → 940 → 3 → 2
Missed retrieval8.7 → 8.3 → 10.37 → 17 → 1810 → 13 → 15
Reader error13.3 → 20 → 14.726 → 37 → 3216 → 37 → 33
Planning/execution10.7 → 13 → 1115 → 22 → 1411 → 16 → 16
Other3.7 → 3.7 → 2.74 → 5 → 73 → 3 → 3
All failures60 → 47.3 → 41.388 → 94 → 8080 → 72 → 69
Evidence loss, desc@256 → desc@102422 → 23.737 → 3638 → 40

Design guidance, description vs. frames.

Descriptions appear to be preferable when a model loses less by captioning an image than by reading it later in the context of the task (Fig. 5). For GLM, frames save 23 evidence-loss failures but add 29 downstream failures, so descriptions achieve better performance overall. For Gemini and GPT the saving is larger than the added reader errors, so frames are better than descriptions. However, descriptions have fundamental limitations that cannot be addressed by a stronger model, and we provide further explanations for this next. The failure attribution also reveals that increasing description length does not reduce evidence-loss failures, a reflection of the inherent difficulty with storing details when the future task is unknown. With frames being stored, the remaining failures are mostly due to missed retrieval, reader errors, or planner errors. Any remaining evidence-loss failures is due to the task-necessary fact being present in the textual observation but not recorded by the model at writing time, and GLM has a higher remaining evidence-loss failures than the other models. As model capabilities grow, remaining retrieval, reader, and planner errors will likely see further reduction, and therefore the forward-looking guidance would be to store frames and scene graphs when cost is not a concern, as graphs help with the interpretation of frames, and the combination of these modalities is likely the first to see near-zero overall failure with improved model capabilities. We have updated §5.2.1 and §5.2.2 with these design guidelines.

Q2. New graph experiments and failure analysis show that frames and graphs are complementary.

We demonstrate that frames and graphs encode complementary information through two fronts:

  • Our new experiment with the graph condition shows that scene graphs alone offer limited improvement over the textual baseline, and frames+graph exceeds graph by 25.7% for Gemini [15.3, 36.6], 21.4% for GLM [11.3, 31.4], and 16.4% for GPT [7.5, 25.8]. The bootstrap interval is positive in every backend.
    • Qualitatively, scene graphs hold object identities, 3D positions, and support/containment, but not appearance or visual order, which come from visual perception. For Gemini, switching from graph to frames+graph led to +58% in spatial configuration, +49% in appearance binding, and +22% in object location.
  • Our failure attribution reveals that graphs help models use frames at the time of memory query by lowering reader failures for all three models (Gemini 20 → 14.7, GLM 37 → 32, GPT 37 → 33 when going from frames → frames+graph).

W1. Our contribution beyond previous works.

We acknowledge that components of our work have semblance with other prior works on memory evaluation. The most distinctive aspect of our work is that MeMoREval isolates the effects of the modality of stored information. In FindingDory, the text and visual agents differ in input coverage and format, GraphEQA builds memory during active exploration while conditioned on the future question, and WorldLines only involves text-only backends, so none can attribute differences in success or cost to the stored representation. MeMoREval fixes the recorded experience, writer input, reader, and retrieval interface and varies only the stored modality (text, graph, descriptions at three budgets, frames, frames + scene graph) across multiple memory backends, testing each fact with both a question and an executed manipulation, and measuring write- and read-time cost separately. This makes two kinds of attribution possible:

  • Error attribution by modality. Each failure is traced to the stage where it occurred (Fig. 5). Descriptions fail because evidence is lost when they are written, frames lead to reader failures in interpretation, and adding scene graphs reduces reading errors. These are results about the modalities, not about one system, and are made possible by our design.
  • Cost attribution by modality. Write and read cost are measured separately for the same agent, so a modality’s cost is not mixed with the cost of a different pipeline. This mechanism allowed us to make the slightly surprising observation that passing more information (e.g. frames) to the reader actually reduces reasoning token usage and tool call frequency, hence is not more expensive than passing less information (e.g. short descriptions). The outcome is that MeMoREval provides not only a ranking, but also guidance on what to improve: captioning for descriptions, retrieval and reading for frames, and we can also evaluate any new model or any new memory backend through the same failure attribution.

W2 / Q3. Cost, closed-loop execution, and writer images.

Costs.

We acknowledge that a complete robotic system involves costs at many different stages. In MeMoREval, we report model inference cost because it is the cost most directly incurred by memory design. We have added an explanation on the other costs in §5.1, and we provide some details below.

  • Storage. The storage footprint is dominated by one 640×480 frame per skill step. Each frame is under 1 MB even uncompressed, so a robot taking 1,000 steps a day sums to less than 1 GB a day, which is small for on-board or cloud storage, and a fraction of the storage footprint of videos. Keeping only writer-selected frames (Fig. 6a) reduces this further at a cost of 4–17% drop in success, and the 8–55% of frames that are kept still match or exceed full descriptions in each backend.
  • Mapping and sensing. We consider robot-controlling embodied agents with navigation and manipulation capabilities. For a robot that navigates within an environment, it internally maintains a 3D object-level map (e.g. through SLAM techniques), and for a robot that can manipulate objects, it is necessary to have visual perception. Therefore, the memory system reuses the spatial map and visual perception from the robot’s existing capabilities.

No new observations.

We deliberately make the design of not allowing new observations during requests, and you have listed it as a strength of our design. If the agent could observe the environment again at request time, success would measure the agent’s ability to explore and recover, not the content of its memory.

Writer images.

Whether the memory writer benefits from seeing images and the extent of such benefits depend on the memory structure. We observed that across the four writer backends, the change in task success is minor and not uniform (range of −9% to +3%), and the write-time cost sees both increases and decreases for different backends (Table 8). Since this effect is dependent on memory backends, and this paper focuses on memory modality evaluation and not one particular memory system design, we only draw the conclusion that the benchmark provides the setting to study this effect.

AuthorsOpenReview ↗

Follow-up on your review

Dear Reviewer xQ5g,

Thank you once again for your detailed and insightful review. Your suggestion to separate failure stages was the centerpiece of our revision and helped turn our empirical rankings into actionable design guidance.

In the revised paper, we added a stage-by-stage failure attribution analysis across all models and conditions, explaining the GLM ranking reversal and showing that descriptions suffer from persistent evidence loss while scene graphs reduce the reader errors introduced by frames. We also added the new graph (text + scene graph) condition across all models and backends—demonstrating strong complementarity between frames and scene graphs—clarified how isolating the stored modality enables both failure and cost attribution beyond prior benchmarks, and expanded our discussion of storage, mapping, and writer-image costs.

We believe these updates directly address the concerns behind your initial evaluation and clearly establish the paper's contribution. As the discussion phase comes to an end soon, we would be very grateful for your thoughts on the revision and whether our responses have addressed your concerns. Please let us know if you have any remaining questions we can answer.

Best regards,

The Authors

Dear Reviewer xQ5g,

As the discussion period draws to a close, we wanted to thank you again for encouraging us to look beyond aggregate rankings. Your insightful feedback led to our development of a failure attribution pipeline for our benchmark, showing that descriptions fail mainly from evidence loss, raw frames trade evidence loss for higher reader errors (explaining the GLM reversal), and scene graphs help all models interpret frames and reduce reader errors. MeMoREval thus isolates the stored modality to diagnose whether failures stem from evidence retention or downstream multimodal reasoning.

We would greatly value your thoughts on these revisions and are happy to answer any remaining questions before the discussion closes today.

Best regards,

The Authors

Reviewer_JziNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MeMoREval compares memory formats before future requests are known. Its 53 tasks span four simulated scenes. Agents answer questions or execute open-loop plans from shared text plus captions, frames, or frames+graph, with separate write/read inference costs (pp.4–6).

Critical strengths
  1. Shared observations, models, textual memory, and step-based access reduce confounds; captions and frames bypass further writer compression (pp.4–6).

  2. Compared with captions, frames improve primary-model overall task success by 9.3 percentage points (95% interval: [4.5,14.5]). Action success is 77% versus 71%; weighted read tokens per successful action request fall from 13.6k to 12.7k, including image costs. GLM reverses the ordering (pp.14–15, Tables 2–5). Costs exclude sensing, mapping, and storage.

Critical weaknesses
  1. Missing text+graph leaves frames’ incremental value given spatial records unresolved; graph retrieval already returns text (pp.6,18).

  2. Four fixed scenes and recurring templates limit robustness; repeated runs do not add scenes, and staged, same-model video/text comparisons already exist (pp.14–19).

Questions for authors
  1. Can you report matched-budget text+graph success, inference costs, and category effects?

  2. Does the frame advantage survive per-scene/template analysis and a detail-focused caption prompt?

4 comments

Thank you for the precise review. We performed specific analyses for both of your questions and report the results below. In the following responses, intervals are 95% paired bootstrap intervals over tasks.

Q1 / W1. New experiment: matched-budget graph runs. Frames+graph adds 16–26% task success on top of graph.

In this new experiment, graph stores the textual memory and scene-graph snapshots with no frames or descriptions, under the same limits as frames+graph (context limit, six turns per request, six objects per graph call). The following table gives the detailed breakdown of performance metrics with Gemini 3.7 Flash, across all tasks, backends, and three seeds:

MemoryTask successInformation requestsAction requestsRead $ per taskRead $ per successful task
text39.4%53.0%51.4%0.0410.104
graph (new)49.1%59.7%66.8%0.0650.133
desc@102460.3%73.2%71.0%0.0530.088
frames69.6%81.5%76.5%0.0520.075
frames+graph74.7%84.9%81.7%0.0590.079

Success metrics.

  • Frames+graph exceeds graph by +25.7% [15.3, 36.6].
  • The result holds in each of the five backends (+19.5 to +30.2), including the transcript backend, which has no LLM memory writer.
  • The result also holds on GLM (+21.4% [11.3, 31.4]) and GPT (+16.4% [7.5, 25.8]).

Cost metrics.

  • At write time, graph does not add inference cost, as scene graph snapshots are archived without model calls.
  • At read time, graph is the most expensive condition per successful task ($0.133 vs $0.079 for frames+graph). Compared to frames+graph, the agent makes more graph retrievals (3.2 vs 1.4 per task).

Task success (%) by category

Category (# of tasks)textgraphframes+graph
Appearance binding (10)202069
Spatial configuration (11)41977
Object location (11)757698
Occluded state (4)174353
User preference (5)616371
Set aggregation (8)586860
Skill outcome (4)458273
  • Graphs alone improve performance the most in skill outcome and occluded state.
  • Adding frames contribute most for appearance and spatial configuration, which require information not recorded in scene graphs or are hard to understand without images.

Q2 / W2. New analysis: the frame advantage survives every leave-one-scene-out and leave-one-task-family-out.

We analyzed the robustness of the gap between frames and desc@1024, with a new table added to Appendix A. The table below summarizes the results:

CheckResult
Per sceneoffice +16.1, studio +10.5, clinic +9.7, home +1.9
Leave one scene out+7.3 to +12.0
Leave one template task family out+7.5 to +10.3

As reflected above, the gap is not driven by a single scene or template task family, signifying that frames offer meaningful improvement over descriptions.

Caption prompt. Our describer prompt is already detail-focused, as it asks for exact, complete descriptions of objects, attributes, and arrangement (the prompt is added to Appendix B for reference). Longer descriptions don’t improve recall, as the performance is roughly constant from 256 to 1024 tokens on all three models (Gemini +0.8 [−2.1, 3.8], GLM +1.9 [−3.8, 8.2], and GPT +2.5 [−2.5, 7.5]; no interval excludes zero), and the failure analysis shows no consistent reduction in evidence loss (Fig. 5): Gemini 22 → 23.7, GLM 37 → 36, and GPT 38 → 40 failed runs. Since hand-written oracle memory solves 53/53 tasks, textual modality has the capacity to encode relevant information, but the difficulty is choosing what information to retain before that request is known.

Existing staged video/text comparisons. We agree that FindingDory and OpenEQA also compare textual and visual memory, but neither isolates the stored representation. In FindingDory, the text agent summarizes the full video into fixed-schema chunk captions, while the VLM agents receive subsampled frames, so the comparison varies coverage, format, and model together. OpenEQA evaluates captions, scene-graph text and frames as separate alternatives at a single caption length, and reports no inference cost. Both works place content directly in the model context rather than in a memory system. We instead hold the experience, writer, reader and retrieval interface fixed across memory backends, vary description length, combine frames with scene graphs, and report write- and read-time cost separately. This lets us attribute differences in success and cost to the stored representation rather than to the surrounding pipeline, as well as provide meaningful failure categorization. We have added this clarification in §2.

AuthorsOpenReview ↗

Follow-up on your review

Dear Reviewer JziN,

Thank you once again for your careful and constructive review, which pointed to analyses that substantially improved our paper. In our revision, we ran the full matched-budget graph (text + scene graph) experiment across all models and backends, confirming that frames+graph consistently outperforms graph at lower read cost per successful task. We also added per-scene and leave-one-out robustness analyses, included our detail-focused caption prompt, clarified how our controlled setup differs from FindingDory and OpenEQA, and added a failure attribution analysis showing that larger description budgets do not reduce evidence-loss failures.

Together, these additions establish both the incremental value of visual frames alongside spatial records and the robustness of our conclusions. As the discussion period comes to an end soon, we would love to hear whether our updates satisfactorily address your questions, and we remain happy to clarify anything further.

Best regards,

The Authors

Thank you for the additional experiments. The new text+graph control resolves my main concern about the value of frames when spatial records are already available. I remain supportive of acceptance and will retain my score of 3 (Weak acceptance).

For the primary model, adding frames raises task success from 49.1% to 74.7%, with a reported gain of 25.7 percentage points and a 95% paired-bootstrap interval of [15.3, 36.6]. The gains with the other two models strengthen this result. The cost and category breakdowns also make clear where adding frames helps and what it costs.

The per-scene and leave-one-out analyses show that the primary-model advantage over descriptions is not driven by a single scene or task family. Providing the caption prompt and the failure analysis also makes the comparison easier to assess, particularly the trade-off between preserving evidence and interpreting it correctly.

These additions strengthen the paper’s controlled evaluation and its usefulness for studying memory design. My remaining reservation is the breadth of the evaluation, which covers four scenes and 53 tasks. Within that scope, I find the revised evidence convincing.

Dear Reviewer JziN,

Thank you very much for taking the time to re-evaluate our revised paper and for your supportive assessment. We are very glad to hear that our additional experiments and analyses resolved your main concerns and that you find the evidence convincing within the benchmark's scope.

As robots move closer to long-term, real-world deployment, memory becomes a central bottleneck—not just retaining information, but deciding what representations preserve critical evidence at sustainable cost. We are deeply committed to this direction, and your feedback helped sharpen MeMoREval into a clearer foundation for studying those trade-offs. We look forward to releasing the benchmark and continuing to build toward capable, long-horizon embodied agents.

Thank you again for your thoughtful comments and support.

Best regards,

The Authors

Reviewer_tCJwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

this work introduces MeMoREval, a benchmark for evaluating memory modalities in embodied agents. It separates tasks into an experience phase and a request phase, requiring agents to answer questions or execute actions using previously stored memories. The benchmark evaluates task success as well as inference costs during memory construction and use. It contains 53 tasks across four simulated environments and 7categories of memory demands. Authors compare text, generated image descriptions, raw images, scene graphs, and combinations of these modalities, while also examining different memory backends, base models, and deployment-oriented variants. The results show that combining images with scene graphs achieves the highest aggregate task success across all tested models. For the primary model, storing images directly also achieves higher success and lower write-time cost than generating textual descriptions. Two real-robot demonstrations further illustrate the feasibility of the evaluation protocol.

Critical strengths
  1. Controlled evaluation protocol. Separating experience from requests, and preventing access to new environmental observations during the request phase, allows the experiments to examine more directly how memory contributes to subsequent information retrieval and action planning, while reducing confounds from exploration ability.

  2. Systematic experimental comparisons. The paper covers multiple memory modalities, 5 memory backends and 3 base models, an separately evaluates task success, write-time cost, and read-time cost. In particular, the direct comparison between images and generated descriptions provides useful empirical evidence for the design of memory representations.

Critical weaknesses
  1. Limited experimental coverage. The benchmark contains 53 tasks across four simulated environments, while the real-robot evaluation consists of only two demonstrations and does not systematically compare different memory modalities in the real world. The current evidence therefore provides limited support for strong conclusions about broader deployment scenarios.

  2. Limitations in external validity due to implementation choices. The authors state that A-MEM, Mem0, and ObsMem are adaptations to a shared interface rather than reproductions of the original systems. Changes to retrieval mechanisms and other implementation details may affect the results, so the conclusions should be interpreted as applying to the implementations evaluated in this paper. In addition, the cost analysis focuses primarily on model inference and excludes storage, sensing, and spatial-mapping costs, limiting its account of total deployment cost.

Questions for authors

see weaknesses

3 comments

Thank you for the careful review. We agree with both weaknesses as statements about scope. Below are our new analyses on the robustness of the findings. In the following responses, intervals are 95% paired bootstrap intervals over tasks.

W1. We address limited coverage with new robustness analyses.

As adding environments and running all conditions on new environments is too time-limited within the discussion period, we tested whether our conclusions depend on particular parts of the existing suite.

  • Frames exceed desc@1024 by +9.3% [4.5, 14.5], and
    • +7.3 to +12.0 with any one scene left out
    • +7.5 to +10.3 with any one of 14 families left out
  • With the new graph experiment, frames+graph exceeds graph by +25.7 (Gemini), +21.4 (GLM), +16.4 (GPT) with each interval above zero, so that frames+graph is the best configuration for each model.
  • On stacked tasks, the average ordering is unchanged from single tasks: text 25%, desc@1024 48%, frames 55%, frames+graph 60%. The transcript backend is the one expected exception that has degraded performance once the interaction logs exceed the context limit.

These checks demonstrate stability within the benchmark.

Real-world robot demonstrations. The benchmark addresses robot-controlling embodied agents that are capable of manipulating objects and navigating within a space. These capabilities require camera frames, object segmentation, and a spatial map as precondition, so a memory system can directly utilize these multimodal information streams. The demonstrations are intended to demonstrate the feasibility of constructing a memory system that runs on such a robot, and not to extend comparisons across different conditions to the real world.

W2. Results’ external validity despite implementation choices.

We acknowledge your concern and would like to provide three notes on why modality conclusions should be generalizable beyond our implementation choices:

  1. In terms of our retrieval interface, our new failure mode categorization reveals that retrieval errors do not contribute to failures as much as evidence loss and reader errors, with smaller variations between descriptions, frames, and frames+graph, so they do not contribute to the comparison across modalities as much as evidence loss and reader errors.
  2. The modality differences appear even in the transcript backend, which has no LLM writer and therefore none of our memory-system adaptations.
  3. Key comparisons that we draw hold in each backend separately, on all 53 tasks (full table added to Appendix A):
transcript (no writer)wikiA-MEMMem0ObsMem
frames+graph − frames+2.5+3.8+5.0+2.5+11.9
frames+graph − graph+30.2+19.5+30.2+24.5+23.9
frames − desc@1024+3.1+14.5+6.3+12.6+10.1

The paper does not draw conclusions about which backend is better, and we present the backends as the benchmark’s capability of supporting different memory structures and could enable evaluation of novel memory systems for embodied agents.

W2. Addressing storage, mapping, and sensing costs.

We acknowledge that a complete robotic system involves costs at many different stages. In MeMoREval, we report model inference cost because it is the cost most directly incurred by memory design. We have added an explanation on the other costs in §5.1, and we provide some details below.

  • Storage. The storage footprint is dominated by one 640×480 frame per skill step. Each frame is under 1 MB even uncompressed, so a robot taking 1,000 steps a day sums to less than 1 GB a day, which is small for on-board or cloud storage, and a fraction of the storage footprint of videos. Keeping only writer-selected frames (Fig. 6a) reduces this further at a cost of 4–17% drop in success, and the 8–55% of frames that are kept still match or exceed full descriptions in each backend.
  • Mapping and sensing. We consider robot-controlling embodied agents with navigation and manipulation capabilities. For a robot that navigates within an environment, it internally maintains a 3D object-level map (e.g. through SLAM techniques), and for a robot that can manipulate objects, it is necessary to have visual perception. Therefore, the memory system reuses the spatial map and visual perception from the robot’s existing capabilities.
AuthorsOpenReview ↗

Follow-up on your review

Dear Reviewer tCJw,

Thank you once again for your thoughtful and supportive review, which was very helpful in strengthening our work. During the discussion period, we revised the paper to address your concerns on coverage, external validity, and cost scope: we added leave-one-scene-out and leave-one-task-family-out robustness checks, verified that our modality rankings hold independently in every backend (including the writer-free transcript baseline), and clarified the scope of storage, sensing, and mapping costs. We also added a matched-budget graph condition across all models and backends, along with a failure attribution analysis showing that retrieval differences do not drive the modality gaps.

We believe these updates meaningfully reinforce the soundness and generalizability of MeMoREval's findings. As the discussion phase comes to an end soon, we would greatly appreciate hearing whether our responses and revisions have addressed your concerns, and we would be happy to answer any remaining questions.

Best regards,

The Authors

Thank you for the response. I decided to keep my score

Comments on the paper

2 comments

We thank the reviewers for their careful and constructive reviews. All three reviewers valued that our evaluation protocol allows differences in success and cost to be attributed to the memory modality itself. We have revised our paper (changes in blue) with a focus on elaborating the mechanisms that lead to the aggregate success by tracing each failure to the stage where it occurred, which shows what each modality trades off and where each can be improved. Below we summarize the main changes to the paper in more detail.

Failure attribution and resulting design guidance (Reviewer xQ5g)

We conducted an analysis to break down the root cause of all failed tasks. We found that the dominant failure mode for descriptions is that the memory does not contain relevant evidence for solving the task. Longer descriptions still leave a large gap from frames, which have minimal errors of this failure mode. However, frames are more difficult for the model to process and lead to higher reader errors, while adding scene graphs help with mitigating these errors. This results in the overall design guidance of storing frames over descriptions, and additionally scene graphs when performance is more important than cost. Details can be found in §5.2.1 and the response to Reviewer xQ5g. This sequential failure identification process is rule-based, as the necessary set of memorized facts per task is available by task design, and it will be included in the released benchmark so that failures can be attributed to root causes. We have updated §4.2 accordingly.

Note on GPT results. While attributing failures for the new Fig. 5, we found that some failed GPT description runs (95 of 477) were refusals from the content filter set by the provider, not agent errors. Because this does not reflect the model's capability, we reran those runs through a different provider. GPT desc@64/256/1024 is now 44/47/50% (originally 35/40/39%). The ordering of conditions for GPT is unchanged (frames+graph > frames > descriptions > text), and both statements the paper makes about GPT still hold: frames+graph has the highest success, and frames alone score higher than desc@1024 (+5.0 [−1.3, 11.3]). We checked every GPT run in every condition, and refusals occurred only in the description conditions, and no GPT run now contains one. Table 5 and Fig. 4 are updated, and every number in our responses uses the result after the reruns.

Extra modality: graph on all three models (Reviewers JziN, xQ5g)

We conducted a new experiment with another memory modality, scene graphs without images, in order to single out the contribution of frames when spatial records are available. All relevant plots and tables in the paper have been updated to include this new result. The table below shows the resulting task success (%) by condition. All intervals are 95% paired bootstrap intervals over the 53 tasks.

Modeltextgraph (new)desc@1024framesframes+graphframes+graph − graph
Gemini 3.7 Flash (5 backends × 3 seeds)39.449.160.369.674.7+25.7 [15.3, 36.6]
Gemini 3.7 Flash (seed 0)39.646.561.669.872.3+25.8 [13.8, 37.7]
GLM 5.3 Flash (seed 0)32.728.344.740.949.7+21.4 [11.3, 31.4]
GPT 5.6 Sol (seed 0)35.840.349.754.756.6+16.4 [7.5, 25.8]
  • Frames+graph still shows the highest success for every model, and its gap over graph is positive in each of the five backends (+19.5 to +30.2), including the transcript backend, which has no LLM memory writer.
  • Over frames alone, graphs add +5.2 [1.5, 9.4] for Gemini, +8.8 [1.9, 15.7] for GLM, and +1.9 [−5.0, 8.8] for GPT.

Robustness of our conclusions (Reviewers tCJw, JziN)

We investigated whether our claimed advantage of frames being better-performing than descriptions, through per-scene, per-task-family, and per-backend analyses. Results show that the frame advantage survives every leave-one-out test (+7.3% to +12.0%).

Dear Area Chair and Reviewers,

As the author-reviewer discussion period draws to a close, we would like to take this chance to thank you all for the time and constructive feedback you brought to our submission throughout the review and discussion process.

Guided by your questions, we made substantial additions to the paper during this period: a failure attribution mechanism for all failed tasks, a matched-budget graph control across all models and backends, robustness checks, clearer scope, and prior-work discussions. Together, these updates strengthened both our empirical conclusions on configuration rankings as well as the causes behind them, specifically demonstrating that isolating stored memory modality reveals details such as generated descriptions lose critical evidence, raw frames shift failures to multimodal reading, and pairing frames with scene graphs consistently helps embodied agents succeed across models and memory backends.

As robots with top-level agentic orchestrators move closer to long-term, real-world deployment, deciding what these agents should remember at sustainable cost becomes fundamental. Your feedback helped turn MeMoREval into a stronger benchmark for studying those trade-offs, and we are excited to release the benchmark and code to support future work in this direction.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.