- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces MeMoREval, a benchmark for studying what forms of past experience embodied agents should retain. A fixed experience phase precedes previously unknown information and action requests. Each request starts from a fresh context, and action plans execute without new observations, reducing the opportunity to recover missing memory through renewed exploration. The benchmark comprises 53 tasks in four simulated environments and compares textual memory, three description budgets, stored frames, and frames plus scene graphs under shared memory interfaces. The primary-model experiments find that frames outperform descriptions, while frames plus graphs achieve the highest average success among the tested conditions. Separate write/read inference costs, stacked histories, writer variants, and two physical-robot demonstrations complement the main comparison.
Critical strengths
- The protocol and controls are well aligned with the research question. Fixed experiences, unknown future requests, fresh contexts, and the absence of new request-time observations help separate memory evaluation from exploration. The no-memory and full-information controls providing useful checks of memory dependence and solvability for the primary model .
- The comparison between stored frames and generated descriptions is carefully controlled.
Critical weaknesses
- The contribution beyond existing embodied-memory evaluations is limited. FindingDory already separates scripted experience collection from subsequent tasks and includes visual-history and text-summary agents . GraphEQA combines images and scene graphs as memory, while WorldLines evaluates memory QA and downstream planning . The submission cites these works. Its more distinctive contribution is the tighter control of writer inputs and retrieval interfaces, together with observation-free requests and separate inference-cost accounting. These are useful refinements, but I am not yet convinced that the resulting evaluation capability constitutes a sufficiently substantial advance.
- The empirical findings provide limited explanation beyond configuration rankings. Retaining visual details that task-agnostic descriptions may discard, and supplying explicit spatial records, have clear reasons to help. Quantifying these benefits is useful, but the category breakdowns do not sufficiently distinguish evidence loss, retrieval failure, reader interpretation, and planning errors . The reversal between frames and descriptions on GLM, and the inconsistent benefit of visual writer input, could yield more informative conclusions if their causes were analyzed. Similarly, lower write-time inference cost for frames follows in part from omitting description-generation calls while excluding storage costs. The reduction in read cost per successful request is a meaningful empirical result, but it does not establish the preferred modality under storage constraints or mapping costs. The stated cost scope is clear; the concern is how much new design guidance these results provide.
Questions for authors
- Does the frame–description gap mainly reflect information loss during captioning or differences in how the reader uses evidence?
- Do frames and scene graphs provide complementary information, or largely encode the same facts differently?
- How dependent are the findings on unconstrained storage and the prohibition on new observations?
- What explains the GLM ranking reversal, and when should descriptions be preferred over frames?
4 comments
W1. Our contribution beyond previous works.
We acknowledge that components of our work have semblance with other prior works on memory evaluation. The most distinctive aspect of our work is that MeMoREval isolates the effects of the modality of stored information. In FindingDory, the text and visual agents differ in input coverage and format, GraphEQA builds memory during active exploration while conditioned on the future question, and WorldLines only involves text-only backends, so none can attribute differences in success or cost to the stored representation. MeMoREval fixes the recorded experience, writer input, reader, and retrieval interface and varies only the stored modality (text, graph, descriptions at three budgets, frames, frames + scene graph) across multiple memory backends, testing each fact with both a question and an executed manipulation, and measuring write- and read-time cost separately. This makes two kinds of attribution possible:
- Error attribution by modality. Each failure is traced to the stage where it occurred (Fig. 5). Descriptions fail because evidence is lost when they are written, frames lead to reader failures in interpretation, and adding scene graphs reduces reading errors. These are results about the modalities, not about one system, and are made possible by our design.
- Cost attribution by modality. Write and read cost are measured separately for the same agent, so a modality’s cost is not mixed with the cost of a different pipeline. This mechanism allowed us to make the slightly surprising observation that passing more information (e.g. frames) to the reader actually reduces reasoning token usage and tool call frequency, hence is not more expensive than passing less information (e.g. short descriptions). The outcome is that MeMoREval provides not only a ranking, but also guidance on what to improve: captioning for descriptions, retrieval and reading for frames, and we can also evaluate any new model or any new memory backend through the same failure attribution.
W2 / Q3. Cost, closed-loop execution, and writer images.
Costs.
We acknowledge that a complete robotic system involves costs at many different stages. In MeMoREval, we report model inference cost because it is the cost most directly incurred by memory design. We have added an explanation on the other costs in §5.1, and we provide some details below.
- Storage. The storage footprint is dominated by one 640×480 frame per skill step. Each frame is under 1 MB even uncompressed, so a robot taking 1,000 steps a day sums to less than 1 GB a day, which is small for on-board or cloud storage, and a fraction of the storage footprint of videos. Keeping only writer-selected frames (Fig. 6a) reduces this further at a cost of 4–17% drop in success, and the 8–55% of frames that are kept still match or exceed full descriptions in each backend.
- Mapping and sensing. We consider robot-controlling embodied agents with navigation and manipulation capabilities. For a robot that navigates within an environment, it internally maintains a 3D object-level map (e.g. through SLAM techniques), and for a robot that can manipulate objects, it is necessary to have visual perception. Therefore, the memory system reuses the spatial map and visual perception from the robot’s existing capabilities.
No new observations.
We deliberately make the design of not allowing new observations during requests, and you have listed it as a strength of our design. If the agent could observe the environment again at request time, success would measure the agent’s ability to explore and recover, not the content of its memory.
Writer images.
Whether the memory writer benefits from seeing images and the extent of such benefits depend on the memory structure. We observed that across the four writer backends, the change in task success is minor and not uniform (range of −9% to +3%), and the write-time cost sees both increases and decreases for different backends (Table 8). Since this effect is dependent on memory backends, and this paper focuses on memory modality evaluation and not one particular memory system design, we only draw the conclusion that the benchmark provides the setting to study this effect.
Follow-up on your review
Dear Reviewer xQ5g,
Thank you once again for your detailed and insightful review. Your suggestion to separate failure stages was the centerpiece of our revision and helped turn our empirical rankings into actionable design guidance.
In the revised paper, we added a stage-by-stage failure attribution analysis across all models and conditions, explaining the GLM ranking reversal and showing that descriptions suffer from persistent evidence loss while scene graphs reduce the reader errors introduced by frames. We also added the new graph (text + scene graph) condition across all models and backends—demonstrating strong complementarity between frames and scene graphs—clarified how isolating the stored modality enables both failure and cost attribution beyond prior benchmarks, and expanded our discussion of storage, mapping, and writer-image costs.
We believe these updates directly address the concerns behind your initial evaluation and clearly establish the paper's contribution. As the discussion phase comes to an end soon, we would be very grateful for your thoughts on the revision and whether our responses have addressed your concerns. Please let us know if you have any remaining questions we can answer.
Best regards,
The Authors
Dear Reviewer xQ5g,
As the discussion period draws to a close, we wanted to thank you again for encouraging us to look beyond aggregate rankings. Your insightful feedback led to our development of a failure attribution pipeline for our benchmark, showing that descriptions fail mainly from evidence loss, raw frames trade evidence loss for higher reader errors (explaining the GLM reversal), and scene graphs help all models interpret frames and reduce reader errors. MeMoREval thus isolates the stored modality to diagnose whether failures stem from evidence retention or downstream multimodal reasoning.
We would greatly value your thoughts on these revisions and are happy to answer any remaining questions before the discussion closes today.
Best regards,
The Authors
Thank you for the detailed review. We appreciate your insight on classifying different failure modes, and performing such an analysis was the main focus of our revision. With additional failure attribution analyses, we find that our controlled evaluation protocol lends itself to useful conclusions that support design guidance for memory systems. In the following responses, intervals are 95% paired bootstrap intervals over tasks.
W2 / Q1 / Q4. New analysis: where each memory type fails, and resulting design guidance.
We assigned each failed run to the first stage it failed:
The result of the analysis is shown in Fig. 5 in the revised paper and discussed in §5.2.1. We randomly examined 5% of all rule-based classifications and found 100% agreement. The results reveal some recurring patterns across all three models. Descriptions cause evidence loss failures, and frames have minimal evidence loss failures but cause more reader errors. Scene graphs make frames easier to interpret and thus reduce downstream errors, which is why frames+graph have the highest success on every model. The table below summarizes Fig. 5 with error counts by category across three models and the three backends run on all models. Gemini values are means across three seeds, and GLM and GPT use seed 0.
Design guidance, description vs. frames.
Descriptions appear to be preferable when a model loses less by captioning an image than by reading it later in the context of the task (Fig. 5). For GLM, frames save 23 evidence-loss failures but add 29 downstream failures, so descriptions achieve better performance overall. For Gemini and GPT the saving is larger than the added reader errors, so frames are better than descriptions. However, descriptions have fundamental limitations that cannot be addressed by a stronger model, and we provide further explanations for this next. The failure attribution also reveals that increasing description length does not reduce evidence-loss failures, a reflection of the inherent difficulty with storing details when the future task is unknown. With frames being stored, the remaining failures are mostly due to missed retrieval, reader errors, or planner errors. Any remaining evidence-loss failures is due to the task-necessary fact being present in the textual observation but not recorded by the model at writing time, and GLM has a higher remaining evidence-loss failures than the other models. As model capabilities grow, remaining retrieval, reader, and planner errors will likely see further reduction, and therefore the forward-looking guidance would be to store frames and scene graphs when cost is not a concern, as graphs help with the interpretation of frames, and the combination of these modalities is likely the first to see near-zero overall failure with improved model capabilities. We have updated §5.2.1 and §5.2.2 with these design guidelines.
Q2. New graph experiments and failure analysis show that frames and graphs are complementary.
We demonstrate that frames and graphs encode complementary information through two fronts: