acceptodds
Under review as a conference paper at ICLR 2027

ReSight: Task-Driven Self-Improving Memory for Embodied Agents

Abstract

Robots can use historical observations to answer spatiotemporal queries in familiar environments, but the presence of relevant evidence does not imply that existing tools support the operations a task requires. We introduce ReSight, a task-driven self-improving memory system for embodied agents that expands the capabilities for processing and accessing historical observations as task demands arise. ReSight retains keyframes and their spatiotemporal associations, separating observation evidence from extensible derived fields so that stored observations can be reprocessed for new needs. An execution agent and a coding agent share this memory through different interfaces: the former calls registered tools within a bounded online budget, while the latter uses a programmable workspace and a larger exploration budget to inspect evidence, write programs, and acquire models as needed. To realize capability expansion in ReSight, we develop Explore–Distill–Mount (EDM), which identifies capability gaps from execution and exploration records without task reference answers and converts exploration-derived solutions into reusable memory builders, query tools, and skills, extending memory representations and access capabilities as needed. Experiments on three public benchmarks demonstrate effective base memory access and capability expansion. With the execution model and online budget held fixed within each benchmark, EDM reduces mean position error on OC-NaVQA from 33.10 m to 24.72 m and improves high-level success on FindingDory from 38.3% to 63.8%. With the tools and skill frozen, success still improves by 28.3 percentage points on FindingDory episodes excluded from exploration and distillation. Experiments on real robot patrol memories further demonstrate reuse of these capabilities for subsequent queries and new memories, with fewer online tool calls on average.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied agents, Agent self-improvement, Spatiotemporal memory

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_HNC1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReSight is a memory system for embodied agents that retains visual observations and their spatiotemporal relationships while supporting new representations and access tools after data collection. It pairs an execution agent constrained to registered tools with a coding agent given a programmable workspace and a larger exploration budget. Its Explore–Distill–Mount (EDM) procedure uses execution traces and exploration-generated programs to identify capability gaps and build reusable tools, memory builders, and skills without task reference answers. Evaluations cover memory retrieval on GOAT-core, capability expansion on OC-NaVQA and FindingDory, and reuse on real-robot patrol memories. The strongest evidence comes from FindingDory: with capabilities frozen, success rises from 37.5% to 65.8% on episodes held out from both exploration and distillation.

Critical strengths
  1. Substantial gains persist on held-out episodes with frozen capabilities.
    Table 4 provides strong evidence of reuse: tools and a skill developed on FindingDory episodes 91–100 are applied without further updates to 597 queries from episodes 81–90. Success increases from 37.5% to 65.8%, while online token usage falls from 128.8k to 39.4k per query. This suggests that EDM produces reusable operations that extend beyond the exploration queries.

  2. The separation between retained evidence and extensible processing is useful and concretely demonstrated.
    Preserved observations support additional capabilities, including OCR, geometric localization, and interaction-history analysis. The execution traces illustrate why this matters: a localization tool extracts object positions from observations already retrieved by the executor, while interaction tools identify post-placement evidence rather than a nearby but inappropriate frame. These examples clearly distinguish having relevant evidence from having the tools needed to interpret it.

  3. Additional experiments support applicability across host systems and annotation models.
    Improvements on RAVEN as a second host suggest that the approach is not limited to the primary memory system. The annotation-model replacement experiments further examine its dependence on a particular model choice. Together, these results broaden the empirical support for EDM beyond the main configuration.

Critical weaknesses
  1. The benefit of EDM’s staged procedure is not clearly separated from that of additional tools and evidence access.
    The no-Explore ablation restricts access to development evidence as well as removing exploration. As described in Appendix C.3, it cannot directly inspect memory images or annotation files and receives no exploration outputs, although its code can process self-test recordings. The 46.4% versus 65.8% gap therefore supports the full setup but does not establish whether EDM outperforms direct tool synthesis under equivalent conditions. A baseline with matched evidence access and construction budgets would help resolve this question. Likewise, the OC-NaVQA comparison demonstrates the usefulness of geometric processing without isolating the advantage of EDM’s construction procedure.

  2. Capability construction is not evaluated across independent runs.
    The generated library is a central output, yet the experiments do not show how consistently independent EDM runs produce effective capabilities. Repeating executor evaluations, including the five executions per query in the real-robot experiment, does not measure this construction variability. Although Algorithm 1 and Appendix B.4 outline the workflow, experiment-specific budgets, replay settings, artifact-selection criteria, and any manual modifications remain insufficiently documented. These details and repeated construction runs would strengthen the reproducibility evidence.

  3. The efficiency analysis covers only part of the computational cost.
    Lower online token usage is a useful result, but exploration, distillation, builder execution on new memories, and computation within mounted tools are not fully quantified. Consequently, the reported savings do not establish lower overall cost or latency. Separately reporting construction, deployment, and query-time costs would clarify the practical benefit and the amount of reuse needed to recover the initial investment.

Questions for authors
  1. Could you compare EDM against direct tool synthesis with matched development data, execution traces, evidence access, and construction budgets? Does the advantage persist when the baseline can directly inspect memory images and annotations?

  2. How many independent capability-construction runs were conducted per configuration? Please explain how the reported library was selected, disclose any manual revisions to generated artifacts, and report performance variability across independently constructed libraries, if available.

  3. What exploration and distillation budgets, replay counts, and stopping criteria were used for each benchmark? Please distinguish construction costs, builder runtime on new memories, and online execution costs, and clarify whether reported token and request counts include model calls within tools.

  4. Could you report per-episode performance for the held-out FindingDory evaluation? Since the 597 queries span ten episodes, this would show whether gains are consistent across memories. Where feasible, please include uncertainty estimates that account for queries sharing an episode.

Reviewer_cz4KOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on a task-driven, self-improving memory system for embodied agents. Historical observations already contain the information needed to solve the current task, but the agent may be unable to effectively utilize this information due to a lack of appropriate memory processing and retrieval capabilities. The author argues that embodied memory not only needs to store and retrieve historical information but should also be capable of expanding its processing capabilities regarding historical observations based on the requirements of new tasks. The overall system design in the paper is sound, and the experimental results are reasonably persuasive. However, I still have some questions regarding the independent contribution of the automated capability discovery mechanism, the validation of long-term self-improvement capabilities, and the analysis of the full computational overhead.

Critical strengths

The research question has practical usage and proposes a design approach for scalable embodied memory. I think the main advantages are as follows:

  1. This paper separates the original observational evidence from the subsequently generated memory fields, enabling the system to reprocess existing observations based on new task requirements without the need to re-collect environmental data. I believe this design has practical value for embodied agents operating over the long term. Because when robots construct memories, it is difficult for them to determine in advance all the information and processing methods required for future tasks.

  2. This paper adopts a design that separates the execution agent from the coding agent, allowing the two agents to handle real-time task execution and complex capability development. This allows complex offline exploration to be transformed into reusable online capabilities, resulting in a well-designed system.

Critical weaknesses

The main issue with this article is that it has not yet sufficiently demonstrated its ability to continuously improve itself. Current experiments generally employ a single EDM process, followed by an evaluation of subsequent tasks using the frozen toolset. As the system continues to receive new tasks with varying requirements, there is still a lack of direct experimental validation regarding whether it can repeatedly identify new capability gaps, develop new tools, and achieve cumulative improvements based on the existing tool library. As I recall, as the toolkit continues to grow, issues such as redundant tool functionality, conflicts between different tools, and increased complexity in tool selection may arise. Therefore, the main argument of this paper is that while existing experiments support ReSight’s ability to scale and reuse capabilities in a task-driven manner, the evidence remains insufficient to support its core claim of long-term, continuous self-improvement.

In addition, the authors did not fully quantify the offline computational resources and time required for the entire EDM process. Existing experiments have effectively demonstrated that EDM can reduce online execution overhead, but they do not yet fully prove that it offers higher computational efficiency over the entire system lifecycle.

Questions for authors

These issues correspond to the shortcomings mentioned in the weaknesses section.

  1. Has the author tested the capability of EDM to expand its capabilities over multiple consecutive iterations? For example, after integrating existing tools, introducing new task requirements, and then identifying capability gaps and developing new tools? I believe this experiment is crucial for validating ReSight’s ability to continuously improve itself.

  2. I hope the author can quantify the offline computational overhead of Explore, Distill, and Memory Builder, and analyze how many subsequent tasks are needed to amortize these costs. This would provide a more comprehensive illustration of EDM’s actual computational efficiency.

3 comments

Thank you for the additional experiments and detailed responses. I appreciate that the authors have addressed my concerns by expanding the multi-round capability and analyzing the computational cost. The new results provide encouraging evidence of cumulative capability reuse. However, the sequential expansion is still evaluated on a limited set of predefined tasks, and it remains unclear how well this process scales to more diverse and unexpected task requirements. The cost analysis is also based on a relatively small evaluation setting. While I acknowledge the improvements, I believe further evidence is needed to establish the generality and scalability of the proposed self-improvement mechanism. Therefore, I will maintain my original rating.

Dear Reviewer cz4K,

Thank you for acknowledging that the multi-round experiment and cost analysis addressed your original questions. We would like to clarify the scale of our evaluation and the scope of our claim.

Evaluation scale and cost analysis. In addition to the 30 real-robot trials, we evaluated 1,403 queries across FindingDory and OC-NaVQA. On the 597 held-out FindingDory queries, frozen capabilities raised overall success from 37.5% to 65.8%, with gains in 10 of 11 task categories, while reducing mean online token usage by 69.4% (128.8k → 39.4k tokens per query, Tables 4 and 10). At the token rate used in Appendix D, we estimate that these savings would offset a development API budget of about \$290 after roughly 810 subsequent queries with the same task mix.

Scope of our claim. ReSight’s contribution is task-driven memory improvement. As new tasks reveal gaps, ReSight extends its persistent memory representations, tools, and skills without task reference answers (§§1 and 3.4). Since the original submission, §5 has explained that predefined tasks enable controlled comparisons. Evaluating continued improvement over longer deployments with a broader range of emerging tasks remains future work.

Given this scope and the evidence that the acquired capabilities improve both task success and online efficiency on held-out queries, would you consider revisiting your rating during the final discussion?

One real-robot result speaks to your question about unexpected task requirements. During development, ReSight built a trajectory tool to find the point farthest from a given location. On a new floor, the frozen tool was reused to answer a different question: how far had the robot traveled during its patrol? Success rose from 0/5 to 5/5 (Table 5).

This example shows reuse beyond the task that prompted the tool’s development. We agree that sustained adaptation to a longer sequence of unforeseen tasks remains open. A benchmark that reveals new tasks over time and tests both newly acquired and previously learned capabilities would be a useful next step.

Reviewer_AQKcOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReSight, a task-driven memory system for embodied agents. It investigates how to expand an agent’s processing capabilities when task-required evidence already exists in historical observations, yet existing tools cannot effectively leverage such evidence within a limited online budget. The system retains revisitable keyframes and their spatiotemporal associations, decouples observational evidence from extensible derived fields, and provides distinct access interfaces and compute budgets for the execution agent and the coding agent.

The authors put forward the Explore–Distill–Mount (EDM) pipeline, which identifies capability gaps from execution traces and the coding agent’s exploration results, and converts solutions into reusable memory builders, query tools, and usage guidelines. System improvements are delivered primarily at the level of tools, memory representations, and access mechanisms, while the execution model and online budget stay fixed before and after mounting. Experiments cover GOAT-core, OC-NaVQA, FindingDory, and real-robot patrol memories, evaluating base retrieval, capability expansion, and reuse under frozen capabilities respectively.

Critical strengths

Strength 1: Clear problem framing; the system design addresses practical gaps in memory processing.

The paper draws a sharp distinction between relevant information being present in historical observations and the current execution system being able to make effective use of it. This distinction carries real practical weight: once relevant images are retrieved, a task may still demand extra operations such as text reading, object coordinate estimation, interaction event structuring, or duration calculation.

ReSight combines reprocessable observational evidence, extensible memory fields, and differentiated access paths for execution and development, so that new task demands can drive the system to add new processing routines and query interfaces. Instead of forcing the online execution model to repeat all processing from scratch for every query, EDM packages solutions from one exploration session into reusable tools. This is a cleanly reasoned system contribution with tangible engineering value.

Strength 2: Held-out and real-robot evaluations support frozen-capability reuse, while cross-host experiments demonstrate the applicability of EDM to a second execution system.

The most persuasive results come from the held-out test on FindingDory. The authors freeze the tools and usage rules developed on episodes 91–100 and evaluate them on episodes 81–90, which took no part in exploration or distillation. Overall success improves from 37.5% to 65.8%, and average online input+output tokens fall from 128.8k to 39.4k. This indicates the value of the acquired capabilities is not confined to the specific memories and queries seen during development.

In addition, EDM yields clear gains on RAVEN, a distinct host system, and the real-robot experiments further show that frozen programs can be reused for new queries and memories from an unseen floor. Together, these results support EDM as a capability-expansion mechanism applicable across different execution systems.

Critical weaknesses

Weakness 1: The unique contribution of EDM’s specific organizational pipeline is not sufficiently established relative to direct tool synthesis and existing memory evolution methods.

The paper’s core contribution lies in how it organizes memory representations, tool construction, and capability development workflows. Section 2.3 already reviews automatic tool generation, skill accumulation, agent harness evolution, and automated improvement of memory building and retrieval procedures. For that reason, simply showing that a coding agent produces programs which the execution agent later invokes is not enough to clearly define the paper’s incremental advance over the closest prior work.

The main experiments compare system performance before and after capability mounting. They confirm that adding tools and structured memory helps, but they do not adequately disentangle whether the gains come from EDM’s specific phased structure, capability ledger, and gap diagnosis, or chiefly from the extra development budget and from filling in operations the execution side previously lacked. For instance, on OC-NaVQA the base executor returns an observation’s camera pose, while the added tool estimates the object’s own position. This comparison clearly demonstrates the value of adding geometric computation, but it cannot by itself prove that EDM is more effective than a more direct tool-construction approach.

The “EDM w/o Explore” ablation also does not fully resolve this question. That setting not only removes exploration programs, notes, and answers, but also bars the coding agent from directly viewing memory images or reading memory annotation files. Because the comparison alters both exploration material and evidence access at the same time, it cannot cleanly isolate the effect of EDM’s specific organizational steps.

My central concern is: given the same raw observations, coding model, and capability-building budget, why is this particular EDM organizational pipeline necessary? Could a more straightforward automatic tool-construction pipeline achieve comparable results?

This is not an objection to the system’s use of closed-source models or existing components. It is about the absence of a controlled comparison for the paper’s main methodological claim. The held-out evaluation with frozen capabilities shows that the resulting tools are reusable, but it does not show that EDM is better than alternative development workflows at producing those reusable capabilities.

Weakness 2: Gains are strongly task-dependent, and current reporting is insufficient to fully judge trade-offs among accuracy, compute cost, and scalability.

First, EDM does not produce consistent improvements across all question types and metrics. On OC-NaVQA, mean position error drops from 33.10 m to 24.72 m, but binary accuracy falls from 86.6% to 82.1%, and text accuracy remains at 63.6%. The appendix reports a McNemar test for the binary change, so a statistically confirmed systematic degradation cannot be claimed. Even so, these results make clear that the benefits of capability expansion are unevenly distributed.

On the held-out FindingDory episodes, interaction ordering, ordered revisitation, and duration tracking rise to 82.5%, 96.7%, and 76.7% respectively, but spatial relation tasks improve only from 4.0% to 12.0%, and object recall stays at 45.0%. This pattern suggests that current gains are concentrated on capabilities addressable through well-defined event structuring, sequence sorting, and temporal statistics routines; notable gaps remain for spatial relation reasoning and certain visual memory tasks. This is not to say EDM cannot in principle handle those tasks, but the paper does not adequately explain why those gaps were not effectively identified and closed by the current pipeline.

Second, the advantage of ReSight’s base architecture relative to other hosts needs sharper positioning. In the main FindingDory experiments, RAVEN + EDM reaches 66.6% overall success, above ReSight + EDM at 63.8%; meanwhile ReSight uses fewer online tokens, 43k versus 150k for RAVEN. The results therefore support an accuracy-vs-online-cost trade-off rather than an across-the-board superiority of ReSight in task success. The two hosts also differ in retrieval returns and image context handling, so neither success rate nor token count alone can settle which architecture is better overall.

Finally, offline memory construction and capability development costs are underreported. The system must generate scene and object descriptions for retained keyframes, extract feature embeddings, and during EDM develop programs and build new fields. Appendix D.1 states that these steps run offline and can serve multiple queries, but does not fully quantify the actual costs of index building, exploration, debugging, and index reconstruction. Lower online token usage is therefore not enough to judge total deployment cost, or to estimate how many downstream queries are needed to amortize the preprocessing investment.

Keyframe selection also directly affects both cost and evidence coverage. The appendix specifies the SigLIP2 similarity filtering rule for FindingDory, but for GOAT-core and OC-NaVQA mostly only final keyframe counts are given; the selection procedure, raw observation volume, and retention ratio remain unclear. Such information matters for understanding the system’s practical scope and its scalability in long-term operation.

Questions for authors

Question 1 What additional value does EDM deliver compared to a more straightforward automatic tool-construction pipeline? This is the central issue shaping my judgment of the method’s novelty.

I suggest adding a controlled comparison on one major benchmark, matched for information access and capability-building budget: use the same coding model, grant it access to the same raw observations, base memory and execution traces, and let it directly generate, test and mount reusable memory builders, query tools and usage guidelines — but without enforcing EDM’s phased structure, capability ledger or gap-summarization workflow. The resulting capabilities should then be frozen and evaluated on the same held-out episodes under the same execution model and online budget.

A more targeted alternative would be to share identical Explore outputs and compare “directly packaging exploration programs” against the full Distill pipeline, to isolate the independent contribution of the capability ledger and gap diagnosis.

This is an experimental request targeting the core contribution, not a demand to rerun all experiments across every benchmark. In addition, please clarify the number of generation attempts, debugging iterations, replay rounds, candidate-tool selection criteria and any human intervention involved in actual capability development, so that the building budgets can be assessed as comparable.

Question 2 Why are performance gains so uneven across task types? And how should the advantage of ReSight relative to RAVEN be positioned?

Based on existing execution traces, please analyze at which stage difficulties on spatial-relation and object-recall tasks primarily arise: whether key evidence was not retained, base retrieval failed to locate the evidence, capability gaps were insufficiently identified, the generated tools cannot perform the required operations, or the executor fails to use the tools correctly.

In particular, please explain why EDM substantially improves temporal, ordering and statistical tasks, yet does not adequately address other exposed capability gaps. I do not expect every metric to improve; what I would like is for the authors to clarify the conditions under which the current pipeline works well and under which it does not.

Furthermore, RAVEN + EDM achieves higher overall success than ReSight + EDM, while ReSight consumes fewer online tokens. Please clearly separate the claim that “EDM is effective across different hosts” from the claim that “ReSight’s base memory architecture has an inherent advantage”, and explain to what extent current experiments support the latter. If existing run logs or limited budget comparisons can illuminate the accuracy–cost trade-off more clearly, that would be helpful.

Question 3 What is the full cost of base memory construction and capability development? Is per-keyframe semantic description generation necessary?

Please report separately the costs of keyframe annotation, visual and text encoding, index building, as well as EDM exploration, program development and new field generation. Priority should be given to total processing time, model calls or token usage, and compute resources from existing logs, with a breakdown between one-time index-construction cost, capability-development cost and subsequent incremental update cost.

The paper already includes MoM’s visual-only retrieval configuration as a baseline on GOAT-core, but there is still no direct analysis of the necessity and cost–benefit of ReSight’s own various semantic fields. With keyframes, localization backend and remaining retrieval settings held fixed, could you compare configurations using only visual features, plus scene descriptions, and plus object descriptions? If results from low-density annotation or on-demand annotation are available, please include those as well.

This question aims to judge whether richer initial memory is worth its extra preprocessing cost, not to argue for eliminating offline processing entirely.

Question 4 How are keyframes selected for each benchmark, and does the selection process affect evidence completeness and comparison fairness?

Appendix C.3 states that FindingDory retains a candidate frame when its SigLIP2 feature cosine similarity to the previous retained frame is below 0.90. Please supplement the keyframe selection methods for GOAT-core, OC-NaVQA and the real-robot patrol memories, and state whether preprocessing results from the datasets or baselines were reused directly.

Please also clarify the raw observation frame count, the final retention ratio, the computation required for filtering, and whether compared systems use the same observation set. This information is necessary for reproducing and understanding the existing experiments.

For the impact of keyframe density, existing sensitivity results or failure-case analyses would suffice: does sparser sampling miss briefly appearing signs, objects, or pick/place events? Does such omission limit the effectiveness of subsequent EDM reprocessing of historical evidence? This is intended as a suggestive analysis, and does not require a large-scale parameter sweep.

Reviewer_QHRKOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes ReSight, a memory system for embodied agents. ReSight retains keyframes and their spatiotemporal associations, and this memory is shared by two agents, an execution agent and a coding agent, through different interfaces. The authors further introduce a procedure called EDM, which turns the solutions obtained through exploration into reusable tools, skills, and memory builders. The experiments suggest that the approach is likely to work in practice.

Critical strengths

[S1] I believe the system could improve performance when deployed in practice. The design components of the memory system are reasonable, and the policy is well supplied with the information it lacks. Also, the empirical results demonstrate its effectiveness.

Critical weaknesses

[W1] My main concern is that the paper offers limited value as an academic paper. It reads more like a product manual, and both its presentation and validation need improvement. A few examples follow.

  • The definitions of “capability gap” and “capability gap task” are vague. The main text describes capability gap as a phenomenon rather than defining it, and the capability gap task is explained as “A task exposes a capability gap when the current tools, skills, or memory access mechanisms cannot solve it reliably within the online budget.”, and it is unclear whether the capability refers to the policy or the harness. Capability gap tasks are set as the design objective in Eq. 2, yet the experiments neither optimize nor measure it.
  • The novelty over skill/tool libraries and harness evolution is limited. Capability expansion largely resembles components proposed in prior work, such as skill-library accumulation [1,2], tool making [3], and harness or memory evolution [4], yet the paper does not empirically compare against any of them.
  • The main text is not self-contained, and each design choice lacks justification and validation. Many new terms are introduced (“late binding,” “database view,” “memory builder,” etc.), each with only a brief definition, so it is hard to understand how they work in practice or what form they take. This makes Section 3 hard to follow. The accuracy of each component is also not validated, e.g., whether a memory builder achieves sufficient coverage when generating fields.

[1] Voyager: An Open-Ended Embodied Agent with Large Language Models, Wang et al., https://arxiv.org/abs/2305.16291

[2] ASPIRE: Agentic /Skills Discovery for Robotics, Lu et al., https://arxiv.org/abs/2607.00272

[3] Large Language Models as Tool Makers, Cai et al., https://arxiv.org/abs/2305.17126

[4] Meta-Harness: End-to-End Optimization of Model Harnesses, Lee et al., https://arxiv.org/abs/2603.28052

[W2] It is not shown that the failures come from the harness rather than the policy, and the gains may instead come from distilling a stronger model’s solutions.

  • The explorer uses a stronger model (Opus-5), so it is unclear whether EDM’s gains come from properly handling the harness or simply from distilling the stronger model’s solutions.
  • This work proceeds on the assumption that the executor already has the ability and is limited by its harness, but this assumption is not tested.
Questions for authors
  • [Q1] Retained observations reflect the agent’s belief state about the environment, which may differ from the current world state. How does the system handle this situation?

  • [Q2] Because an embodied agent operates continuously, its memory grows over time. How does the system manage this growth, and how is performance affected as the memory scales?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.