- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ReSight is a memory system for embodied agents that retains visual observations and their spatiotemporal relationships while supporting new representations and access tools after data collection. It pairs an execution agent constrained to registered tools with a coding agent given a programmable workspace and a larger exploration budget. Its Explore–Distill–Mount (EDM) procedure uses execution traces and exploration-generated programs to identify capability gaps and build reusable tools, memory builders, and skills without task reference answers. Evaluations cover memory retrieval on GOAT-core, capability expansion on OC-NaVQA and FindingDory, and reuse on real-robot patrol memories. The strongest evidence comes from FindingDory: with capabilities frozen, success rises from 37.5% to 65.8% on episodes held out from both exploration and distillation.
Critical strengths
-
Substantial gains persist on held-out episodes with frozen capabilities.
Table 4 provides strong evidence of reuse: tools and a skill developed on FindingDory episodes 91–100 are applied without further updates to 597 queries from episodes 81–90. Success increases from 37.5% to 65.8%, while online token usage falls from 128.8k to 39.4k per query. This suggests that EDM produces reusable operations that extend beyond the exploration queries. -
The separation between retained evidence and extensible processing is useful and concretely demonstrated.
Preserved observations support additional capabilities, including OCR, geometric localization, and interaction-history analysis. The execution traces illustrate why this matters: a localization tool extracts object positions from observations already retrieved by the executor, while interaction tools identify post-placement evidence rather than a nearby but inappropriate frame. These examples clearly distinguish having relevant evidence from having the tools needed to interpret it. -
Additional experiments support applicability across host systems and annotation models.
Improvements on RAVEN as a second host suggest that the approach is not limited to the primary memory system. The annotation-model replacement experiments further examine its dependence on a particular model choice. Together, these results broaden the empirical support for EDM beyond the main configuration.
Critical weaknesses
-
The benefit of EDM’s staged procedure is not clearly separated from that of additional tools and evidence access.
The no-Explore ablation restricts access to development evidence as well as removing exploration. As described in Appendix C.3, it cannot directly inspect memory images or annotation files and receives no exploration outputs, although its code can process self-test recordings. The 46.4% versus 65.8% gap therefore supports the full setup but does not establish whether EDM outperforms direct tool synthesis under equivalent conditions. A baseline with matched evidence access and construction budgets would help resolve this question. Likewise, the OC-NaVQA comparison demonstrates the usefulness of geometric processing without isolating the advantage of EDM’s construction procedure. -
Capability construction is not evaluated across independent runs.
The generated library is a central output, yet the experiments do not show how consistently independent EDM runs produce effective capabilities. Repeating executor evaluations, including the five executions per query in the real-robot experiment, does not measure this construction variability. Although Algorithm 1 and Appendix B.4 outline the workflow, experiment-specific budgets, replay settings, artifact-selection criteria, and any manual modifications remain insufficiently documented. These details and repeated construction runs would strengthen the reproducibility evidence. -
The efficiency analysis covers only part of the computational cost.
Lower online token usage is a useful result, but exploration, distillation, builder execution on new memories, and computation within mounted tools are not fully quantified. Consequently, the reported savings do not establish lower overall cost or latency. Separately reporting construction, deployment, and query-time costs would clarify the practical benefit and the amount of reuse needed to recover the initial investment.
Questions for authors
-
Could you compare EDM against direct tool synthesis with matched development data, execution traces, evidence access, and construction budgets? Does the advantage persist when the baseline can directly inspect memory images and annotations?
-
How many independent capability-construction runs were conducted per configuration? Please explain how the reported library was selected, disclose any manual revisions to generated artifacts, and report performance variability across independently constructed libraries, if available.
-
What exploration and distillation budgets, replay counts, and stopping criteria were used for each benchmark? Please distinguish construction costs, builder runtime on new memories, and online execution costs, and clarify whether reported token and request counts include model calls within tools.
-
Could you report per-episode performance for the held-out FindingDory evaluation? Since the 597 queries span ten episodes, this would show whether gains are consistent across memories. Where feasible, please include uncertainty estimates that account for queries sharing an episode.
Thank you for the additional experiments and detailed responses. I appreciate that the authors have addressed my concerns by expanding the multi-round capability and analyzing the computational cost. The new results provide encouraging evidence of cumulative capability reuse. However, the sequential expansion is still evaluated on a limited set of predefined tasks, and it remains unclear how well this process scales to more diverse and unexpected task requirements. The cost analysis is also based on a relatively small evaluation setting. While I acknowledge the improvements, I believe further evidence is needed to establish the generality and scalability of the proposed self-improvement mechanism. Therefore, I will maintain my original rating.