- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper presents SOMo, a scene memory for retrieving objects from a robot’s past observations using semantic, spatial and temporal information. The agent makes a unified "search_objects" call with object descriptions and a relation, and the backend does the filtering with a hierarchical spatial memory, observation associations and an adaptive planner. Memory construction itself is from DAAAM. Experiments on several QA and grounding benchmarks show lower error and fewer tool calls than previous methods.
Critical strengths
The problem of retrieving contextual information from long-term robot memory is relevant, and handling relational constraints inside retrieval rather than across LLM tool calls is a reasonable idea. The reduction in tool calls seems plausible.
Critical weaknesses
-
Unclear problem formulation and contribution. The paper does not clearly formulate the problem it addresses, and the contribution is not clearly specified at the end of the introduction. There are many components stacked on top of DAAAM, but it is not clear which of them is essential or new, or why the whole system needs to be this complicated. A lot of it looks like standard spatial indexing, and several design choices are stated without much motivation. SOMo also changes the retrieval interface and backend at the same time, without an ablation separating their effects, so it is difficult to tell what is actually responsible for the reported gains. Overall, I found the method and the paper hard to follow.
-
Quality control of the manuscript. The eMEM reference lists affiliations as author names (and invents a first name for the first author), and several other references contain conflicting publication years. The authors disclose AI assistance for implementation and writing, which is fine by itself, but it does make me wonder whether the generated content was checked carefully.
Questions for authors
- Could the authors state the problem formulation precisely and say what they consider the main contribution?
- What does DAAAM plus a simple joint-search tool achieve, without the hierarchy and planner? More generally, which components are actually responsible for the gains?
- How does SOMo perform on the generated benchmarks when the agent is not explicitly told which retrieval relation or tool to use?
2 comments
Q3. The tools’ behavior and arguments are described in their docstrings and tool descriptions, which are presented to the agent. The agent constructs tool calls using this documentation and the question. Table 4 distinguishes this setting from using tool documentation alone. We found that removing explicit tool guidance slightly lowers SOMo's performance. However, the best call argument to use often remains quite identifiable from the question so the impact remains limited.
Table 4. CODaAG results across four query categories for SOMo with Qwen3.5-9B under different agent-guidance settings. Pos. is positional error (m); Temp. is temporal error (min); lower is better.
| Agent guidance | Single-Object Pos. ↓ | Single-Object Temp. ↓ | Multi-Object Co-Observation Pos. ↓ | Multi-Object Co-Observation Temp. ↓ | Multi-Object in Proximity Pos. ↓ | Multi-Object in Proximity Temp. ↓ | Robot-Relative Angle Pos. ↓ | Robot-Relative Angle Temp. ↓ |
|---|---|---|---|---|---|---|---|---|
| Tool descriptions + tool guidance | 17.69 | 0.952 | 22.82 | 1.162 | 24.42 | 1.124 | 25.26 | 1.213 |
| Tool docstrings/descriptions only | 17.71 | 0.952 | 23.25 | 1.168 | 24.96 | 1.141 | 25.31 | 1.219 |
W1 / Q1. We thank the reviewer for pointing out that the problem formulation and contribution need to be stated more clearly. Given a memory of mapped objects, semantic features, observation histories, and robot trajectory samples, the problem is to retrieve an ordered assignment of distinct objects to the descriptions in a request, subject to spatial and observation constraints. For example, “the bike near a bench that you saw together between time A and B” requires a bike–bench assignment satisfying both proximity and common observation support within the requested period. Among feasible assignments, SOMo ranks the combinations by their semantic match to all requested roles and returns the selected objects together with supporting observation evidence.
The main contribution is contextual retrieval that jointly selects objects and their relevant encounters under a shared set of conditions. Separate object searches leave the agent to assemble and verify combinations after retrieval, and an early choice of a reference object can exclude the correct target. SOMo moves these checks into retrieval, allowing the relationship and observation context to influence which objects are selected. Its joint ranking prioritizes the weakest semantic role match, while returned observation evidence supports subsequent answer generation. The sparse hierarchy, object–visit associations, and adaptive planner provide an efficient implementation of these retrieval operations. We will revise the introduction and method overview to distinguish the retrieval formulation and its effect on answer quality from the execution mechanisms and their effect on efficiency.
W2. We thank the reviewer for identifying this issue. We collected all BibTeX entries manually, and the incorrect eMEM entry may have originated from an upstream metadata error in Semantic Scholar at the time of collection. We will correct the affected references, including the author information and publication-year inconsistencies, in the manuscript.
Q2. DAAAM with a simple pair-search tool improves over DAAAM, but remains below SOMo. The following NaVQA results use Qwen3.5-9B throughout.
Table 1. DAAAM with an explicit object-pair search tool compared with SOMo.
DAAAM equipped with our full unified search tool—including joint constraint verification, role assignment and ranking, and returned observation evidence—effectively becomes SOMo at the retrieval level, since SOMo already uses DAAAM's memory-construction pipeline. The hierarchy and planner accelerate execution of this retrieval contract. Replacing them with exhaustive evaluation over the same records and constraints, while preserving ranking and returned evidence, preserves the retrieval results and therefore the information available to the answering agent. We do not attribute task-accuracy gains to the hierarchy or planner.
To isolate the value of retrieving multiple objects, we also restrict SOMo to single-object queries while holding everything else fixed. Performance decreases substantially.
Table 2. NaVQA ablation restricting SOMo to single-object queries, with Qwen3.5-9B throughout.
We further separate ranking and evidence materialization using 105 joint-object CODaAG questions. All arms share an exhaustive backend, fixed requests, and identical prompts; answer generation proceeds without agent-driven query selection or follow-up searches.
Table 3. Fixed-query ablation of joint ranking and returned evidence.
Leximin combined with materialized observation witnesses gives the lowest errors in this study. Leximin alone does not improve over mean ranking with raw evidence, so these results support the combination of joint ranking and usable observation evidence rather than an independent benefit from every component. Together, these ablations distinguish the quality benefits of joint retrieval and evidence presentation from the computational benefits of indexing and planning.