- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work focuses on the spatially grounded conversational reasoning task for embodied assistants. It proposes to jointly preserve observed scenes and past dialogue to address complex requests. To facilitate research on this task, the authors construct the EGO-SPACR dataset based on ScanNet. Furthermore, they propose a framework named SPAC-MEM to tackle the SPACR task.
Critical strengths
The authors introduce the SPACR task and construct a corresponding dataset. In my view, this task represents a meaningful step forward for long-horizon embodied tasks. However, the insufficient analysis of dataset examples weakens the contributions of both the proposed task and the dataset.
Critical weaknesses
- The proposed method appears to be a combination of existing techniques, integrating 3D reconstruction, instance segmentation, object association, and large language models. Although it adopts an object-centric memory structure, its technological novelty remains limited.
- The presentation and validation of dataset quality are insufficient. The paper lacks complete examples connecting actual visual observations, continuous dialogue, object references, spatial evidence, and answers. This makes it difficult to assess the naturalness of the synthetic conversations, the answerability of the questions, and the practical value of the proposed task.
- The implementation details and robustness evaluation of the object consolidation mechanism are insufficient. The authors repeatedly use vague descriptions such as “overlap substantially” and “similar appearance” without specifying the relevant thresholds or how appearance similarity is computed, limiting reproducibility. The paper also lacks a sensitivity analysis of the merging thresholds and examples of incorrect merges or splits. Furthermore, its reliance on spatial alignment raises concerns about handling object movement in real-world applications, which could lead to duplicate records or outdated spatial information.
- The proposed method is evaluated only on the authors’ own dataset, without experiments on other well-established benchmarks such as OpenEQA [1].
- The paper lacks a quantitative analysis of overall computational efficiency. [1] Majumdar A, Ajay A, Zhang X, et al. Openeqa: Embodied question answering in the era of foundation models[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 16488-16498.
Questions for authors
- Please provide experiments on other relevant datasets.
- Please clarify object consolidation mechanism and give ablation studies towards its robustness.
- Please give more illustration of the proposed dataset.
3 comments
W3 / Q2. Examples of incorrect merges or splits.
Since we cannot add images in the discussion, we added examples of incorrect merges and splits to App. M, Fig. 13. These errors hold adjacent objects that look similar in one record or one object in several, which is a known limitation of most segmentation models. However, most objects still end up in memory near their true location (91% within 0.5 m).
W3 / Q2. Object movement: duplicate records and outdated spatial information.
Like existing 3D scene memory datasets such as OpenEQA [6] and SIMMC 2.1 [7], Ego-SpaCR assumes static rooms so that the answers are exact and executable.
SpaC-MEM already handles semantic change: it stores the dialogue on each object in temporal order, so when the user changes their mind about an object, the new statement is kept after the earlier ones rather than overwriting them. Objects that physically move are an interesting next step for this work.
W4 / Q1. The proposed method is evaluated only on the authors’ own dataset, without experiments on other well-established benchmarks such as OpenEQA.
OpenEQA is not for this problem because it has no spatially grounded conversation. In SpaCR, the conversation evolves through the frames; each statement is grounded to an object in view, and this dialogue plays as important a role as the object records and the frames.
OpenEQA also draws on ScanNet, so we already use the same scans for object spatial information.
W5. The paper lacks a quantitative analysis of overall computational efficiency.
We add the qualitative analysis in App. L, Tab. 12. Spatial reconstruction runs offline on past frames and is shared across questions. It can be batched over each set of past frames before the questions that refer to them. In our experiment, offline processing takes about 3 min per batch on a single RTX 3090, so it does not add to answer latency:
| Perception stage | Model | Per batch (avg. 187 frames) |
|---|---|---|
| Depth and camera pose | Depth Anything 3 | 1 min 48 s |
| Instance masks | SegVGGT | 1 min 2 s |
| Appearance embeddings | DINOv2 | 25 s |
| Total (GPU) | 3 min 15 s |
In comparison, the native-video and keyframe methods re-process the prefix at every query, and native Gemini uses 3–4× our tokens. In practice, large-VLM tokens are the dominant recurring inference cost, since they are paid at every question. Here is the latency breakdown at query time:
| Method | Answer latency in query time, median s |
|---|---|
| Native Video (Gemini 3.7 Flash) | 11.4 |
| Native Video (Qwen3.8 27B) | 57.4 |
| Keyframe selection (GPT-5.6 Terra) | 31–32 |
| Gemini Agentic (Gemini 3.7 Flash) | 22.5 |
| M2A (GPT-5.6 Terra) | 35.2 |
| SpaC-MEM (GLM-5.3 Flash) | 13.7 |
| SpaC-MEM (GPT-5.6 Luna) | 21.8 |
Note that these are latencies induced by API calls, which vary due to network conditions.
W3 / Q2. Give ablation studies towards its robustness.
We add additional experiments in ablation studies (cf. Figure 9) on robustness by measuring object recall across different merging thresholds against the ScanNet annotation.
- Instance recall is the share of annotated objects matched one-to-one to a memory record within 0.5 m
- class-correct recall also requires that the record carries the right class.
Recall stays close to the released settings across all merge thresholds (89.9–93.8% vs. 92.0%); only merging fragments seen in the same frame hurts substantially (No co-visibility veto) (49.7%). Turning merging off raises recall slightly (95.1%), since any nearby record counts as a hit, but it splits the same scenes into 484 records instead of 390 and scatters each object's statements across fragments. Merging keeps one record per object for the conversational memory to attach statements to.
| Ablated setting | Parameter | Instance recall within 0.5 m (%) | Class-correct recall (%) |
|---|---|---|---|
| Released settings | – | 92.0 | 77.1 |
| Merge: appearance cosine (0.5) | 0.3 / 0.4 / 0.6 / 0.7 | 91.7 / 91.3 / 92.4 / 93.8 | 77.4 / 76.4 / 78.1 / 78.5 |
| Merge: point overlap (0.5) | 0.3 / 0.4 / 0.6 / 0.7 | 89.9 / 91.0 / 92.7 / 93.1 | 73.6 / 75.3 / 77.8 / 77.4 |
| Merge: radius (0.20 m) | 0.10 / 0.15 / 0.25 / 0.30 | 93.8 / 93.1 / 91.3 / 90.6 | 78.5 / 77.1 / 76.0 / 74.7 |
| Minimum observations (4) | 2 / 8 | 97.2 / 90.6 | 82.3 / 75.0 |
| Match: radius (0.10 m) | 0.05 / 0.20 | 95.5 / 88.9 | 81.2 / 74.3 |
| Match: point overlap (0.15) | 0.10 / 0.30 | 92.0 / 94.1 | 77.1 / 78.8 |
| Match: ambiguity margin (0.10) | 0 / 0.20 | 96.2 / 91.0 | 82.6 / 77.4 |
| Match: centroid gate (1.0 m) | 0.5 / 2.0 | 88.2 / 91.7 | 73.3 / 77.4 |
| Lifting: SegVGGT score (0.10) | 0.05 / 0.20 | 91.7 / 92.0 | 75.3 / 78.5 |
| No fragment merging | – | 95.1 | 79.5 |
| No co-visibility veto | – | 49.7 | 41.3 |
Due to the constraint in discussion time, we only run on 10 scenes; we will run the full sweep on all scenes for the camera-ready version. The 10 scenes are harder than average for class labels, which explains the lower class-correct recall in the released-settings row (77.1%, vs. 82% over all scenes).
W1. What is the contribution beyond combining existing techniques?
We want to clarify that our work does not address an embodied-agent problem. We study agentic conversational memory for an egocentric personal assistant: how it remembers past user interactions and preserves the surrounding spatial context across days and locations. The conversation that contains user history is the first-class part of the memory we want to preserve, while the 3D representation keeps the physical context needed to interpret it. To our knowledge, prior work on conversational memory has not studied multi-session, multi-room dialogue grounded in persistent physical space. We do not claim novelty in the existing spatial reconstruction techniques we use. Building on this idea, we make the following contributions:
Problem and benchmark. We define SpaCR and build Ego-SpaCR to test the model's ability to memorize user semantic information that needs to be grounded in the surrounding environment. This makes conversation a first-class part of the task. Its questions start from what the user said and additionally require metric 3D relations over the objects those statements refer to. Existing scene-memory and conversational-memory benchmarks do not test these together.
Finding. RQ2 and RQ3 show that preserving user semantics across sessions requires keeping the geometry of the objects those statements refer to. The spatial information supports the conversational memory by maintaining this physical context instead of asking the answer model to infer it again from frames.
Conversational Memory. SpaC-MEM treats each user statement as a first-class memory record and links it to the persistent physical object. The answer model reasons from the conversation and uses the object's geometry as spatial context.
W2 / Q3. A complete dataset example
We added a complete end-to-end worked example to App. N (Fig. 14, Tab. 14). That shows the exemplar dialogue within the datasets and a walk-through of the SpaC-MEM.
This response cannot show the frames and figures, so we include a short dialogue excerpt below. The conversation carries the key facts that decide the answer, while knowing which object each statement refers to, and how large it is, requires spatial information reconstructed from the frames:
For answerability, we describe a strict verification process in Fig. 4 and App. B.6, with an independent Writer and Reader, to make sure all the clues required to answer are in the history.
We also ran a human review of 100 questions from 20 conversations. Checking a question by hand is hard: a conversation spans 7.6 sessions and 268 turns on average, and almost every question an annotator rejected needs sizes, distances, or viewpoints judged by eye from the video. Even so, two independent annotators confirmed most questions as answerable with a correct answer (87%), with high agreement between them (92%; App. O).
For practical value, the five tasks (inventory, room tour, lab or office setup, workspace setup, cleaning and organizing) are everyday situations in which people attach facts to objects.
W3 / Q2. The exact consolidation rules, thresholds, and appearance similarity.
We added them to the App. M. Here is the summary:
Lifting: DA3 confidence ≥ 0.5, SegVGGT score ≥ 0.10, depth-consistent mask core, 5 cm voxels.
Per-frame match: same class, centroid within 1.0 m, ≥ 0.15 point overlap within 0.10 m, and a 0.10 margin over the runner-up. Assignment is joint per frame, so co-visible same-class objects never merge.
Fragment merge: ≥ 50% overlap within 0.20 m, never co-visible, and a shared SegVGGT track or cosine similarity ≥ 0.5 of DINOv2 descriptors pooled over each object's masked pixels. Objects with fewer than 4 observations are dropped.