acceptodds
Under review as a conference paper at ICLR 2027

Have I Scene This Before? Spatially Grounded Conversational Memory for Complex Queries in Egocentric Assistants

Abstract

Personal assistants are evolving from search and text interfaces toward interactive multimodal systems that retain spatial and conversational history to support ongoing tasks. Effective assistants must therefore reason jointly over observed scenes and past dialogue to address complex requests. We formalize this challenge as *Spatially grounded Conversational Reasoning* (SPACR), covering cross-scene, recall-oriented, and counterfactual spatial queries that require accurate geometric reasoning and the integration of dialogue with spatial evidence. Existing approaches are poorly suited to this setting. Direct VLMs incur high inference costs and context limits as multimodal histories grow. Keyframe selection and retrieval can omit evidence needed for complete recall, while memory systems must preserve both conversational facts and object geometry to support spatial reasoning. To bridge this gap, we propose *Spatially grounded Conversational Memory* (SPAC-MEM), an object-centric working memory that applies 3D reconstruction and segmentation to egocentric observations, thereby spatially grounding conversational context directly to physical objects. It compresses long multimodal histories while preserving spatial and conversational evidence and allowing object-specific information to be updated through later dialogue. We further introduce *EGO-SPACR*, an egocentric SPACR dataset comprising 620 video sessions from ScanNet, augmented with 95 conversations averaging 230 dialogue turns each, and 3,100 evaluation queries. Experiments show that SPAC-MEM achieves the highest overall answer accuracy among the evaluated methods and improves object recall, while requiring substantially fewer reference input tokens than native-video baselines. Analysis shows that removing 3D spatial information substantially degrades performance, highlighting the importance of preserving both spatial and conversational evidence for an egocentric assistant.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
conversational agent, egocentric, spatial reasoning, spatial memory, conversational memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WZDEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work focuses on the spatially grounded conversational reasoning task for embodied assistants. It proposes to jointly preserve observed scenes and past dialogue to address complex requests. To facilitate research on this task, the authors construct the EGO-SPACR dataset based on ScanNet. Furthermore, they propose a framework named SPAC-MEM to tackle the SPACR task.

Critical strengths

The authors introduce the SPACR task and construct a corresponding dataset. In my view, this task represents a meaningful step forward for long-horizon embodied tasks. However, the insufficient analysis of dataset examples weakens the contributions of both the proposed task and the dataset.

Critical weaknesses
  1. The proposed method appears to be a combination of existing techniques, integrating 3D reconstruction, instance segmentation, object association, and large language models. Although it adopts an object-centric memory structure, its technological novelty remains limited.
  2. The presentation and validation of dataset quality are insufficient. The paper lacks complete examples connecting actual visual observations, continuous dialogue, object references, spatial evidence, and answers. This makes it difficult to assess the naturalness of the synthetic conversations, the answerability of the questions, and the practical value of the proposed task.
  3. The implementation details and robustness evaluation of the object consolidation mechanism are insufficient. The authors repeatedly use vague descriptions such as “overlap substantially” and “similar appearance” without specifying the relevant thresholds or how appearance similarity is computed, limiting reproducibility. The paper also lacks a sensitivity analysis of the merging thresholds and examples of incorrect merges or splits. Furthermore, its reliance on spatial alignment raises concerns about handling object movement in real-world applications, which could lead to duplicate records or outdated spatial information.
  4. The proposed method is evaluated only on the authors’ own dataset, without experiments on other well-established benchmarks such as OpenEQA [1].
  5. The paper lacks a quantitative analysis of overall computational efficiency. [1] Majumdar A, Ajay A, Zhang X, et al. Openeqa: Embodied question answering in the era of foundation models[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 16488-16498.
Questions for authors
  1. Please provide experiments on other relevant datasets.
  2. Please clarify object consolidation mechanism and give ablation studies towards its robustness.
  3. Please give more illustration of the proposed dataset.
3 comments

W1. What is the contribution beyond combining existing techniques?

We want to clarify that our work does not address an embodied-agent problem. We study agentic conversational memory for an egocentric personal assistant: how it remembers past user interactions and preserves the surrounding spatial context across days and locations. The conversation that contains user history is the first-class part of the memory we want to preserve, while the 3D representation keeps the physical context needed to interpret it. To our knowledge, prior work on conversational memory has not studied multi-session, multi-room dialogue grounded in persistent physical space. We do not claim novelty in the existing spatial reconstruction techniques we use. Building on this idea, we make the following contributions:

  • Problem and benchmark. We define SpaCR and build Ego-SpaCR to test the model's ability to memorize user semantic information that needs to be grounded in the surrounding environment. This makes conversation a first-class part of the task. Its questions start from what the user said and additionally require metric 3D relations over the objects those statements refer to. Existing scene-memory and conversational-memory benchmarks do not test these together.

  • Finding. RQ2 and RQ3 show that preserving user semantics across sessions requires keeping the geometry of the objects those statements refer to. The spatial information supports the conversational memory by maintaining this physical context instead of asking the answer model to infer it again from frames.

  • Conversational Memory. SpaC-MEM treats each user statement as a first-class memory record and links it to the persistent physical object. The answer model reasons from the conversation and uses the object's geometry as spatial context.

W2 / Q3. A complete dataset example

We added a complete end-to-end worked example to App. N (Fig. 14, Tab. 14). That shows the exemplar dialogue within the datasets and a walk-through of the SpaC-MEM.

This response cannot show the frames and figures, so we include a short dialogue excerpt below. The conversation carries the key facts that decide the answer, while knowing which object each statement refers to, and how large it is, requires spatial information reconstructed from the frames:

Day · roomSpeakerTurn (verbatim)
1 · officeUserI have decided to remove the table. I plan to handle the table later, and I want to leave the table until last.
………
2 · classroomUserKeep the backpack accessible, and do not touch the backpack.
………
3 · roomUserI want to leave the box until last. I defer the decision on the box, I plan to review the box later, and Sam is responsible for the box.
3 · roomAssistantI’ll place Sam’s responsibility with the box entry and keep the later review separate from its deferred decision and last priority.
………
3 · roomUser (question)Among all distinct objects we saw that I said “do not touch” about or said were last, how many are small enough in every dimension to fit within the space the table occupies in this scene?

For answerability, we describe a strict verification process in Fig. 4 and App. B.6, with an independent Writer and Reader, to make sure all the clues required to answer are in the history.

We also ran a human review of 100 questions from 20 conversations. Checking a question by hand is hard: a conversation spans 7.6 sessions and 268 turns on average, and almost every question an annotator rejected needs sizes, distances, or viewpoints judged by eye from the video. Even so, two independent annotators confirmed most questions as answerable with a correct answer (87%), with high agreement between them (92%; App. O).

For practical value, the five tasks (inventory, room tour, lab or office setup, workspace setup, cleaning and organizing) are everyday situations in which people attach facts to objects.

W3 / Q2. The exact consolidation rules, thresholds, and appearance similarity.

We added them to the App. M. Here is the summary:

  • Lifting: DA3 confidence ≥ 0.5, SegVGGT score ≥ 0.10, depth-consistent mask core, 5 cm voxels.

  • Per-frame match: same class, centroid within 1.0 m, ≥ 0.15 point overlap within 0.10 m, and a 0.10 margin over the runner-up. Assignment is joint per frame, so co-visible same-class objects never merge.

  • Fragment merge: ≥ 50% overlap within 0.20 m, never co-visible, and a shared SegVGGT track or cosine similarity ≥ 0.5 of DINOv2 descriptors pooled over each object's masked pixels. Objects with fewer than 4 observations are dropped.

W3 / Q2. Examples of incorrect merges or splits.

Since we cannot add images in the discussion, we added examples of incorrect merges and splits to App. M, Fig. 13. These errors hold adjacent objects that look similar in one record or one object in several, which is a known limitation of most segmentation models. However, most objects still end up in memory near their true location (91% within 0.5 m).

W3 / Q2. Object movement: duplicate records and outdated spatial information.

Like existing 3D scene memory datasets such as OpenEQA [6] and SIMMC 2.1 [7], Ego-SpaCR assumes static rooms so that the answers are exact and executable.

SpaC-MEM already handles semantic change: it stores the dialogue on each object in temporal order, so when the user changes their mind about an object, the new statement is kept after the earlier ones rather than overwriting them. Objects that physically move are an interesting next step for this work.

W4 / Q1. The proposed method is evaluated only on the authors’ own dataset, without experiments on other well-established benchmarks such as OpenEQA.

OpenEQA is not for this problem because it has no spatially grounded conversation. In SpaCR, the conversation evolves through the frames; each statement is grounded to an object in view, and this dialogue plays as important a role as the object records and the frames.

OpenEQA also draws on ScanNet, so we already use the same scans for object spatial information.

W5. The paper lacks a quantitative analysis of overall computational efficiency.

We add the qualitative analysis in App. L, Tab. 12. Spatial reconstruction runs offline on past frames and is shared across questions. It can be batched over each set of past frames before the questions that refer to them. In our experiment, offline processing takes about 3 min per batch on a single RTX 3090, so it does not add to answer latency:

Perception stageModelPer batch (avg. 187 frames)
Depth and camera poseDepth Anything 31 min 48 s
Instance masksSegVGGT1 min 2 s
Appearance embeddingsDINOv225 s
Total (GPU)3 min 15 s

In comparison, the native-video and keyframe methods re-process the prefix at every query, and native Gemini uses 3–4× our tokens. In practice, large-VLM tokens are the dominant recurring inference cost, since they are paid at every question. Here is the latency breakdown at query time:

MethodAnswer latency in query time, median s
Native Video (Gemini 3.7 Flash)11.4
Native Video (Qwen3.8 27B)57.4
Keyframe selection (GPT-5.6 Terra)31–32
Gemini Agentic (Gemini 3.7 Flash)22.5
M2A (GPT-5.6 Terra)35.2
SpaC-MEM (GLM-5.3 Flash)13.7
SpaC-MEM (GPT-5.6 Luna)21.8

Note that these are latencies induced by API calls, which vary due to network conditions.

W3 / Q2. Give ablation studies towards its robustness.

We add additional experiments in ablation studies (cf. Figure 9) on robustness by measuring object recall across different merging thresholds against the ScanNet annotation.

  • Instance recall is the share of annotated objects matched one-to-one to a memory record within 0.5 m
  • class-correct recall also requires that the record carries the right class.

Recall stays close to the released settings across all merge thresholds (89.9–93.8% vs. 92.0%); only merging fragments seen in the same frame hurts substantially (No co-visibility veto) (49.7%). Turning merging off raises recall slightly (95.1%), since any nearby record counts as a hit, but it splits the same scenes into 484 records instead of 390 and scatters each object's statements across fragments. Merging keeps one record per object for the conversational memory to attach statements to.

Ablated settingParameterInstance recall within 0.5 m (%)Class-correct recall (%)
Released settings–92.077.1
Merge: appearance cosine (0.5)0.3 / 0.4 / 0.6 / 0.791.7 / 91.3 / 92.4 / 93.877.4 / 76.4 / 78.1 / 78.5
Merge: point overlap (0.5)0.3 / 0.4 / 0.6 / 0.789.9 / 91.0 / 92.7 / 93.173.6 / 75.3 / 77.8 / 77.4
Merge: radius (0.20 m)0.10 / 0.15 / 0.25 / 0.3093.8 / 93.1 / 91.3 / 90.678.5 / 77.1 / 76.0 / 74.7
Minimum observations (4)2 / 897.2 / 90.682.3 / 75.0
Match: radius (0.10 m)0.05 / 0.2095.5 / 88.981.2 / 74.3
Match: point overlap (0.15)0.10 / 0.3092.0 / 94.177.1 / 78.8
Match: ambiguity margin (0.10)0 / 0.2096.2 / 91.082.6 / 77.4
Match: centroid gate (1.0 m)0.5 / 2.088.2 / 91.773.3 / 77.4
Lifting: SegVGGT score (0.10)0.05 / 0.2091.7 / 92.075.3 / 78.5
No fragment merging–95.179.5
No co-visibility veto–49.741.3

Due to the constraint in discussion time, we only run on 10 scenes; we will run the full sweep on all scenes for the camera-ready version. The 10 scenes are harder than average for class labels, which explains the lower class-correct recall in the released-settings row (77.1%, vs. 82% over all scenes).

Reviewer_ugrfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a new long video spatial understanding benchmark that aims to capture multiple aspects of spatial understanding and reasoning from long videos that capture days of interactions. The authors propose to use multiple tools to frame an object centric memory that is directly fed into the vlm to aid the question answering. The proposed benchmark indicates several severe limitations of sota vlms and also validates that a simple and intuitive text memory can help the VLM spatial grounding.

Critical strengths

The paper is well presented and the benchmark tackles a useful and challenging task. Several of the approaches of the proposed method are intuitive and rational while the proposed approach also reduces the token count of the VLM. The authors also design several informative ablations and evaluate multiple state-of-the-art VLMs.

Critical weaknesses

Dialogues design. Given that the dialogues were generated from a VLM, it is not fully validated that the benchmark can capture the true real-world performance of the models. Have the authors validated any potential biases that these generated dialogues may contain?

Scaling experiments. The proposed approach of feeding the whole memory in the VLM has a notable limitation and cannot scale in real world where humans interact with thousands of objects weekly. The authors have not fully experimented on the memory scalability and have not thoroughly discussed this limitation.

Ablations. a) Given that the proposed object centric memory could explode in large scenes with thousands of objects, it would be extremely beneficial to propose some ways to mitigate such cases. E.g. have the authors tested any relevance filtering in the memory before feeding to the VLM? Would that introduce any significant errors?

b) The geometry understanding is relatively poor compared to other type of questions. Therefore one concern that comes to my mind is that the structure of the geometry that is fed through text does not enable the full VLM capabilities. Have the authors validated this technical choice against any other alternatives e.g. VGGT features, functional coding etc?

Comparison with existing approaches. The authors do provide an extensive discussion with some existing approaches in the field but they do not compare their approach and their benchmark design with some very relevant existing works in long video understanding (e.g. [1,2,3]. It would be useful if the authors could discuss their main contributions against those approaches.

Unfair evaluation. The proposed approach gets explicit 3D information from external tools whereas the baseline models receive only raw videos. A more fair comparison would be to either consider spatially grounded VLMs or to also provide the 3D information to the baselines.

Inference runtime The inference runtime cost is not only tight to tokens as the proposed approach requires a series of tools (which all are quite expensive) that would highly affect the performance of the model. A complete inference runtime comparison would be a more fair experiment in this case.

Dynamic videos The authors should explicitly stretch the limitation that the proposed approach and benchmark can only capture static scenes.

[1]: Jiang et al. MEMENTO: TOWARD AN ALL-DAY PROACTIVE ASSISTANT FOR ULTRA-LONG STREAMING VIDEO, ICLR 2026 [2]: Alam et al. SuperMemory-VQA: An Egocentric Visual Question-Answering Benchmark for Long-Horizon Memory, 2026 [3]: Zilliotto et al. Personal-3D: A Comprehensive Benchmark for Personalized Embodied AI Agents, IJCV 2026

Questions for authors

I would appreciate if the authors could clarify my concerns especially regarding the scaling of the memory, the unfair evaluation and the dialogue designs. Those are my main concerns about this work and clarifying them would significantly strengthen my opinion on this work.

3 comments

W1. Are the generated dialogues validated, and have their biases been checked?

As we show in Fig. 4 and App. B.6, there is a strict validation process for the generated dialogue.

The Writer never sees the answer or the matching set. Ground truth is executed over facts an independent Reader recovers from the generated dialogue, and a question is kept only if the Reader's facts, every reference resolution, and the executed answer agree with the plan (Sec. 5.1, App. B.6). In addition, any referring expression is used only where an independent VLM, without the target cue, resolves it to that object alone (Tab. 5).

As Table 2 shows, shortcuts do not work: guessing the answer prior scores 14.1%, and reading only the dialogue scores 27.1%. We also added an end-to-end worked example to App. N. We ran a human review of 100 questions from 20 conversations. Checking a question by hand is hard: a conversation spans 7.6 sessions and 268 turns on average, and almost every question an annotator rejected needs sizes, distances, or viewpoints judged by eye from the video. Even so, two independent annotators confirmed most questions as answerable with a correct answer (87%), with high agreement between them (92%; App. O).

W2. How does the memory scale as objects grow?

Memory grows by about 112 tokens per distinct object rather than by frames or dialogue turns. Across the observed range, object recall remains about 71% from fewer than 50 to 300 objects, while accuracy declines from 58% to 50% (App. H.2, Fig. 11; App. K, Tab. 11). However, the record filter reduces the full-set input from 20.1k to 12.8k tokens at similar accuracy and improves the ≥300-object subset from 34.6% to 44.9% (Sec. 5.2, Fig. 8). We added the scaling curve and this discussion to Sec. 5.2 (Fig. 8) and App. K.

W3a. Does relevance filtering before the VLM work, and does it introduce errors?

We ran additional experiments on relevance filtering, and it preserves overall accuracy while reducing the context:

SpaC-MEM + GLM-5.3 FlashAccuracyObject recallTokens (k)
All questions, full memory51.468.920.1
All questions, filtered50.567.912.8
≥300 objects, full memory34.657.245.7
≥300 objects, filtered44.970.727.6

The filter keeps 39% of the records and uses about two-thirds of the tokens, and on the largest memories it is better than the full memory. We added this to Sec. 5.2 (Fig. 8) and App. K.

SpaC-MEM can apply this filter because each semantic piece of information sits on its object. Once association has run, the filter is deterministic and makes no large context LLM call.

W3b. Geometry is relatively poor compared with other question types. Does the text format limit the VLM, and was it validated against alternatives such as VGGT features or functional coding?

As Fig. 7c shows, annotated geometry in the same format with the same answer model raises accuracy by 11 points on C3 and 11–19 points on C4. This shows that reconstruction quality is a major source of the remaining error, rather than the text format alone. Text is also standard LLM input for 3D scenes (ConceptGraphs [1], SayPlan [2]), and TRACE [3] and Thinking in Space [4] show that explicit representations improve MLLM spatial reasoning.

  • For VGGT features, we keep geometry as text by design, because any answer model can read it without training or tools, while features need a trained projector tied to one model.

  • For functional coding, we read this as letting the model compute through code, and we ran a pilot on C3. It runs on the same text records and helps, but it stays below simply reading annotated geometry as text, and it costs about 3.4× the input tokens. We added this to App. I.

C3, GLM-5.3 FlashAccuracyInput tokens (k)
Reconstructed geometry, text3819.1
Reconstructed geometry, computed in Python4565.8
Annotated geometry, text5314.8

These results show that reading geometry as text is not the main problem. Reconstruction quality is still a major source of error.

W4. How do the method and benchmark compare with MEMENTO, SuperMemory-VQA and Personal-3D?

We added all three to Sec. 2 and Tab. 3. None of them can evaluate SpaCR, because they all lack (1) a conversation whose user-stated facts are grounded to the object in view, (2) metric and counterfactual spatial reasoning over those objects, and (3) recall over the complete object set.

W5. Unfair evaluation

Every method starts from the same RGB video and dialogue. We compared each baseline's complete system, and no method receives information outside its own methodology, so we think this is a fair comparison.

SpaC-MEM estimates its geometry as part of the proposed methodology. Nevertheless, we also provide additional experiments below to address the matched-information question.

W5. Unfair evaluation: How do the baselines perform when given the 3D information?

For 3D information, as we show in RQ2 (Fig. 6), we already drew ground-truth boxes or exact measurements into the frames. Gemini native video gets a modest gain and stays below SpaC-MEM with estimated geometry.

We also added SpaC-MEM's object list (the estimated class, 3D position and size of every reconstructed object, without the attached dialogue) to the unchanged inputs of AKS. Given the discussion time, we use a sample of 400 C3 + 150 C4 questions, stratified by task.

MethodC3C4
AKS + GLM-5.3 Flash20.820.0
AKS + object list + GLM-5.3 Flash33.036.0
Gemini Agentic + object list34.237.2
SpaC-MEM + GLM-5.3 Flash39.559.3

Adding the object list improves AKS because it provides more object records than the method's original keyframe-based input. However, its accuracy remains lower than SpaC-MEM. We added this result to App. I.

W5. Unfair evaluation: A fairer comparison would be to consider spatially grounded VLMs.

We chose Cambrian-S-7B [5], the largest release of a recent video model trained for spatial reasoning. It reaches 15.8% overall:

Cambrian-S-7BC1C2C3C4All
Accuracy19.822.49.520.315.8

Its accuracy falls from 31.6% after one session to 8.8% after seven or more. Like the other video models we test (Fig. 5), it loses earlier context as the history grows, which its authors also report on long videos [5]. We added this to App. I.

W6. Inference runtime

Spatial reconstruction runs offline on past frames and is shared across questions. It can be batched over each set of past frames before the questions that refer to them. In our experiment, offline processing takes about 3 min per batch on a single RTX 3090, so it does not add to answer latency

Perception stageModelPer batch (avg. 187 frames)
Depth and camera poseDepth Anything 31 min 48 s
Instance masksSegVGGT1 min 2 s
Appearance embeddingsDINOv225 s
Total (GPU)3 min 15 s

We report query-time latency separately, which is required at every question:

MethodAnswer latency in query time, median s
Native Video (Gemini 3.7 Flash)11.4
Native Video (Qwen3.8 27B)57.4
Keyframe selection (GPT-5.6 Terra)31–32
Gemini Agentic (Gemini 3.7 Flash)22.5
M2A (GPT-5.6 Terra)35.2
SpaC-MEM (GLM-5.3 Flash)13.7
SpaC-MEM (GPT-5.6 Luna)21.8

Note that these are latencies induced by API calls, which vary due to network conditions. We added this to App. L, Tab. 12.

W7. Dynamic videos.

Like existing 3D scene memory datasets such as OpenEQA [6] and SIMMC 2.1 [7], Ego-SpaCR assumes static rooms so that the answers are exact and executable.

SpaC-MEM already handles semantic change: it stores the dialogue on each object in temporal order, so when the user changes their mind about an object, the new statement is kept after the earlier ones rather than overwriting them. Objects that physically move are an interesting next step for this work.

References

[1] Q. Gu et al. ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning. ICRA 2024.

[2] K. Rana et al. SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning. CoRL 2023.

[3] J. Hua et al. Unleashing Spatial Reasoning in Multimodal Large Language Models via Textual Representation Guided Reasoning. ACL 2026.

[4] J. Yang et al. Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces. CVPR 2025.

[5] S. Yang et al. Cambrian-S: Towards Spatial Supersensing in Video. ICLR 2026.

[6] A. Majumdar et al. OpenEQA: Embodied Question Answering in the Era of Foundation Models. CVPR 2024.

[7] S. Kottur et al. SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations. EMNLP 2021 (SIMMC 2.1 is its DSTC11 release).

I thank the authors for their reply. They have addressed my concerns.

Reviewer_3M6zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies egocentric assistants that must answer spatial questions about objects users discussed previously, across rooms and sessions (e.g., “How many of Dana’s things on the tables would fit in this box?”). It formalizes this task as Spatially Grounded Conversational Reasoning (SpaCR), characterized by four levels of spatial complexity—object grounding, qualitative relations, geometric properties, and counterfactual arrangements—as well as a recall-breadth dimension. It also distinguishes corrections to previously stated facts from genuine changes in the underlying state.

The proposed method, SpaC-MEM, constructs an object-centric working memory through five stages:

  1. 3D reconstruction: Reconstruct per-frame 3D object observations from RGB inputs using Depth Anything 3 and a segmentation model with a predefined vocabulary.
  2. Object association: Merge observations into persistent object identities based on same-class 3D overlap.
  3. Dialogue grounding: Use an LLM to associate each dialogue turn with relevant object records while preserving the verbatim utterance.
  4. Memory organization: Maintain one persistent record per object.
  5. Answer generation: Provide the full memory to an LLM to answer spatial questions.

The authors also introduce Ego-SpaCR, a benchmark comprising 95 LLM-generated conversations (approximately 230 turns each) spanning 620 ScanNet clips across 39 scenes. It includes 3,100 open-ended questions whose answers are computed programmatically from ScanNet geometry.

SpaC-MEM achieves 51.5–52.5% accuracy, outperforming the Gemini agentic baseline (43.1%), Gemini’s native-video approach (41.4%), and the strongest baseline using the same GLM answer model (27.6%). It also consumes approximately one-quarter of the answer-model input tokens required by native-video methods. Ablating 3D information reduces accuracy to 34.9%, highlighting the importance of explicit 3D spatial representations for conversational spatial reasoning.

Critical strengths
  1. Well-motivated problem and a clean formalization. The paper combines long-term conversational memory with 3D spatial reasoning over persistent objects. The fact representation ϕ = (o, p, x), the correction vs. state-change distinction, and the complexity × recall-breadth design (Sec. 3, Fig. 2) are clear and reusable, and arguably the most novel part of the paper.
  2. Verifiable benchmark construction. Answers come from deterministic programs over ScanNet geometry and the conversational state recovered by the Reader. Queries are kept only when their prerequisites precede them and execution reproduces the target answer (Sec. 5.1, App. B).
  3. Large gains with a matched answer model. With GLM-5.3 Flash as the answerer for all methods, SpaC-MEM reaches 51.5% vs. 27.6% for the best baseline (Table 2/6). On C4 counterfactual queries the gap is largest: about 60% vs. at most 37% across all baselines.
  4. Useful diagnostics. These include object recall as history grows (about 70% for SpaC-MEM vs. 73% → 35% for per-frame memory; App. H.1), LLM vs. human dialogue association (89% of accuracy retained), and reconstruction fidelity against ScanNet annotations (App. H.3).
Critical weaknesses
  1. Limited novelty; the new component is not isolated (Q1). Most of the pipeline (lifting masks to 3D, merging by overlap, serialising objects for an LLM) follows ConceptGraphs-style object mapping, which the paper cites but does not compare against. The only new step, attaching dialogue turns to object records, is never ablated: Fig. 7c tests 3D, and Fig. 7d tests association accuracy.
  2. Baselines lack 3D information (Q2) and unclear evaluation reporting (Q3). 45.9% of questions are metric, yet only SpaC-MEM receives 3D boxes and centroids; all baselines see RGB only. Evaluation results are not broken down by answer type (85% count, 13% object, 1% Boolean).
Questions for authors
  1. Beyond attaching dialogue turns to object records, what does SpaC-MEM add over existing object-level 3D maps such as ConceptGraphs? How does "same 3D object list + full dialogue in context, without per-object attachment" perform under the same answer model?
  2. How do the strongest baselines (Gemini agentic, AKS + GPT-5.6 Terra) perform when given the same reconstructed object list or a measurement tool?
  3. Since the answer type is strongly imbalanced, could you report accuracy by answer type (count / object / Boolean)?
8 comments

W1 / Q1. Beyond attaching dialogue turns to object records, what does SpaC-MEM add over existing object-level 3D maps such as ConceptGraphs?

SpaC-MEM addresses a different problem and domain from ConceptGraphs. We want to clarify that our work does not address an embodied-agent problem. We study agentic conversational memory for an egocentric assistant: how it remembers past user interactions and preserves the surrounding spatial context across days and locations. The conversation is the memory we want to preserve, while the 3D representation keeps the physical context needed to interpret it. ConceptGraphs is designed for embodied agents that map and navigate a physical environment.

To our knowledge, prior work on conversational memory has not studied multi-session dialogue grounded in persistent physical space. Our paper is therefore about conversational memory grounded in space, not a new 3D mapping pipeline. Building on this idea, we have the following contributions:

  • Problem and benchmark. We define SpaCR and build Ego-SpaCR to test the model's ability to memorize user semantic information that needs to be grounded in the surrounding environment. This makes conversation a first-class part of the task. Its questions start from what the user said and additionally require metric 3D relations over the objects those statements refer to. Existing scene-memory and conversational-memory benchmarks do not test these together.

  • Finding. RQ2 and RQ3 show that preserving user semantics across sessions requires keeping the geometry of the objects those statements refer to. The spatial information supports the conversational memory by maintaining this physical context instead of asking the answer model to infer it again from frames.

  • Conversational Memory. SpaC-MEM treats each user statement as a first-class memory record and links it to the persistent physical object. The answer model reasons from the conversation and uses the object's geometry as spatial context. Reconstruction, segmentation, and object mapping are replaceable components, not our novelty claim.

W1 / Q1. How does "same 3D object list + full dialogue in context, without per-object attachment" perform under the same answer model?

The key issue is that the answer model in the variant must connect the full dialogue to the object list again for every question. But we agree that the variant should be ablated (added the result to App. I, Tab. 8). We ran this on Ego-SpaCR with the answer models of Table 2. The answer model reads three things: the 3D object list, the full dialogue in context, and a line before each user turn listing the objects in view.

3D object list + full dialogue in context + objects in viewC1C2C3C4All
GPT-5.6 Luna86.664.844.064.656.5
GLM-5.3 Flash88.262.944.872.658.5

The results suggest this variant is slightly more accurate (56.5 / 58.5 vs. 52.6 / 51.9). But the proposed approach of attaching the dialogue has more significant advantages in terms of efficiency and reuse:

  • SpaC-MEM uses 16–18% fewer input tokens overall and 18–20% fewer after seven or more sessions.
Input tokens per query (k), Luna / GLMAfter 1 sessionAfter 7+ sessionsOverall
SpaC-MEM6.9 / 6.732.3 / 33.519.5 / 20.0
This variant7.5 / 7.439.6 / 42.123.3 / 24.5
Saving8–10%18–20%16–18%
  • SpaC-MEM can apply object relevance filtering based on user request because each semantic information is attached to its object (cf. Sec. 5.2, Fig. 8). It brings SpaC-MEM to 12.7k tokens, about half of the variant's 24.5k with GLM (cf. App. K).

  • SpaC-MEM runs association once, and every later question reuses it, while the variant repeats that work inside each answer request, making this difference more pronounced as requests increase.

W2 / Q2. Do the baselines lack 3D input?

Every method starts from the same RGB video and dialogue. We compared each baseline's complete system, and no method receives information outside its own methodology, so we think this is a fair comparison.

SpaC-MEM estimates its geometry as part of the proposed methodology. Nevertheless, we also provide additional experiments below to address the matched-information question.

W2 / Q2: how do they perform when given the object list or measurements?

We give the baselines the same measurements or reconstructed object list. For the measurements, as we show in RQ2 (Fig. 6), we already drew ground-truth boxes or exact measurements into the frames. Gemini native video gets a modest gain and stays below SpaC-MEM with estimated geometry.

For the object list, we added SpaC-MEM's object list to the unchanged inputs of AKS. We report AKS with GLM-5.3 Flash rather than GPT-5.6 Terra because GLM is consistent with SpaC-MEM's answer model, so the difference measures the input and not the answer model. Given the discussion time, we use a sample of 400 C3 + 150 C4 questions, stratified by task.

MethodC3C4
AKS + GLM-5.3 Flash20.820.0
AKS + object list + GLM-5.3 Flash33.036.0
Gemini Agentic + object list34.237.2
SpaC-MEM + GLM-5.3 Flash39.559.3

Adding the object list improves AKS because it provides more object records than the method's original keyframe-based input. However, its accuracy remains lower than SpaC-MEM. We added this result to App. I.

Access to 3D information alone therefore does not explain the gap.

W2 / Q3. Since the answer type is strongly imbalanced, could you report accuracy by answer type (count / object / Boolean)?.

Accuracy by answer type (Boolean question dropped):

Method (answer model)CountObject
Conversation Only (GLM-5.3 Flash)23.251.7
Native Video (Gemini 3.7 Flash)35.773.4
Uniform (GPT-5.6 Terra)32.772.7
AKS (GPT-5.6 Terra)33.070.2
FOCUS (GPT-5.6 Terra)33.369.7
Gemini Agentic (Gemini 3.7 Flash)38.072.7
Multimodal RAG (GPT-5.6 Terra)7.735.0
Summary RAG (GLM-5.3 Flash)12.339.9
M2A (GPT-5.6 Terra)20.056.8
SpaC-MEM (GLM-5.3 Flash)47.377.8
SpaC-MEM (GPT-5.6 Luna)48.476.3

The imbalance does not drive our result: SpaC-MEM is the most accurate on both count and object answers (98.8% of the set). The gain is largest on counts (9–10 points), because a count needs every matching object recalled across sessions, which is what Ego-SpaCR tests (Sec. 3). Boolean questions are only 38, so one question moves accuracy by 2.6 points and that column cannot rank methods. We added this table to App. J, Tab. 10.

Thank you for the detailed response and the substantial additional experiments in the revision. Below is where my concerns stand.

Resolved

  • Accuracy by answer type (Q3). Appendix J answers this. SpaC-MEM is best on count and object questions, which together make up 98.8% of the benchmark. I agree that 38 Boolean questions are too few to rank methods, but for that reason a benchmark should not keep them as a separate answer type. Please either expand the Boolean questions to a size that supports per-type evaluation, or remove them from the benchmark.

Not resolved

  • Novelty and the role of attachment (Q1). Thank you for running the no-attachment control. It is, however, more accurate than SpaC-MEM (56.5 / 58.5 vs. 52.6 / 51.9 with Luna / GLM), and Appendix I states "we do not attribute SPAC-MEM's accuracy to permanent attachment." The only new component of the method therefore saves 16–18% of answer tokens at a cost of 4–7 accuracy points. The abstract ("achieves the highest overall answer accuracy"), Table 2 and the contribution list still present SpaC-MEM as the source of the accuracy gain. They should be revised, and the no-attachment variant should appear in the main table.
  • Baselines with 3D information (Q2). Adding the object list to AKS recovers about 65% (C3) and 41% (C4) of its gap to SpaC-MEM, which confirms that 3D object information is a major source of the gain. The remaining gap does not show that SpaC-MEM's design is responsible: the no-attachment variant from Q1 reaches 44.8 / 72.6 on C3 / C4, above SpaC-MEM (38.2 / 60.3). The remaining gap more likely comes from aligning dialogue turns with the objects visible at that time, which the per-turn visibility line provides without attachment.

New concerns from the revision

  1. Association cost is described inconsistently. Appendix E.2 says f_assoc runs "for each query", while Appendices I and L say it "runs once per session". Please clarify which is correct, which model performs association in the GPT-5.6 Luna setting, and with what decoding settings.
  2. Question-type distribution. Since the benchmark is a main contribution, please report the distribution over C1–C4 subtypes, the complexity × answer-type cross-tabulation, the recall-breadth distribution, and the number of questions per task.
  3. Naturalness of the dialogue. The example in Appendix N, also quoted in the response to Reviewer WZDE, reads as templated, e.g., "I have decided to remove the table. I plan to handle the table later, and I want to leave the table until last." How representative is this of the corpus? Repeated, explicit naming of each object may make dialogue-to-object association easier than in natural conversation.

Overall, the authors' own results show that the proposed memory design does not explain the accuracy gain. I see the paper's value mainly in the task definition and the benchmark, so my main concerns remain. I keep my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.