acceptodds
Withdrawn from ICLR 2027

3DMem: Synergizing Textual and Spatial Memories for Conversational 3D Scene Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing 3D vision-language models are primarily developed for single-turn interactions, whereas natural dialogue requires a model to retain references and incorporate new constraints across turns. We introduce Conversational 3D Scene Understanding and ScanFlow, a benchmark for multi-turn question answering and grounding. Our framework, 3DMem, maintains two complementary recurrent states: a cumulative textual summary and a continuous spatial field aligned with scene superpoints. The spatial state is updated from the current causal prefix and previous predicted region, then read through language-side embeddings and residual modulation of grounding features. We construct continuous spatial supervision directly from point-cloud geometry, separating memory-region targets from exact instance-mask targets. Two supervised stages learn memory prediction and task responses using predicted memory trajectories, followed by IoU-aware optimization of the textual-memory policy with supervised grounding updates. This formulation connects evolving dialogue context to an explicit scene-indexed representation without generating discrete spatial buckets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Scene Understanding, 3D multimodal large language models, multi-turn dialogue

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kSznOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates 3D scene understanding in multi-turn conversations and introduces ScanFlow, a benchmark comprising 679 scenes and 17.3K dialogue turns, covering conversational question answering and 3D object grounding. The authors further propose 3DMem, which integrates textual memory and continuous spatial memory to model contextual information across dialogue turns. The framework is optimized through supervised fine-tuning and IoU-aware GRPO. Experiments compare the proposed method with several existing 3D-LLMs, and ablation studies are conducted to evaluate its effectiveness.

Critical strengths
  1. The research problem is meaningful. Existing 3D-LLMs primarily focus on single-turn interactions. The proposed ScanFlow provides a new benchmark for evaluating cross-turn reference disambiguation, context accumulation, and spatial grounding, addressing a practically relevant research challenge.

  2. The method design is reasonable. The dual-memory mechanism explicitly maintains conversational states and spatial associations, offering a more targeted solution than simply concatenating dialogue history. The experimental results also provide preliminary evidence of its effectiveness.

Critical weaknesses
  1. The fairness of experimental comparisons and the significance of performance improvements require further justification. The baselines in Table 3 are not fine-tuned on ScanFlow, which may exaggerate the reported improvements. Under the standardized SFT setting in Table 5, the performance gains over strong baselines become considerably smaller. Moreover, comparisons of training costs, parameter counts, and inference efficiency are missing, making it difficult to assess whether the additional architectural complexity is justified.

  2. The ablation studies are insufficient. Table 4 primarily analyzes the effects of disabling components during inference, without independently trained comparisons among Text-only, Spatial-only, and Dual Memory configurations. Furthermore, systematic ablations of IoU-aware GRPO and the individual training stages are missing, leaving the necessity of these components insufficiently demonstrated.

Questions for authors
  1. Could the authors provide comparisons with fully fine-tuned baselines under similar training budgets and parameter configurations, along with an analysis of the additional computational costs?

  2. Could the authors conduct independently trained comparisons among Text-only, Spatial-only, and Dual Memory configurations to demonstrate the necessity of the proposed dual-memory mechanism?

  3. Could the authors provide separate ablation studies for the different training stages and IoU-aware GRPO to clarify their individual contributions to the final performance?

  4. Could the authors provide more details about the quality of human verification in ScanFlow and evaluate the model's generalization to longer conversations and diverse dialogue expressions?

Reviewer_8Yb3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

this paper introduces ScanFlow, a benchmark for multi-turn 3D question answering and grounding, and proposes 3DMem to maintain context across dialogue turns. ScanFlow contains 3,524 dialogues over 679 scenes, generated with an MLLM and subsequently filtered and manually verified. Built on 3D-LLaVA, 3DMem maintains a textual summary and a continuous spatial state over scene superpoints. The spatial state is updated across turns and used by both the answer model and the grounding decoder. Training combines memory supervision, answer and grounding supervision using predicted memories, and IoU-aware optimization of the textual-memory policy. Experiments compare against existing 3D-language models, including baselines fine-tuned on ScanFlow.

Critical strengths

1.The task is worth studying. Multi-turn reference tracking is not well captured by single-turn QA.

2.ScanFlow includes both QA and grounding, connecting dialogue understanding to localization.

3.The authors report two independent manual checks of every dialogue after automatic filtering.

4.A scene-aligned spatial state is a sensible complement to textual memory.

5.Answer training uses predicted memories. Evaluation also counts missing triggers and invalid masks as failures.

  1. Table 5 includes task-fine-tuned baselines, which makes it more useful than the initial comparison.
Critical weaknesses
  1. Table 3 gives an overly favorable first impression: the baselines have not been fine-tuned on ScanFlow.
  2. After baseline fine-tuning, the gains are modest and mixed. For example, 3DRS has higher ROUGE-L, and Video-3D-LLM has higher [email protected].
  3. The training comparison remains unclear. Baselines use rank-16 LoRA, while 3DMem is described with larger adapters and additional training stages.
  4. Table 4 only disables components at inference. It does not replace a separately trained text-memory-only baseline or an ablation of policy optimization.
  5. The recurrence gain is small: mIoU rises from 0.272 to 0.281 on 120 dialogues. Its statistical uncertainty is not reported.
  6. The dialogue-length and error analyses concern the earlier discrete-memory model. The current model still needs direct evidence of better cross-turn stability.
Questions for authors

1、What training stages and LoRA ranks are used for the 3DMem row in Table 5?

2.Why do its grounding scores differ between Tables 3 and 5 while its QA scores stay identical?

3.How does a separately trained text-memory-only model perform under a comparable training budget?

4.How much does policy optimization add over the two SFT stages?

5.Does the current model’s advantage over history concatenation grow with dialogue length or turn number?

6.Can you provide paired confidence intervals for the Table 4 differences, preferably accounting for shared scenes?

Reviewer_7wCsOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper is about vision-language models (VLMs). It proposes a new task (a conversational VQA task and a new model as a solution. The challenge lies in maintaining spatial memory between steps/frames.

Critical strengths
  1. The paper is a complete piece of work on VLMs. It contributes a new task, a new dataset for evaluating it, and a new model as a solution, rather than only one of these.
Critical weaknesses
  1. The new task and its evaluation are not explained well enough to be understood or verified.

    • The paper gives no visual example of the new VQA task, neither as a teaser nor in Sec. 3. It is unclear how such conversational VQA works exactly, which makes Sec. 3 extremely difficult to read.
    • The "Autoregressive Protocol" paragraph (L195) is explained badly. I do not understand what L197–198 means in the context of this task, or how it is autoregressive.
    • The task names are inconsistent. L160 lists explicit grounding, direct localization, identity reasoning, spatial reference, visual reference, and interactive disambiguation, but Tab. 1 lists Attribute Description, Semantic Interpretation, Coordinate Retrieval, Causal Inference, Spatial Parsing, and Interactive Disambiguation. Five of the six names do not match and look nothing alike. This suggests the text and the table were written or revised separately. The "Grp." column in Tab. 1 is not explained anywhere, and the rows are not ordered by it.
    • L205: the metrics "Fine-grained Attribute Accuracy", "Multi-turn Context Coherence", and "Spatial Hallucination Rate" lack concrete definitions. I understand not spending space on equation blocks, but the authors should at least explain their formulation at a high level. Spatial Hallucination Rate does not even get a brief description.
  2. The method section is hard to follow because the figure, terms, and notation do not match the text.

    • L214: Memory LoRA and Answer LoRA appear out of nowhere, without a big picture or an overall introduction first (for example, "we introduce two modules for ..."). They are not shown in Fig. 1, the pipeline figure.
    • Fig. 1 is referenced at the start of Sec. 4 (L231) and then ignored until L367. The authors do not walk the reader through it. The terms Omni Superpoint Transformer, Spatial memory head, WvW_v, and WsW_s exist only in the figure and are never used in the writing.
    • Eq. 1 (L293): ckc_k and FF were last used at L240 with little context to recall what they mean. PFP_F, PCP_C, PQP_Q, and PMP_M are described only as "learned projections" instead of feature, center, query, and memory projections, which makes it worse. Eq. 1 is not comprehensible.
    • Eqs. 5 and 6 introduce numerous undefined notations and functions.
    • Naming is inconsistent: L314 says "a subsequent GRPO phase", while L339 says "Stage 3: IoU-Aware Policy Optimization".
    • Fig. 2 is too small to read.
Questions for authors

None.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.