acceptodds
Withdrawn from ICLR 2027

GeoMem3D: Learning to Reason over Structured 3D World Memory

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Large Language Models (MLLMs) have significantly advanced 3D spatial reasoning, yet their ability to maintain consistent spatial understanding across viewpoints remains limited. Recent methods improve spatial reasoning by extracting evidence from continuous visual observations or interactive exploration. However, this paradigm is constrained by observation-level evidence, limiting consistent spatial reasoning across views and queries. Motivated by this observation, we propose GeoMem3D, a learning framework that enables MLLMs to reason over a persistent and structured 3D world memory of all objects and their spatial relations. GeoMem3D reconstructs multi-view scenes within a shared geometric frame and organizes object entities, geometric attributes, and spatial relations into a query-independent structured 3D world memory, which provides persistent and reusable spatial knowledge for reasoning. Leveraging this memory, we explore two complementary reasoning paradigms: GeoMem-Holistic directly reasons over the complete structured memory, while GeoMem-Agentic adaptively accesses fine-grained spatial evidence from the memory during reasoning. We further introduce a two-stage learning strategy that combines supervised fine-tuning with reinforcement learning to establish effective memory utilization behaviors. Experiments on VSI-Bench-Tiny and MindCube-Tiny demonstrate consistent improvements across diverse spatial reasoning tasks, validating the effectiveness and generalizability of structured 3D world memory for spatial reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Spatial Reasoning, 3D World Memory, 3D Scene Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AD4iOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes GeoMem3D, a framework for 3D spatial reasoning that reconstructs multi-view observations into a query-independent, object-centric structured 3D world memory. It introduces two reasoning paradigms: GeoMem-Holistic, which reasons over the complete memory, and GeoMem-Agentic, which adaptively retrieves spatial evidence. Both are optimized through SFT followed by GRPO. Experiments on VSI-Bench-Tiny and MindCube-Tiny show improvements over the Qwen3-VL-8B backbone.

Critical strengths
  1. The paper aims to address an important challenge in 3D spatial reasoning: maintaining coherent geometric and semantic relationships across viewpoint-dependent observations.
  2. Organizing reconstructed geometry into explicit object entities and spatial relations provides an interpretable basis for reasoning.
Critical weaknesses

W1. The central design largely combines established components: 3D reconstruction, object-centric scene graphs, memory retrieval, and SFT–GRPO training. Related approaches, including ConceptGraphs, MSSR (ICLR 2026), and Scene Graph Thinking (ICML 2026), have already explored structured spatial representations or adaptive evidence-based reasoning. The paper does not sufficiently distinguish its methodological contributions from these approaches.

W2. Unclear training-data provenance and confounded performance gains. The authors construct over 5K SFT trajectories using Qwen3-VL-32B, with ground-truth answers provided during trajectory generation. However, the precise training sources, scene-level partitions, and potential overlap with evaluation data are insufficiently documented, raising concerns about answer-conditioned supervision and possible data leakage.

W3. Serious inconsistencies in experimental reporting. Figure 4 reports approximately >80% for GeoMem-Agentic on VSI-Bench-Tiny, whereas Table 1 reports only 59.6%. Similar discrepancies occur for other models and MindCube-Tiny. Furthermore, Figure 4 is labeled with Accuracy (%) despite appearing to reuse the results presented as Explainability Analysis in Figure 1(b). These inconsistencies obscure the reported experimental conclusions.

W4. Insufficient benchmarks and core-component ablations. Evaluation is restricted to VSI-Bench-Tiny and MindCube-Tiny, while Table 3 reports ablations only on the aggregate VSI-Bench-Tiny score. This is inadequate for a method whose principal motivation is improving cross-view consistency and structured spatial reasoning. Without task-specific ablations on relative direction, spatial relations, and route planning, or direct cross-view consistency evaluation, the results do not establish whether the proposed memory addresses the specific limitations identified in the introduction.

Questions for authors

Q: What are the exact sources and scene-level partitions of the 5K SFT trajectories? How is evaluation-data independence ensured when the teacher receives ground-truth answers?

1 comment

We sincerely thank the reviewer for the positive evaluation and recognition of the importance of cross-view spatial consistency and the interpretability of our structured 3D world memory.

Q1: GeoMem3D's Core Contributions

Existing studies can be broadly categorized into three directions:

(1) 3D Scene Modeling and Reconstruction. ConceptGraphs (ICRA 2024), HOV-SG (RSS 2024), and LEXI-SG (2026) primarily focus on reconstructing and organizing 3D scenes into open-vocabulary or hierarchical representations for geometric and semantic scene mapping.

(2) Scene Representation and LLM Integration. 3DGraphLLM (ICCV 2025) investigates encoding object-level semantic and relational structures into learnable representations for LLMs. Scene Graph Thinking (ICML 2026) further employs graph-aligned SFT and reinforcement fine-tuning on scene graphs derived from image-text data to teach MLLMs structured visual reasoning.

(3) Inference-time Spatial Evidence Acquisition. MSSR (ICLR 2026) adopts a dual-agent framework to iteratively acquire and prune spatial information into a minimal sufficient set for each query.

In contrast, GeoMem3D presents a unified framework that integrates 3D reconstruction, scene perception and understanding. Rather than treating 3D geometry merely as an input representation or an external source of tool-based evidence, or implicitly encoding 3D information into the model itself, we combine explicit 3D world memory with the original 2D visual context. Through a lightweight post-training strategy combining supervised fine-tuning and reinforcement learning (SFT + GRPO), we train MLLMs to adaptively select, integrate, and reason over these complementary information sources, enabling more effective spatial reasoning. Our approach aims not only to improve answer accuracy but also to enhance the transparency and interpretability of the reasoning process by grounding predictions in explicit spatial evidence, as illustrated by the results in paper's Figs. 3 and 4.

Q2. Clarification of Training Data Provenance and Evaluation Separation

We thank the reviewer for raising this concern. We clarify that the over 5K SFT trajectories were generated by Qwen3-VL-32B using the VSI-Bench training data, explicitly excluding all scenes and videos in VSI-Bench-Tiny. The training data sources, scene-level partitioning, and evaluation split follow the official VSI-Bench protocol, ensuring no scene-level overlap between training and evaluation data. Regarding ground-truth answers, they are provided to the teacher as auxiliary guidance to improve trajectory generation and facilitate quality verification, rather than to derive or constrain the intermediate reasoning steps.

Q3. Clarification of Answer Accuracy and Reasoning Interpretability Metrics

We would like to clarify that the results in Table 1/2 and Figure 4 correspond to two different evaluations. Tables 1 and 2 report answer accuracy over all benchmark questions, whereas Figure 4 evaluates reasoning interpretability. Specifically, for the interpretability evaluation, we retain only correctly answered instances and employ GPT-4o as an anonymized evaluator to assess the generated reasoning along three dimensions: logical coherence, consistency between reasoning and final answers, and relevance and sufficiency of referenced spatial evidence. Therefore, the scores above 80% in Figure 4 represent reasoning interpretability scores conditioned on correct predictions, rather than benchmark answer accuracy.

Q4: Ablations and Robustness to Viewpoint Variations

(1) Ablation Study. We have reported the ablation study in the revised manuscript by introducing memory-free SFT and GRPO baselines to distinguish the benefits of structured memory from general post-training. Detailed ablation results across all spatial reasoning tasks are provided in Table 7 of the appendix.

(2) Robustness to Viewpoint Variations. We further evaluate spatial reasoning under two different viewpoint-based frame samplings from the same videos, while keeping the questions and ground-truth answers unchanged.

Table 2: Task-specific Spatial Reasoning Performance under Viewpoint Variations on VSI-Bench-Tiny.

Task CategoryMethodOriginal Acc. ↑View A Acc. ↑View B Acc. ↑
Relative DirectionQwen3-VL-8B47.845.646.1
GeoMem3D-Holistic57.356.757.5
GeoMem3D-Agentic53.154.152.7
Spatial Relations (Rel. Dist.)Qwen3-VL-8B50.050.047.9
GeoMem3D-Holistic58.356.058.0
GeoMem3D-Agentic47.948.346.5

GeoMem3D-Holistic consistently outperforms the video-only baseline in both tasks across different viewpoints, suggesting that structured 3D world memory improves the robustness of spatial reasoning to viewpoint variations. We have also incorporated these experiments into the revised manuscript, as presented in Table 6.

Reviewer_rmm3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GeoMem3D organizes multi-view 3D reconstructions into structured memory containing objects, geometric attributes, and spatial relations. The model accesses this memory either holistically or through selective agentic queries, with SFT and GRPO used to learn memory utilization. The method reports improved overall performance on VSI-Bench-Tiny and MindCube-Tiny.

Critical strengths

S1. The motivation is reasonable and the framework is clearly structured. Organizing 3D geometry into directly accessible spatial evidence, rather than primarily providing additional rendered 2D observations, is a worthwhile direction. Decoupling reconstruction, memory, and reasoning, while comparing complete and selective access to the same memory, provides a clear framework for studying how spatial information should be organized and used.

S2. The experiments show positive results. The holistic model improves the VSI-Bench-Tiny aggregate score from 54.9 to 63.5 and increases MindCube-Tiny accuracy from 39.8% to 44.4% without additional adaptation. In Table 3, adding memory alone raises the aggregate score to 57.4, with further gains from SFT and RL. These results provide initial evidence for the usefulness of external spatial information and targeted training.

Critical weaknesses

W1. The distinction from prior work is not sufficiently established. The framework uses an existing reconstruction front-end, object–relation representations, and established SFT/GRPO training methods. Using existing components does not preclude a meaningful contribution, but the manuscript does not sufficiently articulate what is new relative to prior 3D scene-graph representations and their use in language-model reasoning. Please identify the closest approaches, clarify the specific contribution in memory construction, access, or training, and provide targeted comparisons demonstrating its necessity, beyond the general transition from 2D observations to structured 3D representations.

W2. The experiments do not adequately isolate the contribution of structured memory. Table 3 applies additional SFT and RL only to memory-equipped models, without a memory-free backbone trained on the same spatial examples under a comparable optimization budget. This prevents separating memory-specific learning from general post-training benefits. A further control that preserves the geometric information while changing its representation is needed to distinguish additional spatial evidence from the benefits of structured organization. The “same training configuration” claimed for Think3D in Table 2 should also specify which data and reward settings are matched.

W3. VGGT failure cases and downstream error propagation are not analyzed. The method relies on VGGT to construct geometric evidence, but provides no dedicated reconstruction failure cases or analysis of downstream sensitivity to reconstruction quality. Although reconstruction confidence is stored, its role in evidence selection or conflict resolution is unspecified. Since the original visual input remains available, reconstruction errors do not necessarily imply incorrect answers; however, the model’s ability to correct erroneous memory is untested. Please provide failure examples, report downstream performance stratified by reconstruction quality, and explain how conflicting visual observations and memory evidence are resolved.

Questions for authors

Please see strengths and weaknesses.

3 comments

We sincerely thank the reviewer for the positive evaluation and recognition of our framework design.

Q1: GeoMem3D's Core Contributions

Existing studies can be broadly categorized into three directions:

(1) 3D Scene Modeling and Reconstruction. ConceptGraphs (ICRA 2024), HOV-SG (RSS 2024), and LEXI-SG (2026) primarily focus on reconstructing and organizing 3D scenes into open-vocabulary or hierarchical representations for geometric and semantic scene mapping.

(2) Scene Representation and LLM Integration. 3DGraphLLM (ICCV 2025) investigates encoding object-level semantic and relational structures into learnable representations for LLMs.

(3) Inference-time Spatial Evidence Acquisition. MSSR (ICLR 2026) adopts a dual-agent framework to iteratively acquire and prune spatial information into a minimal sufficient set for each query.

In contrast, GeoMem3D presents a unified framework that integrates 3D reconstruction, scene perception and understanding. Rather than treating 3D geometry merely as an input representation or an external source of tool-based evidence, or implicitly encoding 3D information into the model itself, we combine explicit 3D world memory with the original 2D visual context. Through a lightweight post-training strategy combining supervised fine-tuning and reinforcement learning (SFT + GRPO), we train MLLMs to adaptively select, integrate, and reason over these complementary information sources, enabling more effective spatial reasoning. Our approach aims not only to improve answer accuracy but also to enhance the transparency and interpretability of the reasoning process by grounding predictions in explicit spatial evidence, as illustrated by the results in paper's Figs. 3 and 4.

Q2. Validating the Contributions of Structured Memory and Post-training

We extend Table 1 with additional memory-free training baselines and an experiment directly providing unstructured 3D information.

Table 1: Extended Ablation Study of GeoMem3D.

SettingVSI-Bench-Tiny
Qwen3-VL-8B54.9
+ Full 3D Information (Unstructured)×*
+ SFT (Video-only)55.0
+ SFT + GRPO (Video-only)56.6
+ Holistic Memory57.4
+ Holistic Memory (SFT)60.5
+ Holistic Memory (SFT + GRPO)63.5
+ Agentic Memory56.9
+ Agentic Memory (SFT)58.7
+ Agentic Memory (SFT + GRPO)59.6

Directly providing the complete unstructured 3D information exceeds the model's context-length limit, preventing successful evaluation.

The results reveal two key observations. **First, general post-training alone brings limited improvements without structured memory.**This demonstrates that the performance gains cannot be attributed solely to general post-training. Second, directly providing the complete 3D information is impractical due to context-length limitations. This motivates our compact, object-centric structured world memory, which organizes geometric attributes and spatial relations into an accessible representation for memory-grounded reasoning. We have also updated the ablation study in the main manuscript and included detailed task-specific ablation results in Table 7 of the appendix.

Regarding the Think3D comparison in Table 2, we clarify that both Think3D and GeoMem3D are evaluated under matched experimental configurations, including the same Qwen3-VL-8B backbone, training data, VGGT reconstruction model, and training epochs and optimization settings. This ensures that the comparison is conducted under consistent backbone, reconstruction, and training conditions.

Q3: Robustness to Reconstruction Errors and Different 3D Frontends

We conduct additional experiments on object counting to examine the flexibility of our reconstruction frontend.

First, we evaluate an alternative reconstruction frontend. Replacing VGGT with Pi3 while keeping the downstream memory construction and reasoning pipeline unchanged yields comparable performance, demonstrating the flexibility of our modular design.

Second, we investigate robustness to corrupted 3D memory. We introduce controlled errors by removing a subset of object entries relevant to the counting task from the structured 3D memory, while keeping the original video unchanged.

Reconstruction3D Memory ConditionModelTrainingObject Count Accuracy
VGGTOriginalGeoMem3D-HolisticSFT + GRPO78.2
Pi3OriginalGeoMem3D-HolisticSFT + GRPO78.4
VGGTOriginalQwen3-VL-8BZero-shot71.9
VGGTCorruptedGeoMem3D-HolisticSFT + GRPO73.7
VGGTCorruptedQwen3-VL-8BZero-shot65.3

These results demonstrate the adaptability of GeoMem3D to different reconstruction frontends and its robustness to imperfect 3D memory. We have also incorporated these additional experiments and the corresponding analysis into the revised manuscript, as presented in Table 5.

Thanks for the authors' response. The additional experimental results improve the technical quality of the paper. However, the novelty is still somewhat limited. It would be better to further justify which specific problems the proposed method can address that existing approaches struggle to solve. I would like to maintain my original ratings.

Thank you for the further clarification. Existing approaches have advanced 3D spatial reasoning along two complementary directions. Some emphasize explicit 3D geometric evidence, such as reconstructed geometry and structured spatial representations, while others primarily rely on MLLMs' 2D visual understanding to infer 3D spatial relationships, sometimes augmented with additional visual observations or inference-time tools. However, how to effectively combine these complementary sources of spatial information according to the requirements of different reasoning tasks remains insufficiently explored.

GeoMem3D addresses this problem by integrating a persistent, structured 3D world memory with the original 2D visual context. More importantly, through a lightweight post-training strategy combining SFT and GRPO, we teach MLLMs to adaptively utilize and integrate explicit 3D geometric evidence and 2D visual observations, rather than relying on either source alone. Our framework supports both holistic reasoning over the complete 3D memory and agentic reasoning through adaptive retrieval of task-relevant spatial evidence.

Furthermore, GeoMem3D aims not only to improve answer accuracy but also to enhance interpretability and produce more coherent and well-grounded reasoning, as demonstrated by our experimental results. We hope this clarification better highlights the specific problem addressed by GeoMem3D and its methodological distinction from existing approaches.

Reviewer_tprFOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

GeoMem3D uses VGGT to build an object-level 3D scene graph from RGB video, then has Qwen3-VL-8B (trained with SFT and then GRPO) reason over it. The model either reads the whole graph as text (Holistic) or queries it with tool calls (Agentic). It beats Think3D on both VSI-Bench and MindCube.

Critical strengths
  • clean experiment and quantification of the contribution from either the agentic pipeline or the model finetuning
  • two path for the scene query: a comprehensive textual global view injection or a ondemand RAG like query.
Critical weaknesses

there are too many paper trying to build an agentic pipeline to answer the spatial reasoning VQA. This paper brings very limited insights for the community. papers here including: ConceptGraphs(2024, ICRA), HOV-SG(2024, RSS), 3DGraphLLM(2025, ICCV), LEXI-SG(2026, arXiv preprint), RieMind(2026, arXiv preprint), GraFT(2026, arXiv preprint)

Questions for authors
  • did you ever try to use sota (either opensource or commercial) model or coding agent to run zero shot agentic VQA over the questions. larger the model typically they will have more skillful usage of the tools via its general knowledge, maybe simply by prompting.
  • how about dynamic 4D scene understand? they will be more challenge than the static setup.
1 comment

We sincerely thank the reviewer for the positive evaluation and recognition of our systematic experiments.

Q1: GeoMem3D's Core Contributions and Its Relation to Prior Work

We thank the reviewer for highlighting these related works. These studies can be broadly grouped into three directions:

(1) 3D Scene Modeling and Reconstruction. ConceptGraphs (ICRA 2024), HOV-SG (RSS 2024), and LEXI-SG (2026) primarily focus on reconstructing and organizing 3D scenes into open-vocabulary or hierarchical representations, improving geometric and semantic scene mapping.

(2) Scene Representation and LLM Integration. 3DGraphLLM (ICCV 2025) investigates how to encode object-level semantic and relational structures into learnable representations and integrate them into LLMs, enabling the models to process 3D vision-language tasks through model-level representation learning.

(3) Inference-time Spatial Evidence Acquisition. RieMind (2026) and GraFT (2026) explore how MLLMs can acquire and utilize task-relevant spatial evidence through inference-time interactions with geometric tools, BEV representations, and visual frame retrieval, enhancing spatial reasoning without relying solely on implicit visual understanding.

In contrast, GeoMem3D presents a unified framework that integrates 3D reconstruction, scene perception and understanding. Rather than treating 3D geometry merely as an input representation or an external source of tool-based evidence, or implicitly encoding 3D information into the model itself, we combine explicit 3D world memory with the original 2D visual context. Through a lightweight post-training strategy combining supervised fine-tuning and reinforcement learning (SFT + GRPO), we train MLLMs to adaptively select, integrate, and reason over these complementary information sources, enabling more effective spatial reasoning. Our approach aims not only to improve answer accuracy but also to enhance the transparency and interpretability of the reasoning process by grounding predictions in explicit spatial evidence, as illustrated by the results in paper's Figs. 3 and 4.

Q2: Zero-shot General-purpose MLLM Baselines.

To address this question, we evaluate five representative open- and closed-source MLLMs under three settings: (i) video-only input, (ii) video with full 3D memory, and (iii) video with simulated tool-use prompting. Note that tool use for general-purpose MLLMs is simulated through prompting rather than actual external tool execution.

Table 1: Comparison of General-purpose MLLMs under Different 3D Memory Access Settings on VSI-Bench-Tiny.

MethodModel TypeTrainingVideo-onlyVideo + Full MemoryVideo + Tool-use Prompting
Qwen3-VL-8BOpen-sourceZero-shot54.957.456.9
Qwen3-VL-32BOpen-sourceZero-shot57.461.159.8
InternVL3-78BOpen-sourceZero-shot47.548.649.2
GPT-5.5Closed-sourceZero-shot59.661.860.2
Gemini-3 FlashClosed-sourceZero-shot53.756.354.9
GeoMem3D-Holistic (Ours)Open-sourceSFT + GRPO—63.5—
GeoMem3D-Agentic (Ours)Open-sourceSFT + GRPO——59.6

The results reveal two key observations. First, lightweight post-training enables an 8B model to achieve highly competitive spatial reasoning performance. Second, although providing explicit 3D memory consistently improves general-purpose MLLMs, simply describing how to use spatial query tools through prompting does not consistently yield further improvements over direct memory input. This suggests that access to structured 3D information alone is insufficient; models also need to learn how to effectively select, integrate, and reason over such information. GeoMem3D addresses this through SFT and GRPO-based post-training. We have also incorporated these additional comparisons and the corresponding analysis into the revised manuscript, as presented in Table 4.

Q3: Extension to Dynamic 4D Scene Understanding

We agree that dynamic 4D scene understanding is an important and challenging direction. Our current work primarily focuses on static 3D environments. We believe that reliable spatial understanding of static 3D environments remains a challenging and not yet fully solved problem, warranting focused investigation before extending to more complex dynamic scenarios. However, GeoMem3D adopts a modular design that decouples 3D reconstruction from downstream structured memory construction and reasoning. This flexibility allows more advanced dynamic 4D reconstruction methods to be incorporated as they mature. Furthermore, the structured world memory can be extended with temporal object states, motion trajectories, and time-varying spatial relations to support temporal reasoning. We consider this a promising direction for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.