acceptodds
Withdrawn from ICLR 2027

SpaceMind++: Toward Allocentric Cognitive Maps for Spatially Grounded Video MLLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent multimodal large language models (MLLMs) have made remarkable progress in visual understanding and language-based reasoning, yet they lack a persistent world-centered representation for spatially consistent reasoning in 3D environments. Inspired by the mammalian dual-stream system, where semantic and spatial cues are processed separately and integrated into an allocentric cognitive map, we propose SpaceMind++, a video MLLM architecture that explicitly builds a voxelized cognitive map from RGB videos. This map reorganizes fragmented egocentric observations into a shared 3D metric representation, enabling the model to preserve object permanence and spatial topology across changing viewpoints. To make this allocentric representation usable by a pretrained video MLLM without disrupting its native visual-token interface, we introduce Coordinate-Guided Deep Iterative Fusion, a new mechanism that relays map-level spatial knowledge back into the original 2D visual features. This fusion is explicitly guided by coordinate embeddings and 3D Rotary Positional Encoding, which ground semantic interactions in metric 3D space, resembling the entorhinal binding of sensory features to metric space. Extensive experiments show that SpaceMind++ achieves state-of-the-art performance on VSI-Bench, while also demonstrating cross-benchmark transfer on SPBench, SITE-Bench, and SPAR-Bench across changes in scene sources and task formulations.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Multimodal Large Language Models, Spatial Intelligence, 3D Understanding, Cognitive Maps

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vcmzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SpaceMind++, a video MLLM that explicitly organizes multi-view visual and geometric features into a shared voxelized 3D cognitive map for spatial reasoning. It introduces Coordinate-Guided Deep Iterative Fusion (CDIF), which performs reasoning over the 3D map and transfers the resulting spatial information back to the original visual tokens using coordinate embeddings, 3D RoPE, and cross-attention. The model is trained on a 900K spatial instruction corpus and evaluated on VSI-Bench, SPBench, SITE-Bench, and SPAR-Bench, showing improved performance over prior spatial MLLMs.

Critical strengths
  1. The paper introduces a clear inductive bias for multi-view spatial reasoning, which explicitly organizes observations from different views according to their predicted 3D locations and performs reasoning in a shared 3D space. This is conceptually appealing to me, for example, when the camera moves around a table, the same cup may appear in frame 1 from the left side and in frame 20 from the right side. In an implicit representation, these correspond to two distant visual tokens, and the model must learn by itself that they refer to the same or nearby physical location. In an explicit 3D map, the VGGT-predicted coordinates directly project them into the same or neighboring voxels. Cross-view association is imposed by the representation itself.

  2. Another strength is that the representation is world-centered. As the camera moves, the 3D location of an object in the map should remain relatively stable. This makes questions such as “which side of the table is the cup on?” more naturally represented in this world-centered coordinate system.

Critical weaknesses
  1. The main weakness to me is the proposed method itself. It introduces a strong structural prior, but at the cost of reduced flexibility and strong dependence on upstream (VGGT) geometry quality. Errors from VGGT would be inherited into the 3D map. Once 3D coordinates are inaccurate, incorrect cross-view correspondences may be inherited. Voxelization is also inherently lossy, as geometrically different surfaces or small structures may be merged into the same voxel and averaged then.

Relevantly, the paper lacks a control experiment of testing a model without voxelized map construction but with the same 3D coordinate encoding, 3D RoPE, and a comparable attention capacity applied to the original visual tokens. It is hard to say whether the gain comes from the explicit 3D map itself or mainly from stronger coordinate-guided fusion.

From the computational cost perspective, the full method improves over SpaceMind by about 1-1.5 points under matched-data settings, while adding voxelization, coordinate MLPs, 3D RoPE, map self-attention, cross-attention, and gated fusion. The performance-cost trade-off remain a concern to me.

  1. The novelty claim seems somewhat overstated to me. The paper frames its main conceptual contribution as moving from treating geometry as an auxiliary feature stream to explicitly constructing a cognitive map and reasoning within it. However, existing work such as Cog3DMap already builds compact 3D cognitive maps. Map2Thought uses metric cognitive maps, 3DLLM-Mem maintains 3D spatiotemporal memory. The authors position the main distinction of this paper from them as, using the map as an intermediate neural workspace and then writing the reasoning result back to the original visual-token interface. As a result, the novel part seems narrower to me, which is, the integration of “3D map -> map-level attention -> cross-attention back to 2D visual tokens.” The novelty is less on the idea of explicit cognitive maps for MLLMs. Considering that LLaVA-3D already performs voxel-based aggregation and that 2D->3D->2D feature propagation has long been explored in 3D vision, the actual conceptual novelty appears weaker than suggested by the paper’s framing.
Questions for authors

Please see weaknesses.

Reviewer_ZMYuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents SpaceMind++, a video MLLM that organizes semantic and geometric features into a voxelized allocentric map. Built upon InternVL3-8B and a frozen VGGT encoder, the method aggregates observations using predicted 3D coordinates and introduces Coordinate-Guided Deep Iterative Fusion (CDIF), alternating map self-attention with map-to-visual cross-attention. The resulting features retain the original visual-token interface. The model is trained on a 900K mixture of existing spatial QA datasets and evaluated on VSI-Bench, SPBench, the video subset of SITE-Bench, and the high-level subset of SPAR-Bench.

Critical strengths
  1. Competitive spatial reasoning performance. The reported VSI-Bench score of 73.2 is strong, and the improvement on the multi-view subset of SPBench is consistent with the intended benefit of aggregating cross-view observations.
  2. A reasonable architectural design with useful controlled ablations. Organizing observations by predicted 3D location is a sensible inductive bias. The matched-data comparisons between SpaceMind and SpaceMind++ are particularly useful for separating architectural improvements from additional training data.
Critical weaknesses
  1. The generalization evidence is insufficient to resolve concerns about benchmark-specific adaptation. SPBench provides a relatively narrow indoor-scene evaluation, while SITE-Bench and SPAR-Bench are evaluated only on selected subsets. Several central evaluations remain concentrated on related indoor reconstruction domains. Moreover, the same-data InternVL3-8B baseline already reaches 70.8, compared with 73.2 for the full system. This does not establish overfitting, but it makes broader testing and matched-data comparisons outside VSI-Bench important. The current evidence does not adequately distinguish generally transferable spatial capabilities from adaptation to familiar spatial QA distributions.
  2. The incremental methodological contribution and its practical advantage are not sufficiently established. The dual-encoder design and preservation of the visual-token interface are inherited from SpaceMind. Earlier methods such as GS-Reasoner already combine semantic, geometric, and 3D positional information, while Cog3DMap constructs explicit 3D cognitive maps. These methods are not identical to SpaceMind++, but the paper needs a clearer account of its distinctive contribution. The 900K corpus is also described as a mixture of existing datasets rather than newly generated supervision. Under matched 900K training data, the gain over SpaceMind is only 1.1 points, without corresponding evidence about variability or additional inference cost. The claimed state-of-the-art position also needs updating: Cambrian-P reports 73.7 on VSI-Bench, although its 128-frame setting means that a fair comparison requires further information.

[1] Cambrian-P: Pose-Grounded Video Understanding [2] Reasoning in Space via Grounding in the World [3] Cog3DMap: Multi-View Vision-Language Reasoning with 3D Cognitive Maps

Questions for authors
  1. Evaluation settings and score reporting: What are the training and inference frame counts, frame-sampling strategies, input resolutions, and decoding settings?
  2. Generalization: Could you provide representative additional evaluations, such as ViewSpatial or MMSI-Bench, or more complete SITE-Bench coverage? The EASI-leaderboard offers relevant benchmark options. Please include the performance of matched-data InternVL3-8B and SpaceMind baselines in such benchmarks.
  3. Novelty and comparisons: Please clarify the differences from GS-Reasoner and Cog3DMap and update the comparison with recent RGB spatial reasoning methods, including Cambrian-P, under comparable input budgets where feasible. A coordinate-aware fusion baseline without voxel aggregation would help isolate the value of the map itself.
  4. Evidence for map use: Could you provide visualizations and diagnostic interventions showing which map regions contribute to answers about particular objects? Since map fusion precedes language decoding, it would be useful to distinguish meaningful spatial integration from generic auxiliary feature enrichment. For example, masking relevant map regions or perturbing coordinates could complement attention visualizations.
Reviewer_X2CyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds a voxelized "cognitive map" inside a video MLLM. Patch features from InternViT and VGGT are scattered into a 3D voxel grid using VGGT point maps, filtered for outliers, and pooled into map tokens. A fusion module (CDIF) alternates self-attention over map tokens and gated cross-attention from visual tokens to the map, both with 3D RoPE, so the LLM still receives ordinary visual tokens. Built on InternVL3-8B and trained on a 900K spatial QA mix, it reports 73.2 on VSI-Bench plus results on SPBench, SITE-Bench and SPAR-Bench.

Critical strengths
  • Keeping the pretrained visual-token interface while adding a 3D map is a practical design, and the method is described clearly.
  • The data-matched comparison with SpaceMind (626K vs 900K) and the "random map" control are the right ablations to run.
  • Solid VSI-Bench numbers, with the largest gains on appearance order and relative distance.
Critical weaknesses
  • Under matched data the gain over SpaceMind is small: +1.1 on VSI-Bench at 900K (72.1 → 73.2). The plain InternVL3-8B baseline trained on the same 900K already gets 70.8, so much of the headline improvement over prior work in Table 1 comes from data. No seeds or variance are given, and a 1-point difference on VSI-Bench may be noise.
  • The cross-benchmark evaluation is selective. SPAR-Bench reports only the high-level split, and even there distance tasks are far below baselines (Dist OO 39.8 vs 62.4 for SpaceR and 65.0 for GPT-4o; OO-MV 33.0 vs 61.6). SITE-Bench uses only the video subset, and SPBench-SI NQ drops from 66.3 (SpaceMind) to 56.7. None of this is discussed.
  • The novelty over recent map-based MLLMs (Cog3DMap, Map2Thought, LLaVA-3D) is argued in words only, with no empirical comparison. The brain-inspired framing is mostly narrative; the mechanism is voxel pooling plus attention with 3D RoPE.
  • Compute cost (VGGT + L CDIF layers + up to 4000 map tokens) is not reported. The fixed grid (D = 100) discards out-of-range tokens in a relative-scale space, which may hurt large scenes, and the fraction discarded is not reported.
Questions for authors
  1. How many CDIF layers are used, and what are the added latency and memory compared with InternVL3-8B?
  2. What fraction of tokens falls outside the grid on VSI-Bench and on larger scenes?
  3. Can you report the full SPAR-Bench results and explain the drop on distance tasks?
  4. VSI-590K and VLM-3R-data come from the same scene sources as VSI-Bench (ScanNet/ARKitScenes). Did you check for scene overlap?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.