acceptodds
Withdrawn from ICLR 2027

Compositional Spatial Reasoning for Multimodal Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Spatial reasoning is a fundamental capability of multimodal large language models (MLLMs), requiring models to infer complex semantic, geometric, and temporal relationships from visual observations. Recent approaches enhance spatial reasoning by enriching visual representations with additional spatial priors from 3D reconstruction or depth estimation models. However, we find that this uniform representation alone does not guarantee better reasoning, as different queries require different compositions of spatial evidence. In this work, we identify two key challenges for effective spatial reasoning: comprehensive representation and selective evidence composition. We introduce CoSpace, a compositional spatial reasoning framework that follows the principle of representing comprehensively and reasoning selectively. CoSpace organizes heterogeneous visual information into structured semantic, geometric, and temporal representations, and performs query-adaptive evidence composition through hierarchical selection, allocation, and retrieval to construct query-relevant reasoning states. Extensive experiments across diverse spatial reasoning benchmarks demonstrate that CoSpace improves spatial reasoning performance, effectively exploits heterogeneous visual evidence, and enables query-adaptive composition of interpretable spatial representations. The code will be made publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
spatial intelligence, multimodal large language models, compositional representations

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_yXpCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoSpace for combining spatial representations in MLLMs. The shared VGGT features are organized into semantically dense tokens, geometrically multi-scale summaries, and a temporal summary. CoSpace adopts a question-conditional router to select the feature library, an allocator to weight the features according to the question and visual summaries, and independent cross-attention branches to refine a shared geometry-enhanced visual representation. Experiments on MMSI-Bench and ViewSpatial show improvements, but SITE and CV-Bench show a decline compared to the original backbone.

Critical strengths
  1. The well-controlled comparison supports the architectural contributions. Standard settings control CoSpace and related baselines across multiple factors such as training data, backbone, frame count, etc. This is important because spatial instruction tuning alone can yield significant improvements, making comparisons with an untuned backbone insufficient.

  2. Component analysis examines several different aspects of the mechanism. Retraining ablation is supplemented with fixed checkpoint routing and feature library interventions. The paper also clearly distinguishes between residual suppression and weight renormalization and mentions that these interventions may introduce composition-distribution mismatch.

  3. Effective inference-time scaling. A checkpoint trained with fixed frames benefits from additional inference frames without retraining, which provides an interesting point of the framework.

Critical weaknesses
  1. Insufficient isolation of bank organization from multiscale branching and supervised routing. All three banks largely transform the same VGGT features in a heuristic way, and the routing patterns are led by manually specified masks. Furthermore, Equation (3) uses a shared visual token rather than a question representation as the cross-attention query, thus question adaptation relies primarily on feature library selection and weighting. Current experiments more clearly demonstrate the effectiveness of the overall configuration, rather than specific components.

  2. The feature bank specialization validation is insufficient. While the semantic, geometric, and temporal feature libraries are all constructed using shared VGGT feature transforms, their expected functional divisions have not been directly validated. The routing pattern is guided by a predefined task mask, while removing the ablation only proves the branch's usefulness, not the proposed semantic-geometric-temporal interpretation. Even without strict decoupling, comparison with general branches matching the parameter count and token budget is necessary to support this core architectural contribution.

  3. Performance improvements are modest. Improvements over GeoThinker are approximately 0.9 points in the matched regime and 0.8 points in the scaled 32-frame comparison. In Table 3 (cross-benchmark performance), the proposed method also underperforms GeoThinker and the base model on several benchmarks.

Questions for authors
  1. Can you provide the comparisons between CoSpace and direct multiscale banks without semantic-geometric-temporal organization?

  2. Beyond routing patterns encouraged by predefined masks, what evidence shows that each bank correctly captures its intended semantic, geometric, or temporal information?

Reviewer_WrqtOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to use different types of spatial information in visual representations according to the demands of each question. The authors propose CoSpace, following the principle of “represent comprehensively, reason selectively.” It organizes VGGT features into three banks: a semantic bank retaining dense tokens, a geometric bank incorporating local pooling and global summaries, and a temporal bank containing frame summaries, adjacent-frame differences, and camera features.

The model obtains a question representation through an additional text-only forward pass, predicts bank eligibility, and assigns contribution weights using visual summaries. Each bank produces visual residuals through an independent cross-attention branch. These residuals are weighted and injected into the shared visual representation for answer generation by Qwen. Routing is supervised using bank labels derived from manually specified task rules, while the remaining modules are optimized through the answer-generation loss.

The authors evaluate the method using two Qwen backbones across multiple spatial understanding benchmarks and different input frame counts. The results support the benefits of explicit bank organization and adaptive mixing, although claims concerning fine-grained question-conditioned retrieval and functional evidence decomposition require further qualification.

Critical strengths

The paper presents extensive experiments demonstrating the effectiveness of the proposed method.

Critical weaknesses
  1. There is a mismatch between the method’s motivation and the problem it claims to address. A shared visual representation does not imply that all evidence is used uniformly. Standard MLLM decoders already select information through question-conditioned attention. What, then, justifies the claim that existing methods lack selective reasoning? Furthermore, the bank retrieval branches do not take the textual question as input. For a fixed video, does this not mean that each bank’s retrieved residual remains unchanged across questions, with only bank eligibility and scalar contribution weights varying? Consequently, two questions asking about distances between different object pairs would still produce identical within-bank attention maps. This is difficult to reconcile with the claim of retrieving the fine-grained evidence most useful for the current question.

  2. The experiments do not fully substantiate how the proposed design addresses the identified problem, nor do they clearly establish the independent value of the newly introduced components.

Questions for authors
  1. Standard MLLM decoders already use question-conditioned attention, while CoSpace’s bank retrieval in Eq. (3) appears independent of the textual question. What specific limitation does CoSpace address, and how does bank-level gating and scalar weighting resolve it sufficiently to justify the claim of question-specific fine-grained evidence retrieval?

  2. What is CoSpace’s principal methodological advance over existing query-adaptive fusion and bank-retrieval approaches, such as ViPS and GeoWeaver? Please clarify which aspects of bank construction, routing, or allocation constitute distinct contributions and explain their independent value beyond combining established mechanisms.

Reviewer_h35HOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CoSpace, a framework for spatial reasoning in multimodal large language models. It organizes VGGT-derived features into semantic, geometric, and temporal banks, then uses question-conditioned selection and weighting to combine bank-specific features with a shared visual representation. Bank selection is trained using manually defined task-relevance masks, while feature retrieval and weighting are optimized through answer generation. Experiments show improvements on several spatial benchmarks, supported by component ablations and inference-time interventions.

Critical strengths
  1. Clear motivation. The paper addresses a meaningful limitation: richer geometric representations do not consistently improve reasoning, motivating question-dependent evidence composition.
  2. Coherent architecture. Separating representation construction, bank selection, contribution allocation, and feature retrieval provides a clear and modular design.
  3. Useful controlled comparisons. The vanilla setting matches training data, backbone, and evaluation protocols across methods. Retrained ablations provide evidence that the proposed components contribute to performance.
  4. Broad evaluation. Results cover multiple benchmarks, including VSI-Debiased, and demonstrate benefits from additional inference frames without retraining.
Critical weaknesses
  1. Heavy reliance on predefined routing priors. Training uses manually specified bank masks, and inference-time selection depends only on the question. The routing patterns may therefore reflect task classification more than scene-specific evidence selection. Comparisons with rule-based and fully learned routing would strengthen the claim.
  2. Retrieval is not directly question-conditioned. Cross-attention uses visual features as queries; the question mainly controls bank selection and scalar weights. The mechanism therefore supports question-dependent feature composition more clearly than question-specific evidence retrieval.
  3. Bank specialization is insufficiently established. All three banks largely originate from the same VGGT features. Capacity-matched comparisons with generic multibranch or multiscale alternatives are needed to isolate the value of the semantic–geometric–temporal organization.
  4. Modest gains and mixed generalization. Improvements over GeoThinker are relatively small, without reported uncertainty estimates. Performance also declines relative to the original backbone on CV-Bench and SITE, limiting claims of consistently strong generalization.
  5. Incomplete attribution and efficiency analysis. Routing interventions introduce a mismatch with training conditions, while the scaled setting combines additional data and geometric integration. Missing latency, memory, and parameter comparisons make the practical cost and source of the gains difficult to assess.
Questions for authors

See critical weaknesses.

Reviewer_xgA6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoSpace for spatial reasoning with MLLMs. In addition to fusing VGGT features with 2D visual features to form a shared scene context, CoSpace also projects and organizes VGGT features into semantic, geometric, and temporal representation banks. The query-conditioned router, allocator, and cross-attention branches compose evidence from these banks. With 32 input frames, CoSpace outperforms the baselines in the matched Vanilla comparison on VSI-Bench. In the Scaled regime, its Qwen3-VL-8B model achieves the best result among the compared 32-frame models. It also achieves the best results among the compared models on MMSI-Bench and ViewSpatial.

Critical strengths
  1. Clear motivation and a modular framework. The motivation is clear: different spatial queries may require different evidence, and adding geometric features alone may not be enough. CoSpace has a modular framework that clearly separates representation organization from query-adaptive selection, allocation, and retrieval.
  2. Useful controlled experiments. The controlled component ablations provide useful evidence that all three representation banks, the router, and the allocator contribute to performance.
Critical weaknesses
  1. Limited gains from the added mechanisms. Under the same data and protocol in the Vanilla regime, CoSpace improves over the pure 2D baseline from 53.8 to 54.9 (roughly one point). This is a modest gain given the added modules. The scaled regime lacks baselines trained with the same data and protocol for both a pure 2D model and a model that only fuses 2D and VGGT features, without the proposed banks. It is therefore unclear whether CoSpace gains more from the expanded training recipe than either simpler baseline.
  2. Insufficient validation of the Geometry Bank's design and benefits. The paper does not ablate the pooled local tokens, per-frame summaries, and global summary separately. It is therefore unclear why these components are combined or which capabilities each supports. The Geometry Bank ablations and interventions show uneven effects across metric tasks (Tables 4 and 11), so the evidence for consistent gains in metric reasoning is limited.
Questions for authors
  • Matched baselines. Please compare CoSpace with a pure 2D model and direct 2D–VGGT fusion in the Scaled regime using the same backbone. The fusion baseline should retain shared geometry paths while removing the three banks, router, allocator, and their cross-attention refinement branches. How do CoSpace’s gains change from Vanilla to Scaled?

  • Geometry Bank design. What motivates combining pooled local tokens, per-frame summaries, and a global summary? Separate ablations are needed to establish their task-specific contributions and explain the mixed effects on metric tasks.

  • Gain reliability and cost. The Vanilla gain is approximately one point on average. Please report uncertainty through repeated runs or appropriate resampling, together with additional parameters and inference latency under matched frame counts and hardware.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.