acceptodds
Under review as a conference paper at ICLR 2027

OmniView-Space: Reinforcing Spatial Reasoning via Multi-Perspective Spatial Mapping

Abstract

Spatial intelligence remains a challenge for Multimodal Large Language Models (MLLMs), as it requires coherent spatial scene representations beyond basic object recognition. Existing methods typically build such representations through textual reasoning or 3D reconstruction. However, they often falter during multi-step reasoning, particularly when required to dynamically re-anchor evidence to the specific camera-, object-, or direction-centric reference frames demanded by complex queries. To address this, we propose OmniView-Space, a framework to maintain spatial consistency through multimodal egocentric evidence. Our approach consists of three core components: (1) Multi-Perspective Spatial Mapping (MPSM), which re-anchors reconstructed geometry into a query-aligned visual cognitive map and a textual spatial graph; (2) Tool-Guided Egocentric Reasoning, an interleaved policy trained to actively select the ego anchor required by the query and request the corresponding MPSM evidence; and (3) Cognitive-Map Distillation, which uses MPSM-generated trajectories and ego-frame rewards to train the model to reason with self-generated cognitive maps. Experiments on single- and multi-image spatial reasoning benchmarks show that OmniView-Space achieves state-of-the-art performance. Furthermore, the distilled model maintains this performance while reducing reliance on external geometry pipelines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Spatial Reasoning, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7rbTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that a bottleneck in multi-step spatial reasoning is reference-frame mismatch: existing 3D evidence or textual spatial graphs are usually expressed in a system frame, while questions require judgment in an ego frame defined by a camera, an object, or a direction. OmniView-Space uses MPSM to re-anchor a scene reconstructed by Depth Anything 3 and SAM 3 into the query-specified ego frame, returning a bird’s-eye cognitive map and a textual spatial graph. GRPO then trains a tool-use policy that selects the anchor and the evidence format. Finally, MPSM trajectories are used for SFT, and an object- and camera-level map-alignment reward distills this behavior into the model so that inference does not need the full geometry pipeline. The backbone is Qwen3-VL-4B. The tool-using variant scores 50.6 overall on MindCube-Tiny, MMSI, and SPAR; the distilled variant scores 46.5, compared with 33.1 for Qwen3-VL-4B. It also leads the listed open-source models on the allocentric subset of 3DSRBench and the perspective subset of OmniSpatial.

Critical strengths
  1. The problem is concrete, and the interface is executable. The coordinate transforms for camera, object, and direction anchors are specified, and the visual map and textual graph come from the same ego frame. Table 4 shows that prompting the base model with MPSM evidence alone raises the MindCube/MMSI average from 31.8 to 40.4; tool RL reaches 50.6, and distillation still scores 48.7. Appendix C audits image, scene, and question-template overlap between training and evaluation, and it explicitly treats MindCube-Tiny and SPBench as in-domain. This breakdown is more informative than a single overall score.
  2. The out-of-domain benchmarks are not a zero-gain setting. The exact template-overlap audit is zero for MMSI, SPAR, 3DSRBench, and OmniSpatial. MMSI rises from 27.7 to 35.5 for the tool-using model and 31.6 for the distilled model; SPAR rises from 35.6 to 44.8 and 42.3. Appendix Table 8 shows that training directly on MindCube or SpatialLadder does not improve MMSI, so the MMSI gain cannot be attributed simply to exposure to MindCube templates.
Critical weaknesses
  1. The most prominent MindCube result is in a setting the authors themselves mark as in-domain, and the SFT trajectories include answer-conditioned repair. MindCube training templates exactly match 333 of 1,050 MindCube-Tiny questions (31.7%); the overlap on SPBench-SI is 78.1%. In round two of SFT construction, rationales for incorrect samples are rewritten using the ground-truth answer and then filtered by a critic. The tool-using MindCube score of 71.5 is a headline number in the main text, but the main text does not report the zero-template-overlap subset. On SPBench, the text baseline is higher than the full method (63.0 versus 60.0), so the ego map is not a stable gain.
  2. The learned policy adds a limited amount beyond the external geometry pipeline. Prompted MPSM already contributes about 8.6 points on average, and the map-alignment reward adds only about 2.3. The distilled model is 4.1 overall points below the tool-using model. The paper therefore mainly shows that rendering a reconstruction in the query coordinate frame is useful to an MLLM. The main text does not provide a decisive error breakdown of how reconstruction errors propagate into the ego frame, or of whether the model still calls the tool when reconstruction fails.

I would like to raise my score if the authors can solve my concerns.

Questions for authors
  1. Please report accuracy on the 717 MindCube-Tiny samples with zero template overlap, compared under the same protocol with Qwen3-VL-4B, MindCube-3B, and SpatialLadder. If the advantage remains mostly on the 333 overlapping templates, I would lower the conclusion drawn from the main table.
  2. Among tool-call trajectories, what fraction have a clearly wrong MPSM ego frame or miss a key object? How much lower is answer accuracy on those samples than on samples with a correct reconstruction? This determines whether the gain comes from correct geometric re-anchoring or from simply seeing an additional map.
4 comments
AuthorsOpenReview ↗

Rebuttal by Authors [1/3]

W1(1),Q1: Does OmniView-Space generalize beyond overlapping training templates?

To clarify, on the 717 samples without exact training-template overlap, Tool-Integrated and Distilled OmniView-Space outperform the MindCube-3B, by 25.67 and 19.55 percentage points, respectively. These results show that our gains are not concentrated on overlapping templates. We provide two complementary lines of evidence below.

1. Performance with and without training-question overlap.

ModelNon-overlap (717)Overlap (333)Full set (1,050)
Qwen3-VL-4B-Instruct34.7538.3735.90
MindCube-3B49.7755.8651.70
SpatialLadder-3B41.8046.8543.40
Distilled OmniView-Space (Ours)69.3257.9665.70
Tool-Integrated OmniView-Space (Ours)75.4463.0971.52
ComparisonNon-overlapOverlapFull set
Distilled vs. MindCube+19.55+2.10+14.00
Tool-Integrated vs. MindCube+25.67+7.23+19.82
Tool-Integrated vs. Distilled+6.12+5.13+5.82

Both variants show larger gains over MindCube-3B on non-overlapping questions, indicating that their improvements are not concentrated on overlapping templates. Tool integration further improves the distilled model by 6.12 points on non-overlapping questions.

2. Additional evidence from models trained on the same public sources.

To examine whether exposure to MindCube/SpatialLadder-style data consistently improves transfer, we include the following reference results:

MethodAdditional spatial training dataMindCubeMMSI-Bench
Qwen2.5-VL-3BNone37.628.6
MindCube-3BMindCube-10K51.723.9
SpatialLadder-3BSpatialLadder-26K43.425.4

These models improve on in-domain MindCube but do not improve MMSI-Bench over their base model. By comparison, OmniView-Space improves MMSI-Bench from 27.7 to 35.5 on its Qwen3-VL-4B backbone, alongside strong results on SPAR-Bench, 3DSRBench, and OmniSpatial. This suggests that the gains are not simply due to exposure to similar training distributions, but come from query-aligned spatial representation and tool-guided reasoning.

W1(2): Do the reported gains depend on answer-conditioned repair?

To clarify, answer-conditioned repair is used only for the distilled model’s SFT data. The tool-integrated model is trained directly with GRPO, without SFT or repaired rationales. Thus, its 71.5% accuracy on MindCube-Tiny and gains on the 717 non-overlapping-template samples do not depend on this procedure. We will clarify this distinction in the paper.

W1(3): The single-image SPBench result is worse than the text baseline.

To clarify, the slightly lower result on single-image SPBench is mainly because this subset focuses more on perception-level questions, which are less aligned with the core target of OmniView-Space. Since our RL data are mainly derived from multi-image scenarios, the learned policy is more specialized for tool-based reasoning under multi-view evidence aggregation, rather than for single-image perception-only cases. Meanwhile, over 60% of the 1,009 SPBench-SI samples are metric-related questions, such as absolute distance and object size estimation, which can often be answered more directly from the original image and the MLLM’s visual perception.

Importantly, this limitation is mainly observed on simple single-image perception tasks. For more complex spatial reasoning tasks that require viewpoint transformation, cross-view aggregation, or ego-frame re-anchoring, MPSM brings much larger gains. For example, OmniView-Space improves performance by +19.2 on MindCube, and by +5.7 and +7.4 on the out-of-domain benchmarks MMSI-Bench and SPAR-Bench, respectively. We will add this question-type analysis in the revised manuscript.

AuthorsOpenReview ↗

Rebuttal by Authors [2/3]

W2(1): What does RL learning add beyond a prompted geometry pipeline?

Thank you for this insightful comment. RL helps the model (1) select question-aligned reference frames, (2) request relevant objects, (3) interpret tool-provided spatial evidence for reasoning, and (4) decide when to use tools. Across three benchmarks, Tool-Integrated RL improves average accuracy from 40.1% with prompted MPSM to 50.6% (+10.5 pp) (Table 1). The 2.3-point map-alignment reward gain reflects an auxiliary reward’s incremental benefit, not the overall contribution of RL. We examine these four aspects below.

(1) RL improves question-aligned reference selection. We used the same 192 MMSI-Bench questions to evaluate the accuracy of Qwen3-VL-4B-Instruct and the strategies trained through RL in selecting the reference frame.

ModelQuestion-aligned reference selectionsAlignment rate (%)
Qwen3-VL-4B-Instruct, before RL101/19252.60
Tool-Integrated OmniView-Space, after RL132/19268.75

RL increases the alignment rate by 16.2 percentage points, indicating that the trained policy selects reference frames that better match the question.

(2) RL improves task-relevant object requests. On 776 MindCube-Tiny questions, we inspect whether the object hints requested before segmentation cover the key categories involved in the question or its correct object-valued option. GT CogMap labels are used only for this post-hoc audit.

ModelRequests covering all key categories
Prompted Qwen3-VL-4B-Instruct484/776 (62.37%)
Tool-Integrated OmniView-Space622/776 (80.15%)

Complete category coverage increases by 17.78 percentage points, indicating more complete task-relevant object requests in the learned pipeline.

(3) RL improves spatial reasoning through more effective use of MPSM evidence. To assess whether the RL-trained model better uses supplied spatial evidence to answer questions, we randomly sample 200 MindCube-Tiny questions. Both models receive only the question, answer options, and the same query-aligned MPSM evidence. We exclude original images to focus the comparison on reasoning from the supplied MPSM evidence. The evidence is constructed using GT CogMap object names as hints. We use identical prompts and decoding settings, with additional tool calls disabled.

ModelCorrect answers / questionsAccuracy (%)
Qwen3-VL-4B-Instruct, before RL110/20055.00
Tool-Integrated OmniView-Space, after RL142/20071.00

With spatial evidence held fixed, the RL-trained model achieves 16.00 percentage points higher accuracy, supporting improved use of MPSM evidence for downstream reasoning.

(4) The RL-trained policy avoids unnecessary map construction. As reported in Supplementary Material, Table 10, the policy skips MPSM in 75.0% of direct-reasoning cases and achieves 79.2% agreement with human routing labels on the MMSI-Bench Attribute subset, demonstrating conditional tool use.

Sample type# SamplesExpected behaviorPolicy agreement
MPSM-beneficial42Call MPSM88.1%
Direct-reasoning sufficient88Skip MPSM75.0%
Overall130Match human routing label79.2%

Together, these results indicate that RL helps the model adaptively select task-relevant objects and question-aligned ego frames and reason more effectively from the returned spatial evidence. Besides, map distillation and RL further help the model internalize tool-guided spatial capabilities, enabling effective reasoning without external geometry models at inference time.

W2(2): Does the model retry when reconstruction fails?

Thank you for raising this point. The model can revise tool calls when spatial evidence is insufficient or misaligned. On MindCube-Tiny, 93/1,046 tool-using trajectories (8.89%) invoke both BEV-map and spatial-graph tools with revised object hints or ego-reference arguments. We distinguish two failure types:

  • Explicit failures: Tool errors or maps with no detected objects occur in 4/1,046 trajectories (0.38%). In two of these four cases, the model expands its object requests after a graph-generation error and successfully obtains a nonempty graph.
  • Semantic failures: Returned evidence uses an unsuitable reference frame or omits relevant objects. The model can identify these issues and revise its calls—for example, by requesting a sofa-centered graph when a camera-centered BEV does not match the question’s perspective, or expanding object hints when the target backrest is missing.

These examples demonstrate reference revision and supplementary evidence seeking, but recovery remains limited because our framework does not explicitly model failure detection or recovery. We thank the reviewer for motivating explicit consistency checking and reflection training to make them more systematic and reliable in the future.

AuthorsOpenReview ↗

Rebuttal by Authors [3/3]

W2, Q2: Frequency of (1) wrong frames or (2) missing objects , and conditional answer accuracy. (3) Breakdown of how reconstruction errors propagate into the ego frame.

Thank you for this insightful comment. On MindCube-Tiny, ego-frame mismatches occur in 6.51% and 3.33% of evaluable maps for Tool-Integrated and Distilled OmniView-Space, respectively. Key-category omissions are detected in 15.08% and 4.37% of category-evaluable maps. We report their associations with answer accuracy and further examine the role of correct geometric re-anchoring through controlled interventions.

(1) Wrong ego frames. Specifically, we use GPT-6-assisted annotation to identify the camera or object reference specified by each question, and compare it with the map’s declared reference. We examine all 1,050 questions, excluding missing maps and ambiguous references. For camera-motion questions, reference alignment is assessed against the source-camera convention.

ModelQuery-frame mismatches / evaluable mapsAnswer accuracy: reference mismatchedAnswer accuracy: reference matched
Tool-Integrated68/1,045 (6.51%)52.94% (36/68)72.67% (710/977)
Distilled35/1,050 (3.33%)48.57% (17/35)66.31% (673/1,015)

Reference mismatches are associated with accuracy drops of 19.73 and 17.73 percentage points, respectively, indicating the importance of selecting a question-aligned reference.

(2) Missing key objects. We examine all 1,050 questions, excluding samples with ambiguous category correspondence. We identify GT CogMap categories explicitly involved in the question’s task clause or its correct object-valued option, then check their representation in the intermediate map. GT annotations are used only for this post-hoc audit.

ModelDetected omissions / evaluable samplesAccuracy: omission detectedAccuracy: all key categories represented*
Tool-Integrated151/1,001 (15.08%)49.67% (75/151)76.71% (652/850)
Distilled45/1,030 (4.37%)48.89% (22/45)66.70% (657/985)

(3) How object and egp-frame errors affect reasoning. We conduct paired inference-time interventions on all 1,050 MindCube-Tiny questions, keeping the questions, source images, and model weights unchanged. The intervention location differs between the two variants:

  • Tool-Integrated: We modify the object-hint or ego-reference arguments before the first MPSM construction call, regenerate the spatial evidence from the same base reconstruction, and resume reasoning from the returned tool observation.
  • Distilled: We modify the model’s own first predicted <cogmap> JSON before rendering, update the corresponding JSON in the conversation history, and regenerate its BEV map. The model then continues from the renderer’s returned observation, retaining the original images and preceding reasoning context. No externally reconstructed map is injected.

We perturb one component at a time:

  • Object-hint perturbation: We first run Mask2Former trained on ADE20K to extract candidate objects from each scene. We use the MindCube GT CogMap annotations only to identify query-relevant objects and exclude them from the replacement pool. We also exclude the policy-selected ego-anchor object to avoid perturbing the reference frame. We then replace 50% of the non-anchor object hints (at least one when available) with the remaining same-scene candidate objects. The ego anchor and all other inputs remain fixed.
  • Ego-anchor perturbation: We perturb the ego anchor that defines the egocentric coordinate frame used to re-express the CogMap, while keeping the object hints and reconstructed scene geometry unchanged. Specifically, we replace the query-aligned ego anchor with a non-matching but valid anchor from the same scene: another camera for camera-centric questions, another object for object-centric questions, and the reversed source-to-target direction for direction-centric questions.
ConditionTool-IntegratedDistilledΔ vs. Text-RL (52.3%)
Clean: matched control71.565.7+19.2 / +13.4
w/ Object-hint perturbation65.861.8+13.5 / +9.5
w/ Ego-anchor perturbation62.356.3+10.0 / +4.0
  • Controlled effects of map errors. For Tool-Integrated, object-hint and ego-anchor perturbations reduce accuracy from 71.5% to 65.8% (−5.7 pp) and 62.3% (−9.2 pp), respectively. Distilled shows corresponding drops of 3.9 and 9.4 pp.

  • Imperfect maps can still help. Both models remain above Text-RL (52.3%) under both perturbations, suggesting that useful spatial information remains. Object perturbation retains more performance because part of the query-relevant objects are preserved and the replacement objects can still provide coarse scene topology. In contrast, perturbing the ego anchor changes the egocentric coordinate frame used to re-express spatial evidence for the question, leading to a larger drop.

AuthorsOpenReview ↗

Deadline reminder and looking forward to discussion

Dear Reviewer 7rbT,

I hope this message finds you well. Thanks a lot for your time in reviewing and for your insightful comments. We sincerely understand you are busy. But since the discussion deadline is approaching, would you mind checking the response to confirm whether you have any further questions? We highly value your feedback on our paper. We would greatly appreciate it if you could provide your feedback before the deadline expires. Your insights and suggestions are crucial for improving our work.

Thank you for your attention and support. We look forward to hearing from you soon.

Best regards!

Reviewer_ojDUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper introduces OmniView-Space to address spatial reasoning questions that require shifting from the observed camera view to a different reference frame. It constructs query-aligned bird’s-eye-view maps and textual spatial graphs to support reasoning. A learned policy selects spatial anchors and iteratively leverages these representations through multiple tool calls, while a separate model is trained to generate cognitive maps without relying on external tools. The tool-integrated model achieves 71.5 on MindCube-Tiny, 35.5 on MMSI-Bench, and 44.8 on SPAR-Bench.

Critical strengths

Strengths

  1. Addresses an important viewpoint-transformation challenge. Spatial reasoning often requires a reference frame different from the input camera viewpoint. The paper explicitly addresses this mismatch through query-aligned perspective transformation.

  2. Presents a comprehensive tool-integrated framework with a distilled extension. The pipeline combines scene reconstruction, query-conditioned viewpoint selection, visual maps, textual spatial graphs, and interactive tool use. The authors further explore distilling these capabilities into a model that generates cognitive maps without external geometry tools. The method shows substantial gains over the base model, with validation on both an in-domain task family (MindCube-Tiny) and external benchmarks (MMSI-Bench and SPAR-Bench). Its improvements on the external benchmarks, including gains over the Zero configuration, provide evidence that the benefits extend beyond the training task family.

Critical weaknesses

Weaknesses

  1. The distillation analysis could better distinguish the sources of improvement. During SFT, the model learns both to predict MPSM-generated maps and to follow reasoning traces generated by Qwen3-VL-235B. The existing ablations show that the map reward and subsequent RL help, but do not separate the benefit of map prediction from that of teacher reasoning. A matched teacher-rationale baseline without map targets would clarify this contribution.

  2. The presentation could more clearly separate the two pipelines. MPSM construction, tool-use RL, and map distillation are introduced in quick succession, making it difficult to track what is trained at each stage. A compact summary of each stage’s data, training signals, and inference-time tools would make the central contribution easier to follow.

Questions for authors
  1. Clarification on Text-RL training. How does the training setup of Text-RL differ from that of the distilled model? Specifically, do they share the same teacher traces, RL training data, and training budget? What spatial information is provided to Text-RL?

  2. Comparison with 3DThinker. How does OmniView-Space compare with 3DThinker on MindCube-Tiny, MMSI-Bench, and SPAR-Bench? A comparison under consistent evaluation settings, where feasible, would help contextualize the reported improvements.

  3. SPAR-Bench results. Could the authors clarify why Fig. 6 reports 53.3 on SPAR-Bench while Table 1 reports 44.8, and which model or evaluation setting each number represents?

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors [1/2]

W1: Separating map supervision from teacher reasoning

Thank you for this suggestion. We added a matched teacher-rationale-only baseline to distinguish the contribution of explicit map learning from that of teacher reasoning.

Specifically, both students use the same Qwen3-VL-4B initialization, training samples, Qwen3-VL-235B teacher rationales, and SFT budget. The baseline learns only the textual rationales and answers, removing the map-prediction targets and intermediate rendered maps. We compare both models before subsequent RL to avoid confounding this analysis with RL or map rewards. The teacher’s performance is included for reference.

ModelMindCube-TinyMMSI-Bench
Qwen3-VL-235B teacher47.332.6
Qwen3-VL-4B35.927.7
Qwen3-VL-4B + teacher-rationale-only SFT45.828.2
Qwen3-VL-4B + teacher-rationale and map SFT (Ours)50.929.6

The differences between the two methods show an additional benefit from explicit map prediction and map-based reasoning under matched teacher supervision.

W2: Summary of the training and inference pipelines

Thank you for the constructive suggestion. To clarify, in this paper we first introduce MPSM, a toolkit that constructs query-aligned egocentric BEV maps and spatial graphs from input images. We then train two variants: the tool-integrated model learns to call MPSM, while the distilled model learns to predict MPSM maps itself.

Specifically, their training data, training signals, and inference-time tools are summarized below:

VariantTraining dataTraining signalsInference-time tools
Tool-Integrated21K RL samples: 11K from SpatialLadder and 10K from MindCube.Direct GRPO with answer and format rewards; no SFT warm-up.MPSM, including external geometry estimators.
Distilled26K SFT samples from MindCube, SpatialLadder, and VST; the same 21K RL samples.SFT followed by GRPO with answer, format, and map-alignment rewards.A renderer for self-predicted JSON maps; no external geometry estimators.

Both variants support query-aligned spatial reasoning, with the distilled variant internalizing map prediction and avoiding external geometry estimation at inference. We will add this summary to the revised manuscript to clarify the two pipelines.

Q1: Clarification on Text-RL training.

To clarify, Text-RL shares the same RL training data and training budget with the distilled model. They differ in how spatial information is represented and used: Text-RL generates textual CogMaps in a fixed first-view reference frame, while the distilled model generates query-aligned CogMaps and uses their rendered BEV representations during reasoning.

AuthorsOpenReview ↗

Rebuttal by Authors [2/2]

Q2: Comparison with 3DThinker.

Thank you for this constructive suggestion. We compare both distilled and tool-integrated OmniView-Space with the 3DThinker from methodological and empirical perspectives.

1. Firstly, let us outline the similarities between OmniView-Space and 3DThinker:

  • Both approaches introduce geometric representations to support spatial reasoning from limited-view images.
  • Both 3DThinker and distilled OmniView-Space generate spatial representations internally.

2. Next, let us outline the differences between OmniView-Space and 3DThinker:

  • One key difference lies in explicit query-frame alignment. 3DThinker supervises its latents with image-derived VGGT features, whose native reconstruction is anchored to the first input camera. OmniView-Space explicitly selects the query-required camera or object reference and re-expresses spatial evidence in that frame.

  • Another key difference lies in latent geometry versus explicit spatial evidence. While 3DThinker’s latent tokens can be decoded into point clouds, our CogMaps explicitly expose objects, reference anchors, and spatial relations for inspection and quantitative evaluation. We further design a map-alignment reward to supervise internally predicted CogMaps during RL.

  • A third difference lies in complementary textual and visual evidence. Our textual graph supplies numerical evidence for quantitative questions, while the visual BEV map supports qualitative spatial judgments (Fig. 8). This complementarity may support transfer across the different reasoning demands of SPAR-Bench and MMSI-Bench.

  • A fourth difference lies in zero-shot applicability through a standalone toolkit. MPSM can augment an existing MLLM without task-specific fine-tuning, improving Qwen3-VL-4B from 35.9% to 49.1% on MindCube-Tiny before policy training.

We evaluated both the publicly released 3DThinker checkpoint trained on MindCube and a version retrained on our training dataset, as shown below:

ModelMindCube-TinyMMSI-BenchSPAR-Bench
3DThinker (public)68.124.025.8
3DThinker (retrained)68.728.435.1
Distilled OmniView-Space65.731.642.3
Tool-integrated OmniView-Space71.535.544.8

Notably, OmniView-Space’s +7.1-point improvement on the OOD MMSI-Bench over 3DThinker is nontrivial, because gains from spatial training do not necessarily transfer beyond the training distribution. As shown below, MindCube-3B and SpatialLadder-3B improve in-domain MindCube accuracy but perform below their base model on MMSI-Bench:

MethodAdditional spatial training dataMindCubeMMSI-BenchSPAR-Bench
Qwen2.5-VL-3BNone37.628.624.6
MindCube-3BMindCube-10K51.723.920.8
SpatialLadder-3BSpatialLadder-26K43.425.432.9

By comparison, tool-integrated OmniView-Space improves MMSI-Bench from 27.7% to 35.5% over its Qwen3-VL-4B backbone. These results provide complementary evidence that our framework generalizes beyond MindCube-style training questions.

3. Finally, let us emphasize the contributions of OmniView-Space:

  • We formulate multi-image spatial reasoning as query-required ego-frame alignment: before answering, spatial evidence should be re-expressed from the perspective required by the question. This shifts the focus from merely adding stronger geometric evidence to deciding from whose or which perspective the scene should be represented.
  • We propose MPSM as an evidence re-expression toolkit. Table 4 shows that even without further RL training, adding MPSM evidence improves the base model average from 31.8% to 40.4%, indicating that query-aligned spatial evidence itself provides a strong reasoning interface.
  • We train a tool-guided egocentric reasoning policy that actively selects the ego anchor and evidence modality required by the question. This improves performance by +19.2 on MindCube, and by +5.7 and +7.4 on MMSI-Bench and SPAR-Bench. Fig. 8 further shows that the model uses different evidence forms according to task demands: MMSI-Bench uses balanced visual map/text graph calls (46/54), while 3DSRBench relies more on graph evidence (60% graph calls) for detailed 3D relation reasoning.
  • We further distill this process into self-generated cognitive maps with object- and camera-alignment rewards. As shown in Table 4, removing map-alignment reward drops the average performance from 48.7% to 46.4%, demonstrating that explicit intermediate spatial supervision helps preserve ego-frame consistency beyond final-answer optimization.

Q3: Fig. 6 SPAR 53.3 versus Table 1 SPAR 44.8

We apologize for our mistake. The SPAR score of 53.3 in Fig. 6 was incorrect; the correct score is 44.8, as reported in Table 1 for the tool-integrated model. We will correct Fig. 6 in the updated PDF.

AuthorsOpenReview ↗

Deadline reminder and looking forward to discussion

Dear Reviewer ojDU,

I hope this message finds you well. Thanks a lot for your time in reviewing and for your insightful comments. We sincerely understand you are busy. But since the discussion deadline is approaching, would you mind checking the response to confirm whether you have any further questions? We highly value your feedback on our paper. We would greatly appreciate it if you could provide your feedback before the deadline expires. Your insights and suggestions are crucial for improving our work.

Thank you for your attention and support. We look forward to hearing from you soon.

Best regards!

Reviewer_t49cOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents OmniView-Space, a framework for spatial reasoning in multimodal large language models (MLLMs). It introduces Multi-Perspective Spatial Mapping (MPSM), which constructs query-aligned bird's-eye-view (BEV) cognitive maps and textual spatial graphs. The framework further employs reinforcement learning to train a policy that selects appropriate ego-reference frames and invokes spatial tools, followed by cognitive-map distillation with geometric alignment rewards to reduce reliance on external tools at inference time. Experiments across multiple multi-image and single-image spatial reasoning benchmarks demonstrate improvements over several open-source baselines.

Critical strengths
  1. The paper identifies reference-frame inconsistency as a key challenge in complex spatial reasoning and addresses camera-centric, object-centric, and direction-centric perspectives within a unified framework. This is a meaningful problem formulation with potential relevance to multi-view spatial understanding.

  2. The evaluation spans multiple single-image and multi-image spatial reasoning benchmarks and includes modality ablations, map-alignment reward ablations, and tool-use analyses.

Critical weaknesses
  1. MPSM relies on external depth reconstruction, segmentation, and object-orientation estimation, whose errors can propagate into downstream spatial judgments. Although the appendix reports tool failure rates, systematic perturbation experiments examining different error types and severities are lacking, particularly for occluded scenes and sparse views.

  2. The reward, which is based on object centers, object dimensions, and camera poses, may not adequately capture the topological structure of the map or the correctness of spatial relationships. Therefore, it remains unclear whether this reward genuinely promotes the learning of generalizable spatial representations.

Questions for authors

How do the tool-integrated and distilled models perform under controlled perturbations to reconstruction, segmentation, and object-orientation estimates? Can the authors demonstrate that the proposed reward improves topological and relational correctness, rather than primarily optimizing geometric proximity to the reference map?

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors [1/2]

W1,Q1. Performance under reconstruction, segmentation, and orientation errors.

Thank you for this insightful comment. To clarify, our method is relatively robust to the tested depth-scale and segmentation perturbations, but more sensitive to orientation errors and missing views. We explain this from two perspectives: method design and controlled perturbation experiments.

1. Method design: reliable execution and complementary evidence.

  • Observed reliability. The supplementary audit reports toolkit failure rates of 6.2% on MMSI-Bench and 4.1% on MindCube (Sec. E, Table 9), indicating generally reliable tool execution.

  • Complementary evidence. Original images preserve visual details, the textual graph describes spatial relations in the reconstructed scene, and the BEV map summarizes the layout in a shared reference frame. Together, they provide complementary evidence for cross-checking incomplete or inaccurate map estimates.

2. Robustness to occlusion and sparse-view reconstruction.

We apply rectangular occlusions covering 10%, 20%, or 30% of each image, or retain sparse input views for reconstruction. All variants receive identical modified inputs. External spatial evidence is regenerated, while the distilled model predicts its maps internally.

ConditionQwen3-VL-4B+ MPSM-Map+ Multimodal MPSMTool-integratedDistilled
Clean: all original views35.9048.4049.1071.5265.70
Occlusion: 10%34.9547.5748.6770.4865.57
Occlusion: 20%34.1446.9547.3371.1463.38
Occlusion: 30%30.9046.6747.6269.7163.29

Under 30% occlusion, Tool-integrated and Distilled OmniView-Space lose 1.81 and 2.41 percentage points, respectively, showing resilience to the tested occlusions.

MPSM reconstruction inputTool-integrated accuracy (%)
All original views (776-question subset)64.95
Sparse views: Two views58.90
Sparse views: One view54.20

This decline is consistent with the well-known challenges of sparse-view 3D reconstruction, where fewer views provide weaker geometric constraints and make reliable scene reconstruction more difficult.

3. Tool-integrated model: errors in external geometry estimates.

We perturb one external component at a time, keeping the questions and source images unchanged, then regenerate downstream spatial evidence.

PerturbationSeverityAccuracy (%)
Clean: full set (1,050 questions)—71.52
Per-view depth/point-cloud scaling±1% / ±5% / ±10%71.62 / 71.24 / 70.95
Segmentation-mask erosionRadius: 0.5% / 1.5% / 3% of image short side71.24 / 71.71 / 69.54
Segmentation-mask dilationRadius: 0.5% / 1.5% / 3% of image short side71.43 / 69.43 / 70.48
Clean: object-anchored subset (38 questions)—71.05
Object-orientation perturbation: same subset±15° / ±30° / ±60°65.79 / 65.79 / 57.89

As shown in the table, our model is more sensitive to object-orientation errors, which directly affect the ego frame. These results support the importance of correct query-aligned geometric re-anchoring.

4. Distilled model: errors in internally predicted CogMaps.

The distilled model does not invoke external geometry estimators. We therefore perturb its predicted representations directly.

Numerical position and extent errors. We move predicted object centers and camera positions radially relative to the ego origin, or contract/expand BEV boxes while preserving their centers.

PerturbationSeverityAccuracy (%)
Clean—66.10
Object/camera position perturbation±1% / ±5% / ±10% of ego-relative displacement65.81 / 65.33 / 66.19
BEV-box contractionBoundary offset: 0.5% / 1.5% / 3% of map short side65.62 / 65.14 / 66.48
BEV-box expansionBoundary offset: 0.5% / 1.5% / 3% of map short side66.19 / 65.24 / 65.52

The model remains stable under these numerical perturbations. They preserve ego-relative bearings or object centers, helping retain qualitative spatial relations.

Object correspondence and reference selection. Object-hint perturbations replace semantic labels and 2D grounding with same-scene Mask2Former candidates, preserving BEV geometry. Ego-anchor perturbations replace the query-aligned reference with another camera, another object, or the reversed source-to-target direction, depending on the anchor type. (See Reviewer 7rbT, Q2 for intervention details.)

ConditionTool-IntegratedDistilled
Clean: matched control71.565.7
w/ Object-hint perturbation65.861.8
w/ Ego-anchor perturbation62.356.3

Object-hint perturbation reduces accuracy by 5.7 / 3.9 points, while ego-anchor perturbation causes substantially larger drops of 9.2 / 9.4 points for Tool-Integrated / Distilled. These drops highlight the importance of correct object correspondence and query-aligned spatial evidence.

AuthorsOpenReview ↗

Rebuttal by Authors [2/2]

W2,Q1. Does the reward improve relations and topology, beyond geometric proximity?

Thank you for the valuable comments. Indeed, our reward explicitly encourages relational consistency, and additional evaluations support improved relational quality. We clarify this from two perspectives: reward design and experimental evidence.

1. Reward design: explicitly encouraging relational consistency.

Specifically, our map-alignment reward includes a signed pairwise direction-consistency term (Supplementary Material, Sec. D.3), alongside position and orientation alignment. This term compares relative directions between predicted camera pairs with their reference directions. Reversed directions receive negative contributions, penalizing errors such as left–right and front–back reversals. Aligning object centers also indirectly encourages correct object relations (Sec. D.2).

2. Experimental evidence: evaluating relational quality alongside geometric error.

We compare models trained with and without map-alignment reward using three complementary metrics:

  • Normalized center error: Object-center distance to a frozen MPSM pseudo-reference, normalized by the reference layout size and computed on shared matched objects.
  • Direction-relation graph F1: F1 of camera–object and object–object directional edges against MindCube’s symbolic annotations, restricted to the reference object vocabulary. Missing objects reduce recall.
  • QA accuracy: Correctness of final answers.
Reward settingNormalized center error ↓Direction-relation graph F1 (%) ↑MindCube QA accuracy (%) ↑
w/o map-alignment reward1.087844.1263.60
w/ map-alignment reward (Ours)0.965849.3265.70

With map-alignment reward, direction-relation graph F1 increases by 5.20 points, alongside a 2.10-point improvement in MindCube QA accuracy. On OOD MMSI-Bench, QA accuracy also improves from 29.1% to 31.6% (Table 4).

3. Further analysis: relational correctness at similar geometric error.

To examine whether relational gains persist at comparable levels of center error, we group each model’s predictions using the same two error intervals and compare direction-relation graph F1.

Normalized center-error intervalw/o map-alignment rewardw/ map-alignment reward (Ours)
[0, 0.25)46.0851.70
[0.25, ∞)43.3848.51

The model with map-alignment reward achieves higher relation-graph F1 in both intervals, providing additional evidence of improved relational quality. This comparison is descriptive: interval membership can differ between models, and binning does not eliminate all differences in geometric error.

These results support improved relational quality in the generated maps, alongside better downstream reasoning, reinforcing the value of explicitly supervising intermediate spatial representations.

AuthorsOpenReview ↗

Deadline reminder and looking forward to discussion

Dear Reviewer t49c,

I hope this message finds you well. Thanks a lot for your time in reviewing and for your insightful comments. We sincerely understand you are busy. But since the discussion deadline is approaching, would you mind checking the response to confirm whether you have any further questions? We highly value your feedback on our paper. We would greatly appreciate it if you could provide your feedback before the deadline expires. Your insights and suggestions are crucial for improving our work.

Thank you for your attention and support. We look forward to hearing from you soon.

Best regards!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all three reviewers for their thoughtful feedback. Reviewers (t49c, ojDU) highlighted the importance of reasoning from the viewpoint specified by the question. The reviewers also recognized the breadth of our evaluation (t49c), the gains on external benchmarks (ojDU, 7rbT), and the explicit transformations that place visual maps and textual graphs in a shared reference frame (7rbT).

Experimental Additions

We conducted the following experiments in response to the reviewers’ questions.

  • [Reviewer t49c] Both variants retained most of their accuracy with up to 30% of each image occluded. The tool-integrated model was more sensitive to missing views and orientation errors than to the tested depth and segmentation errors, highlighting the importance of sufficient views for reconstruction and reliable orientation estimates for defining the reference frame.
  • [Reviewer t49c] The map-alignment reward improved direction-relation graph F1 on MindCube-Tiny, including within both center-error intervals. This shows that the map reward improves the topological and relational correctness of the generated maps.
  • [Reviewer ojDU] We added a matched teacher-rationale-only baseline. Under the same teacher supervision, adding map prediction and map-based reasoning improved accuracy before RL by 5.1 points on MindCube-Tiny and 1.4 points on MMSI-Bench.
  • [Reviewer ojDU] We compared both variants with public and retrained 3DThinker models. Tool-integrated OmniView-Space scored highest on all three multi-image benchmarks, and the distilled variant outperformed both 3DThinker models on MMSI-Bench and SPAR-Bench.
  • [Reviewer ojDU] Expressing textual maps in the requested viewpoint improved MMSI-Bench accuracy from 30.4% to 32.5%. Adding the visual map raised it to 35.5%, showing that visual evidence remains useful even when the text already uses the correct reference frame.
  • [Reviewer 7rbT] Both variants showed larger gains over MindCube-3B on the 717 questions without exact training-template overlap than on the overlapping questions. The MindCube-Tiny gains therefore extend beyond questions whose templates appeared in training.
  • [Reviewer 7rbT] RL improved reference-frame selection and coverage of task-relevant objects. With identical map evidence, QA accuracy also rose from 55% to 71%, showing that the learned policy improves both the evidence it requests and how it uses that evidence.
  • [Reviewers t49c, 7rbT] The error audit linked wrong reference frames and missing key objects to lower answer accuracy. In controlled interventions, changing the ego anchor caused a larger drop than perturbing object hints for both variants, highlighting the importance of expressing evidence from the correct viewpoint.

Clarifications and Explanations

We also clarified the following points raised by the reviewers.

  • [Reviewers ojDU, 7rbT] We summarized how each variant is trained and uses maps at inference. The tool-integrated model learns to call MPSM, while the distilled model predicts and renders its own maps. We also clarified the training setup of Text-RL. We will include this summary in the revision.
  • [Reviewer ojDU] We clarified how OmniView-Space differs from 3DThinker and ViewFusion, highlighting how our method expresses reconstructed 3D evidence in a query-aligned reference frame through two complementary representations: a visual BEV map and a textual spatial graph.
  • [Reviewer 7rbT] We documented when the policy skips map construction or revises a failed tool call. We also discussed the weaker single-image SPBench result in light of the policy’s multi-view training and the prevalence of questions about absolute distance and object size.
  • [Reviewer ojDU] We clarified that the SPAR-Bench discrepancy was a plotting error and we will correct Fig. 6 in the revision.

Contributions of OmniView-Space

  • Query-aligned egocentric reasoning. We formulate multi-image spatial reasoning as a reference-frame alignment problem. The question determines which camera, object, or viewing direction defines the ego frame. Expressing scene geometry in that frame lets the model interpret relations such as left and right from the requested perspective.
  • MPSM Toolkit. MPSM turns 3D scene geometry into a BEV map and a textual graph in a query-aligned frame, raising Qwen3-VL-4B's accuracy on MindCube-Tiny and MMSI-Bench from 31.8% to 40.4%.
  • Tool-guided reasoning. Our RL policy learns to select the ego anchor and request the spatial evidence needed to answer the question, which outperforms Text-RL by 19.2/5.7/7.4 pp on MindCube/MMSI/SPAR-Bench. Fig. 8 further shows that the model uses different evidence adaptively.
  • Distillation into self-generated maps. - The distilled model predicts its own map and reasons without external geometry tools. Its map-alignment reward raises mean accuracy on MindCube-Tiny and MMSI-Bench from 46.4% to 48.7%.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.