- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that a bottleneck in multi-step spatial reasoning is reference-frame mismatch: existing 3D evidence or textual spatial graphs are usually expressed in a system frame, while questions require judgment in an ego frame defined by a camera, an object, or a direction. OmniView-Space uses MPSM to re-anchor a scene reconstructed by Depth Anything 3 and SAM 3 into the query-specified ego frame, returning a bird’s-eye cognitive map and a textual spatial graph. GRPO then trains a tool-use policy that selects the anchor and the evidence format. Finally, MPSM trajectories are used for SFT, and an object- and camera-level map-alignment reward distills this behavior into the model so that inference does not need the full geometry pipeline. The backbone is Qwen3-VL-4B. The tool-using variant scores 50.6 overall on MindCube-Tiny, MMSI, and SPAR; the distilled variant scores 46.5, compared with 33.1 for Qwen3-VL-4B. It also leads the listed open-source models on the allocentric subset of 3DSRBench and the perspective subset of OmniSpatial.
Critical strengths
- The problem is concrete, and the interface is executable. The coordinate transforms for camera, object, and direction anchors are specified, and the visual map and textual graph come from the same ego frame. Table 4 shows that prompting the base model with MPSM evidence alone raises the MindCube/MMSI average from 31.8 to 40.4; tool RL reaches 50.6, and distillation still scores 48.7. Appendix C audits image, scene, and question-template overlap between training and evaluation, and it explicitly treats MindCube-Tiny and SPBench as in-domain. This breakdown is more informative than a single overall score.
- The out-of-domain benchmarks are not a zero-gain setting. The exact template-overlap audit is zero for MMSI, SPAR, 3DSRBench, and OmniSpatial. MMSI rises from 27.7 to 35.5 for the tool-using model and 31.6 for the distilled model; SPAR rises from 35.6 to 44.8 and 42.3. Appendix Table 8 shows that training directly on MindCube or SpatialLadder does not improve MMSI, so the MMSI gain cannot be attributed simply to exposure to MindCube templates.
Critical weaknesses
- The most prominent MindCube result is in a setting the authors themselves mark as in-domain, and the SFT trajectories include answer-conditioned repair. MindCube training templates exactly match 333 of 1,050 MindCube-Tiny questions (31.7%); the overlap on SPBench-SI is 78.1%. In round two of SFT construction, rationales for incorrect samples are rewritten using the ground-truth answer and then filtered by a critic. The tool-using MindCube score of 71.5 is a headline number in the main text, but the main text does not report the zero-template-overlap subset. On SPBench, the text baseline is higher than the full method (63.0 versus 60.0), so the ego map is not a stable gain.
- The learned policy adds a limited amount beyond the external geometry pipeline. Prompted MPSM already contributes about 8.6 points on average, and the map-alignment reward adds only about 2.3. The distilled model is 4.1 overall points below the tool-using model. The paper therefore mainly shows that rendering a reconstruction in the query coordinate frame is useful to an MLLM. The main text does not provide a decisive error breakdown of how reconstruction errors propagate into the ego frame, or of whether the model still calls the tool when reconstruction fails.
I would like to raise my score if the authors can solve my concerns.
Questions for authors
- Please report accuracy on the 717 MindCube-Tiny samples with zero template overlap, compared under the same protocol with Qwen3-VL-4B, MindCube-3B, and SpatialLadder. If the advantage remains mostly on the 333 overlapping templates, I would lower the conclusion drawn from the main table.
- Among tool-call trajectories, what fraction have a clearly wrong MPSM ego frame or miss a key object? How much lower is answer accuracy on those samples than on samples with a correct reconstruction? This determines whether the gain comes from correct geometric re-anchoring or from simply seeing an additional map.
4 comments
Rebuttal by Authors [2/3]
W2(1): What does RL learning add beyond a prompted geometry pipeline?
Thank you for this insightful comment. RL helps the model (1) select question-aligned reference frames, (2) request relevant objects, (3) interpret tool-provided spatial evidence for reasoning, and (4) decide when to use tools. Across three benchmarks, Tool-Integrated RL improves average accuracy from 40.1% with prompted MPSM to 50.6% (+10.5 pp) (Table 1). The 2.3-point map-alignment reward gain reflects an auxiliary reward’s incremental benefit, not the overall contribution of RL. We examine these four aspects below.
(1) RL improves question-aligned reference selection. We used the same 192 MMSI-Bench questions to evaluate the accuracy of Qwen3-VL-4B-Instruct and the strategies trained through RL in selecting the reference frame.
| Model | Question-aligned reference selections | Alignment rate (%) |
|---|---|---|
| Qwen3-VL-4B-Instruct, before RL | 101/192 | 52.60 |
| Tool-Integrated OmniView-Space, after RL | 132/192 | 68.75 |
RL increases the alignment rate by 16.2 percentage points, indicating that the trained policy selects reference frames that better match the question.
(2) RL improves task-relevant object requests. On 776 MindCube-Tiny questions, we inspect whether the object hints requested before segmentation cover the key categories involved in the question or its correct object-valued option. GT CogMap labels are used only for this post-hoc audit.
| Model | Requests covering all key categories |
|---|---|
| Prompted Qwen3-VL-4B-Instruct | 484/776 (62.37%) |
| Tool-Integrated OmniView-Space | 622/776 (80.15%) |
Complete category coverage increases by 17.78 percentage points, indicating more complete task-relevant object requests in the learned pipeline.
(3) RL improves spatial reasoning through more effective use of MPSM evidence. To assess whether the RL-trained model better uses supplied spatial evidence to answer questions, we randomly sample 200 MindCube-Tiny questions. Both models receive only the question, answer options, and the same query-aligned MPSM evidence. We exclude original images to focus the comparison on reasoning from the supplied MPSM evidence. The evidence is constructed using GT CogMap object names as hints. We use identical prompts and decoding settings, with additional tool calls disabled.
| Model | Correct answers / questions | Accuracy (%) |
|---|---|---|
| Qwen3-VL-4B-Instruct, before RL | 110/200 | 55.00 |
| Tool-Integrated OmniView-Space, after RL | 142/200 | 71.00 |
With spatial evidence held fixed, the RL-trained model achieves 16.00 percentage points higher accuracy, supporting improved use of MPSM evidence for downstream reasoning.
(4) The RL-trained policy avoids unnecessary map construction. As reported in Supplementary Material, Table 10, the policy skips MPSM in 75.0% of direct-reasoning cases and achieves 79.2% agreement with human routing labels on the MMSI-Bench Attribute subset, demonstrating conditional tool use.
| Sample type | # Samples | Expected behavior | Policy agreement |
|---|---|---|---|
| MPSM-beneficial | 42 | Call MPSM | 88.1% |
| Direct-reasoning sufficient | 88 | Skip MPSM | 75.0% |
| Overall | 130 | Match human routing label | 79.2% |
Together, these results indicate that RL helps the model adaptively select task-relevant objects and question-aligned ego frames and reason more effectively from the returned spatial evidence. Besides, map distillation and RL further help the model internalize tool-guided spatial capabilities, enabling effective reasoning without external geometry models at inference time.
W2(2): Does the model retry when reconstruction fails?
Thank you for raising this point. The model can revise tool calls when spatial evidence is insufficient or misaligned. On MindCube-Tiny, 93/1,046 tool-using trajectories (8.89%) invoke both BEV-map and spatial-graph tools with revised object hints or ego-reference arguments. We distinguish two failure types:
- Explicit failures: Tool errors or maps with no detected objects occur in 4/1,046 trajectories (0.38%). In two of these four cases, the model expands its object requests after a graph-generation error and successfully obtains a nonempty graph.
- Semantic failures: Returned evidence uses an unsuitable reference frame or omits relevant objects. The model can identify these issues and revise its calls—for example, by requesting a sofa-centered graph when a camera-centered BEV does not match the question’s perspective, or expanding object hints when the target backrest is missing.
These examples demonstrate reference revision and supplementary evidence seeking, but recovery remains limited because our framework does not explicitly model failure detection or recovery. We thank the reviewer for motivating explicit consistency checking and reflection training to make them more systematic and reliable in the future.
Rebuttal by Authors [3/3]
W2, Q2: Frequency of (1) wrong frames or (2) missing objects , and conditional answer accuracy. (3) Breakdown of how reconstruction errors propagate into the ego frame.
Thank you for this insightful comment. On MindCube-Tiny, ego-frame mismatches occur in 6.51% and 3.33% of evaluable maps for Tool-Integrated and Distilled OmniView-Space, respectively. Key-category omissions are detected in 15.08% and 4.37% of category-evaluable maps. We report their associations with answer accuracy and further examine the role of correct geometric re-anchoring through controlled interventions.
(1) Wrong ego frames. Specifically, we use GPT-6-assisted annotation to identify the camera or object reference specified by each question, and compare it with the map’s declared reference. We examine all 1,050 questions, excluding missing maps and ambiguous references. For camera-motion questions, reference alignment is assessed against the source-camera convention.
| Model | Query-frame mismatches / evaluable maps | Answer accuracy: reference mismatched | Answer accuracy: reference matched |
|---|---|---|---|
| Tool-Integrated | 68/1,045 (6.51%) | 52.94% (36/68) | 72.67% (710/977) |
| Distilled | 35/1,050 (3.33%) | 48.57% (17/35) | 66.31% (673/1,015) |
Reference mismatches are associated with accuracy drops of 19.73 and 17.73 percentage points, respectively, indicating the importance of selecting a question-aligned reference.
(2) Missing key objects. We examine all 1,050 questions, excluding samples with ambiguous category correspondence. We identify GT CogMap categories explicitly involved in the question’s task clause or its correct object-valued option, then check their representation in the intermediate map. GT annotations are used only for this post-hoc audit.
| Model | Detected omissions / evaluable samples | Accuracy: omission detected | Accuracy: all key categories represented* |
|---|---|---|---|
| Tool-Integrated | 151/1,001 (15.08%) | 49.67% (75/151) | 76.71% (652/850) |
| Distilled | 45/1,030 (4.37%) | 48.89% (22/45) | 66.70% (657/985) |
(3) How object and egp-frame errors affect reasoning. We conduct paired inference-time interventions on all 1,050 MindCube-Tiny questions, keeping the questions, source images, and model weights unchanged. The intervention location differs between the two variants:
- Tool-Integrated: We modify the object-hint or ego-reference arguments before the first MPSM construction call, regenerate the spatial evidence from the same base reconstruction, and resume reasoning from the returned tool observation.
- Distilled: We modify the model’s own first predicted
<cogmap>JSON before rendering, update the corresponding JSON in the conversation history, and regenerate its BEV map. The model then continues from the renderer’s returned observation, retaining the original images and preceding reasoning context. No externally reconstructed map is injected.
We perturb one component at a time:
- Object-hint perturbation: We first run Mask2Former trained on ADE20K to extract candidate objects from each scene. We use the MindCube GT CogMap annotations only to identify query-relevant objects and exclude them from the replacement pool. We also exclude the policy-selected ego-anchor object to avoid perturbing the reference frame. We then replace 50% of the non-anchor object hints (at least one when available) with the remaining same-scene candidate objects. The ego anchor and all other inputs remain fixed.
- Ego-anchor perturbation: We perturb the ego anchor that defines the egocentric coordinate frame used to re-express the CogMap, while keeping the object hints and reconstructed scene geometry unchanged. Specifically, we replace the query-aligned ego anchor with a non-matching but valid anchor from the same scene: another camera for camera-centric questions, another object for object-centric questions, and the reversed source-to-target direction for direction-centric questions.
| Condition | Tool-Integrated | Distilled | Δ vs. Text-RL (52.3%) |
|---|---|---|---|
| Clean: matched control | 71.5 | 65.7 | +19.2 / +13.4 |
| w/ Object-hint perturbation | 65.8 | 61.8 | +13.5 / +9.5 |
| w/ Ego-anchor perturbation | 62.3 | 56.3 | +10.0 / +4.0 |
-
Controlled effects of map errors. For Tool-Integrated, object-hint and ego-anchor perturbations reduce accuracy from 71.5% to 65.8% (−5.7 pp) and 62.3% (−9.2 pp), respectively. Distilled shows corresponding drops of 3.9 and 9.4 pp.
-
Imperfect maps can still help. Both models remain above Text-RL (52.3%) under both perturbations, suggesting that useful spatial information remains. Object perturbation retains more performance because part of the query-relevant objects are preserved and the replacement objects can still provide coarse scene topology. In contrast, perturbing the ego anchor changes the egocentric coordinate frame used to re-express spatial evidence for the question, leading to a larger drop.
Deadline reminder and looking forward to discussion
Dear Reviewer 7rbT,
I hope this message finds you well. Thanks a lot for your time in reviewing and for your insightful comments. We sincerely understand you are busy. But since the discussion deadline is approaching, would you mind checking the response to confirm whether you have any further questions? We highly value your feedback on our paper. We would greatly appreciate it if you could provide your feedback before the deadline expires. Your insights and suggestions are crucial for improving our work.
Thank you for your attention and support. We look forward to hearing from you soon.
Best regards!
Rebuttal by Authors [1/3]
To clarify, on the 717 samples without exact training-template overlap, Tool-Integrated and Distilled OmniView-Space outperform the MindCube-3B, by 25.67 and 19.55 percentage points, respectively. These results show that our gains are not concentrated on overlapping templates. We provide two complementary lines of evidence below.
1. Performance with and without training-question overlap.
Both variants show larger gains over MindCube-3B on non-overlapping questions, indicating that their improvements are not concentrated on overlapping templates. Tool integration further improves the distilled model by 6.12 points on non-overlapping questions.
2. Additional evidence from models trained on the same public sources.
To examine whether exposure to MindCube/SpatialLadder-style data consistently improves transfer, we include the following reference results:
These models improve on in-domain MindCube but do not improve MMSI-Bench over their base model. By comparison, OmniView-Space improves MMSI-Bench from 27.7 to 35.5 on its Qwen3-VL-4B backbone, alongside strong results on SPAR-Bench, 3DSRBench, and OmniSpatial. This suggests that the gains are not simply due to exposure to similar training distributions, but come from query-aligned spatial representation and tool-guided reasoning.
To clarify, answer-conditioned repair is used only for the distilled model’s SFT data. The tool-integrated model is trained directly with GRPO, without SFT or repaired rationales. Thus, its 71.5% accuracy on MindCube-Tiny and gains on the 717 non-overlapping-template samples do not depend on this procedure. We will clarify this distinction in the paper.
To clarify, the slightly lower result on single-image SPBench is mainly because this subset focuses more on perception-level questions, which are less aligned with the core target of OmniView-Space. Since our RL data are mainly derived from multi-image scenarios, the learned policy is more specialized for tool-based reasoning under multi-view evidence aggregation, rather than for single-image perception-only cases. Meanwhile, over 60% of the 1,009 SPBench-SI samples are metric-related questions, such as absolute distance and object size estimation, which can often be answered more directly from the original image and the MLLM’s visual perception.
Importantly, this limitation is mainly observed on simple single-image perception tasks. For more complex spatial reasoning tasks that require viewpoint transformation, cross-view aggregation, or ego-frame re-anchoring, MPSM brings much larger gains. For example, OmniView-Space improves performance by +19.2 on MindCube, and by +5.7 and +7.4 on the out-of-domain benchmarks MMSI-Bench and SPAR-Bench, respectively. We will add this question-type analysis in the revised manuscript.