acceptodds
Under review as a conference paper at ICLR 2027

DeSpatial: The devil of Agentic Spatial Reasoning Lies in 3D Itself

16 views

Abstract

Spatial agents have access to reconstruction and segmentation tools, yet inconsistent predictions and fragmented object identities can prevent their outputs from becoming reliable 3D scene information. A simple oracle experiment exposes this perception bottleneck: replacing tool-predicted perception with ground-truth 3D bounding boxes raises an agent to nearly full score across five object-centric VSI-Bench tasks, while retaining the same reasoning tools. Motivated by this finding, we believe the devil of spatial reasoning lies in 3D itself, and introduce DeSpatial(Devil of Spatial), which explicitly decomposes spatial understanding into perception and reasoning. DeSpatial has a shared 3D perception tool to reconcile independent perception outputs into a unified, metric, gravity-aligned 3D scene. This perception tool has two core components: a) a joint segmentation–depth co-refinement module that consolidates masks of the same object and removes projection artifacts caused by inconsistent segmentation and depth within a single frame; b) a cross-view instance fusion module that consolidate voxel-overlapping instances but not allowed to merge same-frame instances. This fusion module additionally fuses spatially disjoint fragments caused by insufficient multi-view observations, by checking their 3D structure and source mask appearance. After receiving the complete and unified 3D scene information, the agent chooses reasoning or computational tools to solve the spatial problem. DeSpatial achieves leading performance on VSI-Bench, MindCube-Tiny, and ViewSpatial, with additional long-video results on VSI-SUPER, revealing the bottleneck of spatial understanding lies in a precise 3D perception tool.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
spatial intelligence, 3D reconstruction, instance segmentation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LrXuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DeSpatial builds a shared metric, gravity-aligned 3D scene for a spatial reasoning agent. It cleans segmentation and depth inconsistencies, fuses object observations across views with same-frame exclusion, and uses visual judgments to associate disjoint fragments. The resulting scene is queried by spatial computation tools and evaluated on four benchmarks.

Critical strengths
  1. The method addresses concrete sources of error in tool-based spatial reasoning. Same-frame exclusion and part-aware filtering have clear purposes, and Table 5 shows an 11.27-point improvement from the full co-refinement pipeline over raw lifting on the fixed subset.

  2. Scores of 69.22 on VSI-Bench, 87.19 on MindCube-Tiny, and 68.35 on ViewSpatial show benefits across object measurements and viewpoint changes. The oracle study motivates improving the scene representation, with its annotation-related limits acknowledged. The controlled component gains and breadth of evaluation support weak acceptance.

Critical weaknesses
  1. Main-table comparisons mix backbones, perception tools, and in some cases evaluation subsets. The 504-question ablation establishes a within-pipeline benefit, but broader matched comparisons would clarify its size relative to prior agents. This limits the precision of the comparative claim.

  2. Direct measurements of instance association and geometric error are missing, as are end-to-end cost and scaling results. These would make the observed accuracy gains easier to explain and the practical tradeoff easier to judge, particularly for long videos with repeated visual identity judgments.

  3. Fusion uses fixed 0.1 m voxels and a hard same-frame exclusion rule. If mask consolidation leaves one object split within a frame, later fusion cannot repair that identity. Sensitivity to voxel size, scale error, and these upstream mistakes is not quantified, so the robustness of the association rules is uncertain.

Questions for authors
  1. Under the same Qwen3-VL-8B backbone, reconstruction inputs, and frame budget, how does DeSpatial compare with the strongest prior scene-building pipeline?

  2. Can you report direct instance-fusion and object-geometry metrics before and after refinement, including false merges and fragmented identities? These would test the proposed mechanism independently of answer accuracy.

  3. What are the end-to-end latency, visual-judge call count, and frame-sampling policy on VSI-Bench versus VSI-SUPER? This would clarify whether the long-video gains require substantially more inference work.

3 comments
AuthorsOpenReview ↗

Response to LrXu Part 1

We thank Reviewer LrXu for the positive evaluation and thoughtful questions on matched comparisons, direct geometric validation, runtime, and robustness.

Q1. Matched comparison with prior scene-building pipelines under identical inputs

Under the exact same Qwen3-VL-8B-Instruct backbone, Pi3 + SAM3 inputs, and 32-frame budget on full VSI-Bench, we compare DeSpatial against matched prior scene-building pipelines (Appendix E.1, Tables 8–9 and Appendix E.3, Table 13):

  • A0 (AlloSpatial): Full AlloSpatial pipeline (world2mind perception + 2D top-view/height-range representation + AlloSpatial reasoning tools).
  • A1 (+DeSpatial Percept.): Replaces only world2mind with DeSpatial's perception module (keeping the 2D grid representation and AlloSpatial's tools).
  • A2 (+3D OBB Repr.): Replaces the 2D grid with DeSpatial's 3D OBB representation (keeping AlloSpatial's tools).
  • OVIR-3D / MaskClustering (adapted): Replaces only DeSpatial's perception module (A3) with adapted OVIR-3D or MaskClustering under the exact same Pi3 + SAM3 inputs, while keeping our 3D OBB representation and reasoning tools unchanged.
  • A3 (Full DeSpatial): DeSpatial perception + 3D OBB representation + 3D OBB reasoning tools.
Pipeline / BackendObj. CountAbs. Dist.Obj. SizeRoom SizeRel. Dist.Rel. Dir.Route PlanAppr. OrderTotal (%)
AlloSpatial (A0, matched)38.9233.5747.3035.7053.9741.6833.7553.9342.35
AlloSpatial + DeSpatial Percept. (A1)63.1941.0360.0038.4952.8642.8638.9252.8648.78
AlloSpatial + 3D OBB Repr. (A2)63.2449.8662.8656.5154.4441.1039.5753.6252.65
OVIR-3D (adapted)42.8346.4355.6363.9753.9060.2156.2563.0255.28
MaskClustering (adapted)54.0250.6359.4160.3756.9775.7859.5668.0560.60
DeSpatial (Full, A3)65.4355.0866.8673.5867.9382.5364.4377.9069.22

Replacing AlloSpatial's perception alone (A0 → A1) yields +6.43 points, while full DeSpatial (A3) outperforms matched AlloSpatial by +26.87 points, adapted OVIR-3D by +13.94 points, and adapted MaskClustering by +8.62 points.


Q2. Direct instance-fusion and object-geometry metrics before and after refinement

Evaluating 184 scenes (1,016 scene-class cells) directly under shared Pi3, SAM3, and GeoCalib inputs shows steady geometric gains from each stage (B0 raw fusion → B1 +duplicate consolidation → B2 +same-frame part consolidation → B3 +cross-frame fragment association):

ConfigCount MAE ↓Fragmented Identities ↓False-merged Groups[email protected][email protected][email protected]
B00.878097 / 9705 / 110347.8564.1654.82
B10.824877 / 9706 / 107349.3264.1655.77
B20.779569 / 9706 / 106050.1764.4356.41
B3 (Full)0.697843 / 9709 / 102856.1565.3060.38

AuthorsOpenReview ↗

Response to LrXu Part 2

Q3. End-to-end latency, visual-judge calls, and frame-sampling policy (VSI-Bench vs. VSI-SUPER)

  • VSI-Bench latency (4 GPUs, 32 frames, full benchmark):
ConfigEnd-to-End Time (min)Main VLM (min)Tool Execution (min)Avg. Tool Calls / Q
A0 (AlloSpatial)179.4095.8268.602.24
A1 (+DeSpatial Percept.)275.9698.99160.052.01
A2 (+3D OBB Repr.)244.1986.11149.461.85
A3 (Full DeSpatial)211.1362.41142.991.75

While 3D scene construction takes longer than AlloSpatial's 2D pipeline (142.99 vs. 68.60 min), structured 3D OBB tools substantially reduce Main VLM inference time (62.41 vs. 95.82 min) because the agent no longer needs repeated point-cloud viewing steps.

  • Visual-judge call count & VSI-SUPER sampling/latency: On VSI-Bench (32 frames), the visual verifier (Qwen3-VL-8B-Instruct) averages 16.5 calls per scene. On VSI-SUPER, following S-Agent, we use an adaptive long-video access pipeline instead of a fixed 32-frame sample: (1) Overview (32 uniform frames for target selection), (2) Full-video scan (LocateAnything at 1 FPS), and (3) Local densification (±2s around each hit, merged windows sampled at 4 FPS). Across video durations from 10 to 240 minutes, per-question runtime and verifier call counts scale roughly linearly with video length:
Video DurationRecall Time (s/Q)Count Time (s/Q)Recall Verifier Calls / QCount Verifier Calls / Q
10 min102.92137.5284.31108.60
30 min290.41384.49179.52233.93
60 min550.87633.13343.09397.58
120 min1,109.971,224.71656.42752.77
240 min2,289.71—1,169.20—
Overall Avg.868.77594.96——

W3. Sensitivity to voxel size and same-frame exclusion

  • Voxel size sensitivity (184 scenes & full VSI-Bench):
Voxel (m)Count MAE ↓[email protected][email protected][email protected]VSI-Bench Total (%)
0.0250.935050.7064.1656.6459.51
0.050.779553.1664.5858.3265.11
0.10 (default)0.697856.1565.3060.3869.22
0.150.688355.3464.5259.5868.50
0.200.685455.5164.6059.7168.66

Too-small voxels (0.025 m) under-merge slightly separated parts (59.51%), whereas performance across 0.10–0.20 m is stable (68.50%–69.22%) because same-frame exclusion prevents nearby objects from collapsing even at coarser voxel resolutions.

  • Trade-off and quantitative impact of hard same-frame exclusion: If an object remains split within a single frame after consolidation, the hard same-frame exclusion rule indeed prevents merging those two groups later. We retain this hard constraint intentionally to keep false merges very low even at coarser voxel sizes (only 9/1028, or <0.9%, in B3). Quantitatively (Q2 above and Appendix E.2, Table 11), the within-frame consolidation stages (B0 → B2) already reduce fragmented identities from 97/970 to 69/970, and cross-frame fragment association (B2 → B3) further links disjoint cross-view observations down to 43/970 (a 55.7% overall reduction) while raising OBB F1 from 56.41 to 60.38.

Thanks for the rebuttal. The rebuttal addresses my main concerns. I have no further questions and will keep my original rating.

Reviewer_dy1sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that perception is the main bottleneck in spatial agents and consequently their performance is highly affected by the quality of the 3D scene information the agent reasons over. They provide an orcale-based experiment showing that with ground-truth scene information the agents can achieve near-perfect performance. Motivated by this, they also propose DeSpatial that aims to improve the perception step by 1) using metric and gravity-aligned reconstruction, 2) Joint segmentation–depth co-refinement, 3) Cross-view instance fusion. They evaluate their method on VSI, Mindcube and ViewSpatial datasets showing improvement over the baseline.

Critical strengths
  1. The main question of the paper is very interesting. It is timely and poses a useful diagnostic about the source of errors in spatial agents. By keeping the reasoning tools fixed and changing the scene information we can see how much the agent depends on perception.

  2. The modifications and additions to their perception pipeline in the DeSpatial method are well motivated, and the design is careful.

  3. The paper reports strong results on four benchmarks using Qwen3-VL-8B as the agent in addition to the ablation study showing the contribution of each component.

Critical weaknesses
  1. The conclusion of the oracle experiment is stronger than what it shows. The oracle uses the same 3D annotations used to generate the answers and only covers five categories. The original VSI-Bench paper also shows improvement when ground-truth maps are provided. How does DeSpatial's oracle relate to this result?

  2. The baseline methods use different backbones and, in some cases, different question sets. This makes the comparisons not apples-to-apples and since the paper is using most recent grounding/masking/reconstuction tools this makes their claimed improvements not valid. Can you compare them using the same backbone and questions? This would help show how much the proposed perception method contributes.

  3. It's not clear which backbones models the tools use, which reconstruction models and VLM judge were used? How did you estimate object orientations? How were the thresholds selected, and were they tuned on the same questions used for the test and ablations? each of these details are important to assess the contribution of the proposed method.

  4. The method merges 2D masks into 3D instances but does not discuss the open-vocabulary 3D instance segmentation literature. Based on my simple search, OVIR-3D [1] is closely related to the voxel-overlap association, MaskClustering [2] to the graph-based cross-view fusion, and Panoptic Lifting [3] to the same-frame exclusion rule. How do the proposed fusion and exclusion rules differ? A comparison with one of these methods as the perception backend would help show what DeSpatial adds.

  5. The paper does not report any runtime or model calls per question. In addition, while it argues for improved perception, it mainly evaluates downstream QA accuracy with no direct evaluation of object detection, 3D box accuracy, and matching objects across views.

I like the main question and the diagnostic experiment, but the comparisons and implementation details are not clear enough to assess the contribution.

References

[1] Lu, S., Chang, H., Jing, E. P., Boularias, A., and Bekris, K. OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Data. CoRL, 2023.

[2] Yan, M., Zhang, J., Zhu, Y., and Wang, H. MaskClustering: View Consensus Based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentation. CVPR, 2024.

[3] Siddiqui, Y., Porzi, L., Rota Bulò, S., Müller, N., Nießner, M., Dai, A., and Kontschieder, P. Panoptic Lifting for 3D Scene Understanding with Neural Fields. CVPR, 2023.

Questions for authors

please see the weaknesses

5 comments
AuthorsOpenReview ↗

Response to dy1s Part 1

We thank Reviewer dy1s for the encouraging comments on our diagnostic framing and for pointing us to relevant 3D instance lifting works. Below we respond directly to each weakness with new controlled experiments and clarifications.

W1. Oracle diagnostic scope and relation to VSI-Bench GT cognitive maps

We updated the Abstract and Section 2 to frame the oracle diagnostic more precisely: it identifies 3D perception as one important bottleneck for the evaluated agent under annotation-aligned 3D boxes.

We also ran GT cognitive map experiments following VSI-Bench's 2D integer-grid protocol on the same 4,030 questions:

3D ToolObj. CountAbs. Dist.Obj. SizeRel. Dist.Rel. Dir.Avg.
SAM3 + Pi3 (raw)42.7032.5449.6847.6279.3750.38
GT 3D Boxes98.7696.7399.3798.5997.7098.23
GT cognitive map98.41//76.1997.10/

Note that in the original VSI-Bench paper, the GT cognitive map experiment was only conducted on Relative Distance. On Relative Distance, our GT cognitive map achieves 76.19%, closely aligning with VSI-Bench's published GT cognitive map result (66.0% at 10×10 and 78.0% at 20×20; Yang et al. [1], Table 2). Extending it to Counting (98.41%) and Relative Direction (97.10%) shows that cognitive maps support these tasks as well, while metric 3D OBBs are additionally required for metric object size and absolute distance. Pure cognitive map does not have these information.

[1] Yang, J., Yang, S., Gupta, A. W., Han, R., Li, F.-F., & Xie, S. Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces. CVPR 2025.


W2. Apples-to-apples comparisons under the same backbone and question set

We evaluated AlloSpatial (A0) and progressive replacements toward DeSpatial (A1 → A3) under identical inputs (Qwen3-VL-8B-Instruct, Pi3, SAM3, 32 frames) on the full VSI-Bench:

ConfigObj. CountAbs. Dist.Obj. SizeRoom SizeRel. Dist.Rel. Dir.Route PlanAppr. OrderTotal (%)Tool Calls
A0 (AlloSpatial)38.9233.5747.3035.7053.9741.6833.7553.9342.352.24
A1 (+DeSpatial Percept.)63.1941.0360.0038.4952.8642.8638.9252.8648.782.01
A2 (+3D OBB Repr.)63.2449.8662.8656.5154.4441.1039.5753.6252.651.85
A3 (Full DeSpatial)65.4355.0866.8673.5867.9382.5364.4377.9069.221.75

Replacing AlloSpatial's perception alone (A0 → A1) already improves overall accuracy by +6.43 points (e.g., Object Counting 38.92 → 63.19, Object Size 47.30 → 60.00), while switching to 3D OBBs and OBB-based query tools (A2 → A3) provides the interface needed to expose the recovered 3D geometry for relational and directional queries (Rel. Dir. 41.10 → 82.53), reaching 69.22% (+26.87 points over matched AlloSpatial) with fewer tool calls (1.75 vs. 2.24). The results are also updated in the paper.


W3. Model details, object orientation, and threshold selection

  • Models & orientation: Both the reasoning agent and the visual verifier use Qwen3-VL-8B-Instruct; 3D reconstruction uses Pi3; 2D segmentation uses SAM3; and gravity calibration uses GeoCalib. For VSI-Bench directional queries, we do not estimate intrinsic object orientations; instead, observer reference frames are computed from the observer-to-target vector (a → b) and the gravity-aligned up vector u (Appendix B.4).
  • Threshold selection: All Appendix B.2 and B.3 hyperparameters (τ_c = 0.08, τ_d = 0.02, λ = 5, τ_ov = 0.1, and 0.10 m voxel size) were tuned during initial development solely on 10 fixed candidate scenes and their corresponding questions, and then held strictly fixed across all remaining evaluation scenes and benchmarks.

AuthorsOpenReview ↗

Response to dy1s Part 2

W4. Comparison with OVIR-3D, MaskClustering, and Panoptic Lifting

We added a discussion in Section 5 (Related Work) and evaluated adapted OVIR-3D and MaskClustering as perception backends under the exact same Pi3, SAM3, Qwen3-VL-8B-Instruct, and 32-frame inputs on full VSI-Bench (Appendix E.3, Table 13):

Perception BackendObj. CountAbs. Dist.Obj. SizeRoom SizeRel. Dist.Rel. Dir.Route PlanAppr. OrderVSI-Bench Total (%)
OVIR-3D42.8346.4355.6363.9753.9060.2156.2563.0255.28
MaskClustering54.0250.6359.4160.3756.9775.7859.5668.0560.60
DeSpatial (Ours)65.4355.0866.8673.5867.9382.5364.4377.9069.22
  • OVIR-3D incrementally fuses RGB-D masks via 3D overlap and frozen textual region features without pre-lifting depth–mask co-refinement or group-level same-frame exclusion (55.28% vs. 69.22%).
  • MaskClustering relies on view consensus requiring candidate masks to co-occur inside a shared enclosing mask in another view; under sparse 32-frame sampling, disjoint partial observations that never share a parent mask remain fragmented (60.60% vs. 69.22%).
  • Panoptic Lifting requires hours of per-scene neural field optimization (~2 hours per scene on our GPUs), making per-scene optimization across hundreds of scenes impractical for an online feed-forward spatial agent. Conceptually, our group-level same-frame exclusion plays an analogous conflict-prevention role to its frame-level assignment, but runs feed-forward on sparse monocular reconstructions.

We also directly compared their 3D geometric accuracy under matched inputs (Appendix E.3, Table 12), where DeSpatial achieves substantially lower count error (Count MAE 0.698 vs. 1.229 for OVIR-3D and 0.893 for MaskClustering) and higher 3D OBB accuracy ([email protected] 60.38 vs. 42.58 and 52.25), confirming that its downstream QA gains stem from cleaner 3D instance geometry.


W5. Runtime, tool-call counts, and direct 3D perception evaluation

  • Runtime & tool calls (full VSI-Bench, 4 GPUs, 32 frames; Appendix E.1, Tables 9–10): DeSpatial takes 211.13 min total (62.41 min Main VLM + 142.99 min Tool Execution, plus ~5.73 min orchestration/I/O) with 1.75 tool calls per question, compared to AlloSpatial's 179.40 min (95.82 min Main VLM + 68.60 min Tool Execution) and 2.24 tool calls.
  • Direct 3D perception & cross-view matching evaluation (184 scenes, 1,016 scene-class cells; Appendix E.2, Table 11): Moving from raw geometric fusion (B0) through duplicate-mask consolidation (B1), same-frame disjoint-part consolidation (B2), and cross-frame visual fragment association (B3, full DeSpatial) steadily improves object counts, 3D OBB accuracy, and cross-view identity consistency:
ConfigCount MAE ↓Fragmented Identities ↓False-merged Groups[email protected][email protected][email protected]
B0 (Raw fusion)0.878097 / 9705 / 110347.8564.1654.82
B1 (+Dup. consolidation)0.824877 / 9706 / 107349.3264.1655.77
B2 (+Same-frame part)0.779569 / 9706 / 106050.1764.4356.41
B3 (+Cross-frame assoc.)0.697843 / 9709 / 102856.1565.3060.38


Thank you for your detailed response. I appreciate the authors' effort to address my concerns. However, I still have a few concerns.

  1. Your matched-backbone evaluation is still very limited to AlloSpatial. GCA is a very strong contender for your method and should be considered with matched backbones.
  2. Even in your matched ablation (new Table 9), the gain from A2 →\rightarrow A3 with the change in the interface is larger than the gains coming from A0 →\rightarrow A1, which is the perception. This suggests a substantial contribution from the reasoning tools and associated prompts. This undermines the paper's claim that perception is the "devil" in 3D spatial reasoning.
  3. DeSpatial’s hyperparameters were tuned on 10 scenes. Did you tune the OVIR-3D and MaskClustering hyperparameters on the same subset with a comparable tuning budget?
Reviewer_pEsXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DeSpatial constructs a shared 3D scene for spatial question answering through geometric filtering, mask consolidation, constrained voxel fusion, and visual fragment association. Spatial tools then query the resulting instances. Evaluation includes an oracle diagnostic, multiple benchmarks, and component ablations.

Critical strengths
  1. The pipeline addresses concrete errors in geometry and object identity. Table 5 reports an improvement from 57.56 to 68.83 with co-refinement, and gains of 2.52 and 2.01 points from exclusion and fragment association.
  2. Experiments cover object measurements, viewpoint changes, and long videos. Appendix A clearly specifies the oracle intervention and retains all 4,030 questions.
Critical weaknesses
  1. The incremental contribution over existing object-centric mapping is insufficiently established. AlloSpatial already uses confidence/density filtering, adaptive DBSCAN, structured spatial maps, and geometry–semantic arbitration (Ruan et al., 2026, §3.2–3.3). DeSpatial adds part-aware retention, group-level exclusion, and visual fragment verification, but Table 5's DBSCAN row does not reproduce that pipeline or sufficiently specify baseline tuning. Table 2 also compares full-benchmark results with AlloSpatial's subset results. Matched backend comparisons are needed to establish the value of these additions.
  2. The proposed error reductions lack direct validation. Table 5 provides downstream averages without false-merge, fragmentation, box-error, or judge-error measurements, or paired scene-level uncertainty. Key implementation details are also missing: facing/observer estimation (Appendix B.4), judge prompts/settings, iteration limits, and long-video sampling/cache policies.
Questions for authors
  1. With matched inputs, perception models, reasoner, and comparable budgets, do the gains persist against AlloSpatial-style processing or tuned geometry-plus-appearance association? How were baseline parameters selected?
  2. How do the modules affect instance recall, false merges, fragmentation, and box error on annotated scenes? What are the paired scene-level intervals for their gains?
  3. If possible, please specify the missing estimators, judge configuration, iteration limits, long-video policy, and actual tool-call counts.
5 comments
AuthorsOpenReview ↗

Response to pEsX Part 1

We thank Reviewer pEsX for the constructive feedback and for recognizing the diagnostic study and ablation results. To address your questions directly, we include the full quantitative tables and implementation details below (also incorporated in blue in Appendix E of the revised manuscript).

Q1. Matched-input comparisons against AlloSpatial and geometry-plus-appearance baselines

We ran controlled comparisons on the full VSI-Bench using the exact same VLM backbone (Qwen3-VL-8B-Instruct), reconstruction model (Pi3), segmentation model (SAM3), and 32-frame input budget.

(a) Progressive replacement from AlloSpatial to DeSpatial

We progressively replace AlloSpatial's components (A0 → A3):

  • A0: Original AlloSpatial (world2mind perception + 2D top-view/height-range representation + AlloSpatial reasoning tools).
  • A1: Replace only world2mind with DeSpatial's perception module (keeping the 2D grid representation and AlloSpatial's tools unchanged).
  • A2: Replace the 2D grid representation with DeSpatial's 3D OBB representation (keeping AlloSpatial's tools).
  • A3: Full DeSpatial (DeSpatial perception + 3D OBB representation + 3D OBB reasoning tools).
ConfigObj. CountAbs. Dist.Obj. SizeRoom SizeRel. Dist.Rel. Dir.Route PlanAppr. OrderTotal (%)Tool CallsTime (min)
A0 (AlloSpatial)38.9233.5747.3035.7053.9741.6833.7553.9342.352.24179.40
A1 (+DeSpatial Percept.)63.1941.0360.0038.4952.8642.8638.9252.8648.782.01275.96
A2 (+3D OBB Repr.)63.2449.8662.8656.5154.4441.1039.5753.6252.651.85244.19
A3 (Full DeSpatial)65.4355.0866.8673.5867.9382.5364.4377.9069.221.75211.13

Replacing AlloSpatial's perception alone (A0 → A1) already improves overall accuracy by +6.43 points (e.g., Object Counting 38.92 → 63.19, Object Size 47.30 → 60.00). While DeSpatial's 3D perception already recovers the object locations and orientations needed for relational and directional queries, AlloSpatial's 2D-grid representation and point-cloud-viewing tools cannot directly read out these 3D geometric relations; switching to 3D OBBs and OBB-based query tools (A2 → A3) provides the necessary interface to expose that recovered 3D information (Rel. Dir. 41.10 → 82.53) with fewer tool calls (1.75 vs. 2.24).

(b) Matched geometry-plus-appearance 3D instance backends

We also adapted OVIR-3D (voxel/3D overlap + frozen region-level appearance similarity) and MaskClustering (view-consensus mask graph) by replacing only DeSpatial's perception module (A3) while keeping the exact same Pi3 and SAM3 inputs, 3D OBB representation, reasoning tools, Qwen3-VL-8B-Instruct backbone, and 32-frame budget unchanged, and evaluated them on 184 scenes (1,016 scene-class cells) and full VSI-Bench:

Perception BackendCount MAE ↓[email protected][email protected][email protected]Obj. CountAbs. Dist.Obj. SizeRoom SizeRel. Dist.Rel. Dir.Route PlanAppr. OrderTotal (%)
OVIR-3D1.22933.5058.4142.5842.8346.4355.6363.9753.9060.2156.2563.0255.28
MaskClustering0.89351.9552.5652.2554.0250.6359.4160.3756.9775.7859.5668.0560.60
DeSpatial (Ours)0.69856.1565.3060.3865.4355.0866.8673.5867.9382.5364.4377.9069.22
  • Baseline parameter selection: AlloSpatial (A0), OVIR-3D, and MaskClustering all receive the exact same Pi3 and SAM3 observations as DeSpatial and use their official default hyperparameters without modification. For the DBSCAN row in Table 5(b), we used standard settings (eps=0.15 m, min_samples=20) as a within-pipeline reference, while A0 in the matched table above evaluates the full official AlloSpatial pipeline end-to-end.

AuthorsOpenReview ↗

Response to pEsX Part 2

Q2. Direct perception metrics and paired scene-level intervals

We directly evaluate 3D perception and instance identity across 184 scenes (1,016 scene-class cells) under identical Pi3, SAM3, and GeoCalib inputs:

  • B0: Raw geometric fusion pipeline
  • B1: B0 + duplicate-mask consolidation
  • B2: B1 + same-frame disjoint-part consolidation
  • B3: B2 + cross-frame visual fragment association (Full DeSpatial)
ConfigCount MAE ↓Fragmented Identities ↓False-merged Groups[email protected][email protected][email protected]
B0 (Raw fusion)0.878097 / 9705 / 110347.8564.1654.82
B1 (+Dup. consolidation)0.824877 / 9706 / 107349.3264.1655.77
B2 (+Same-frame part)0.779569 / 9706 / 106050.1764.4356.41
B3 (+Cross-frame assoc.)0.697843 / 9709 / 102856.1565.3060.38

Each stage reduces Count MAE (0.8780 → 0.6978) and cuts fragmented identities by 55.7% (97/970 → 43/970), while keeping false merges below 0.9% (9/1028) and improving 3D OBB [email protected] from 54.82 to 60.38.

  • Scene-level 95% confidence intervals ([email protected]): Across the evaluated scenes, both the lower and upper bounds of the 95% CIs for 3D OBB [email protected] shift monotonically upward from B0 [49.67, 60.28] to B1 [50.28, 61.60], B2 [50.91, 62.32], and B3 [54.80, 66.16] (raising the lower bound by +5.13 and upper bound by +5.88 overall), showing consistent scene-level improvements across refinement stages.

AuthorsOpenReview ↗

Response to pEsX Part 3

Q3. Implementation details, prompts, budgets, and tool-call counts

  • Facing & observer estimation (Appendix B.4): Thank you for pointing out the potentially misleading wording in Appendix B.4. We have revised the sentence "If a query requires object facing or an observer pose, these attributes are estimated through perception and supplied to the corresponding computation." to "When a question specifies an observer location and a facing target (e.g., standing at object A facing object B) or a camera viewpoint, the observer position a and facing target position b are taken directly from the reconstructed 3D instance centers (or reconstructed camera poses) produced by our perception tool, without a separate object-orientation estimator." The coordinate frame construction remains as detailed in Appendix B.4.
  • Judge configuration & prompts: The visual verifier also uses Qwen3-VL-8B-Instruct and averages 16.5 calls per scene on VSI-Bench. Both same-frame and cross-frame verifiers receive the highlighted source-view crops (plus oblique and top-down gravity-aligned 3D fragment renderings with camera wireframes for cross-frame), with no question text, answer choices, or ground-truth counts, and return structured JSON ({"decision":"same|different|unclear", "evidence":"..."}). Full system prompts, user prompts and visualizations are updated in Appendix E.4.
  • Tool-call counts, iteration limits & long-video policy: The agent is capped at a maximum of 3 iterations (turns) per question, and on full VSI-Bench DeSpatial averages 1.75 tool calls per question (vs. 2.24 for AlloSpatial). On VSI-SUPER, following S-Agent, we use an adaptive long-video access policy rather than a fixed 32-frame sample: (1) Overview: uniformly sample 32 frames across the full video for the main VLM to parse the question and select target classes; (2) Full-video scan: run LocateAnything at 1 FPS across the entire timeline; (3) Local densification: expand ±2s around each hit, merge overlapping windows, and sample at 4 FPS within those windows for further detection. Per-duration VSI-SUPER latency and visual-judge call counts are:
Video DurationRecall Time (s/Q)Count Time (s/Q)Recall Verifier Calls / QCount Verifier Calls / Q
10 min102.92137.5284.31108.60
30 min290.41384.49179.52233.93
60 min550.87633.13343.09397.58
120 min1,109.971,224.71656.42752.77
240 min2,289.71—1,169.20—
Overall Avg.868.77594.96——

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.