- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DeSpatial builds a shared metric, gravity-aligned 3D scene for a spatial reasoning agent. It cleans segmentation and depth inconsistencies, fuses object observations across views with same-frame exclusion, and uses visual judgments to associate disjoint fragments. The resulting scene is queried by spatial computation tools and evaluated on four benchmarks.
Critical strengths
-
The method addresses concrete sources of error in tool-based spatial reasoning. Same-frame exclusion and part-aware filtering have clear purposes, and Table 5 shows an 11.27-point improvement from the full co-refinement pipeline over raw lifting on the fixed subset.
-
Scores of 69.22 on VSI-Bench, 87.19 on MindCube-Tiny, and 68.35 on ViewSpatial show benefits across object measurements and viewpoint changes. The oracle study motivates improving the scene representation, with its annotation-related limits acknowledged. The controlled component gains and breadth of evaluation support weak acceptance.
Critical weaknesses
-
Main-table comparisons mix backbones, perception tools, and in some cases evaluation subsets. The 504-question ablation establishes a within-pipeline benefit, but broader matched comparisons would clarify its size relative to prior agents. This limits the precision of the comparative claim.
-
Direct measurements of instance association and geometric error are missing, as are end-to-end cost and scaling results. These would make the observed accuracy gains easier to explain and the practical tradeoff easier to judge, particularly for long videos with repeated visual identity judgments.
-
Fusion uses fixed 0.1 m voxels and a hard same-frame exclusion rule. If mask consolidation leaves one object split within a frame, later fusion cannot repair that identity. Sensitivity to voxel size, scale error, and these upstream mistakes is not quantified, so the robustness of the association rules is uncertain.
Questions for authors
-
Under the same Qwen3-VL-8B backbone, reconstruction inputs, and frame budget, how does DeSpatial compare with the strongest prior scene-building pipeline?
-
Can you report direct instance-fusion and object-geometry metrics before and after refinement, including false merges and fragmented identities? These would test the proposed mechanism independently of answer accuracy.
-
What are the end-to-end latency, visual-judge call count, and frame-sampling policy on VSI-Bench versus VSI-SUPER? This would clarify whether the long-video gains require substantially more inference work.
3 comments
Response to LrXu Part 2
Q3. End-to-end latency, visual-judge calls, and frame-sampling policy (VSI-Bench vs. VSI-SUPER)
- VSI-Bench latency (4 GPUs, 32 frames, full benchmark):
| Config | End-to-End Time (min) | Main VLM (min) | Tool Execution (min) | Avg. Tool Calls / Q |
|---|---|---|---|---|
| A0 (AlloSpatial) | 179.40 | 95.82 | 68.60 | 2.24 |
| A1 (+DeSpatial Percept.) | 275.96 | 98.99 | 160.05 | 2.01 |
| A2 (+3D OBB Repr.) | 244.19 | 86.11 | 149.46 | 1.85 |
| A3 (Full DeSpatial) | 211.13 | 62.41 | 142.99 | 1.75 |
While 3D scene construction takes longer than AlloSpatial's 2D pipeline (142.99 vs. 68.60 min), structured 3D OBB tools substantially reduce Main VLM inference time (62.41 vs. 95.82 min) because the agent no longer needs repeated point-cloud viewing steps.
- Visual-judge call count & VSI-SUPER sampling/latency: On VSI-Bench (32 frames), the visual verifier (
Qwen3-VL-8B-Instruct) averages 16.5 calls per scene. On VSI-SUPER, following S-Agent, we use an adaptive long-video access pipeline instead of a fixed 32-frame sample: (1) Overview (32 uniform frames for target selection), (2) Full-video scan (LocateAnythingat 1 FPS), and (3) Local densification (±2s around each hit, merged windows sampled at 4 FPS). Across video durations from 10 to 240 minutes, per-question runtime and verifier call counts scale roughly linearly with video length:
| Video Duration | Recall Time (s/Q) | Count Time (s/Q) | Recall Verifier Calls / Q | Count Verifier Calls / Q |
|---|---|---|---|---|
| 10 min | 102.92 | 137.52 | 84.31 | 108.60 |
| 30 min | 290.41 | 384.49 | 179.52 | 233.93 |
| 60 min | 550.87 | 633.13 | 343.09 | 397.58 |
| 120 min | 1,109.97 | 1,224.71 | 656.42 | 752.77 |
| 240 min | 2,289.71 | — | 1,169.20 | — |
| Overall Avg. | 868.77 | 594.96 | — | — |
W3. Sensitivity to voxel size and same-frame exclusion
- Voxel size sensitivity (184 scenes & full VSI-Bench):
| Voxel (m) | Count MAE ↓ | [email protected] | [email protected] | [email protected] | VSI-Bench Total (%) |
|---|---|---|---|---|---|
| 0.025 | 0.9350 | 50.70 | 64.16 | 56.64 | 59.51 |
| 0.05 | 0.7795 | 53.16 | 64.58 | 58.32 | 65.11 |
| 0.10 (default) | 0.6978 | 56.15 | 65.30 | 60.38 | 69.22 |
| 0.15 | 0.6883 | 55.34 | 64.52 | 59.58 | 68.50 |
| 0.20 | 0.6854 | 55.51 | 64.60 | 59.71 | 68.66 |
Too-small voxels (0.025 m) under-merge slightly separated parts (59.51%), whereas performance across 0.10–0.20 m is stable (68.50%–69.22%) because same-frame exclusion prevents nearby objects from collapsing even at coarser voxel resolutions.
- Trade-off and quantitative impact of hard same-frame exclusion: If an object remains split within a single frame after consolidation, the hard same-frame exclusion rule indeed prevents merging those two groups later. We retain this hard constraint intentionally to keep false merges very low even at coarser voxel sizes (only
9/1028, or<0.9%, inB3). Quantitatively (Q2above andAppendix E.2, Table 11), the within-frame consolidation stages (B0→B2) already reduce fragmented identities from97/970to69/970, and cross-frame fragment association (B2→B3) further links disjoint cross-view observations down to43/970(a 55.7% overall reduction) while raising OBB F1 from56.41to60.38.
Thanks for the rebuttal. The rebuttal addresses my main concerns. I have no further questions and will keep my original rating.
Response to LrXu Part 1
We thank Reviewer LrXu for the positive evaluation and thoughtful questions on matched comparisons, direct geometric validation, runtime, and robustness.
Q1. Matched comparison with prior scene-building pipelines under identical inputs
Under the exact same Qwen3-VL-8B-Instruct backbone, Pi3 + SAM3 inputs, and 32-frame budget on full VSI-Bench, we compare DeSpatial against matched prior scene-building pipelines (
Appendix E.1, Tables 8–9andAppendix E.3, Table 13):A0(AlloSpatial): Full AlloSpatial pipeline (world2mindperception + 2D top-view/height-range representation + AlloSpatial reasoning tools).A1(+DeSpatial Percept.): Replaces onlyworld2mindwith DeSpatial's perception module (keeping the 2D grid representation and AlloSpatial's tools).A2(+3D OBB Repr.): Replaces the 2D grid with DeSpatial's 3D OBB representation (keeping AlloSpatial's tools).A3) with adapted OVIR-3D or MaskClustering under the exact same Pi3 + SAM3 inputs, while keeping our 3D OBB representation and reasoning tools unchanged.A3(Full DeSpatial): DeSpatial perception + 3D OBB representation + 3D OBB reasoning tools.A0, matched)A1)A2)A3)Replacing AlloSpatial's perception alone (
A0→A1) yields +6.43 points, while full DeSpatial (A3) outperforms matched AlloSpatial by +26.87 points, adapted OVIR-3D by +13.94 points, and adapted MaskClustering by +8.62 points.Q2. Direct instance-fusion and object-geometry metrics before and after refinement
Evaluating 184 scenes (1,016 scene-class cells) directly under shared Pi3, SAM3, and GeoCalib inputs shows steady geometric gains from each stage (
B0raw fusion →B1+duplicate consolidation →B2+same-frame part consolidation →B3+cross-frame fragment association):