- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces SpatialWeave, which evolves tools, skills and workflows in a harness for solving spatial reasoning questions. Once the harness has evolved, it can be removed during inference, leading to both efficient and performant spatial reasoning systems, as demonstrated on multiple base models and benchmarks.
Critical strengths
(1) The motivation of dynamically optimizing the tools and workflows of VLM tool-use systems, which have been largely reliant on static procedures, is intuitive and insightful.
(2) The performance of SpatialWeave looks strong across diverse base models and benchmarks. And by splitting harness evolution from inference, SpatialWeave also becomes very efficient at inference, retaining only the needed skills.
Critical weaknesses
(1) The distinction between SpatialWeave and the related work on evolving skills or programs in Sec. 2 (last paragraph) remains unclear to me. At the end of the day, the so-called "one harness" of skills, tools and workflows is still just code and a program. We might as well call it a "tool-calling program" (and a "skill" is really just another orchestration of tools and code). Even if the "harness" is removed during inference (which is more efficient and admittedly a benefit), that’s very likely just because the tools and skills have been updated and consolidated during optimization, which is common in related work on tool use.
(2) The writing throughout this paper (especially the method description in Sec. 3, which should have been very clear) is full of jargon and very hard to understand.
Questions for authors
(1) Beyond all the jargon, how does the method differ at its core from the related work in Sec. 2?
(2) What perception and reconstruction errors (Conclusion, last sentence) remain unsolved? Since the method optimizes around so many available tools and skills, which of these errors are addressed and which remain? An analysis and qualitative examples of this would be very valuable for understanding the method in depth.
2 comments
Response to Reviewer Sxis (2/2): Plain Walkthrough & Error Audit (Q3, Q4)
Q3: Rewritten Section 3 Structure & Plain-English Walkthrough (§3, pp. 4–6; Figures 2–3)
We reorganized Section 3 (pp. 4–6) into three direct subsections without dense notation:
§3.1 Explore-to-Compile Lifecycle (p. 4, Figure 2a): Defines base vision tools, single-call Python workflows, and short Markdown guides (Tools + Lessons), and shows how offline exploration reveals repeated tool chains to compile before retiring the playbook.§3.2 Coupled Co-Revision & Validation (p. 5, Figure 2b): Shows how failure clusters yield paired(Python code, check, Markdown guide)updates screened on held-aside dev items before freeze.§3.3 Self-Repairing Inference (pp. 5–6, Figure 3): Shows how the frozen VLM calls tools selectively (~28%answered directly with0tool calls) and recovers when a check fires.
Plain-English walkthrough (Appendix F.2, Case 2, Figure 14, pp. 28–29; Figure 3b, p. 6; GT = 0.30 m):
- Question: "Minimum distance in meters between the door and wall telephone?" across 8 room views.
- Perception error: GroundingDINO's
"door"box in frame 7 bleeds onto the wall by the phone, yielding a false0.06 mdistance. - Python check blocks bad float: Seeing closest-point (
0.06 m) collapse against center distance (0.93 m), code clearsvalue = Noneand returnsretry-grounding(value=Noneadds+11.4 pp, , over text warnings alone;§4.5, p. 10). - Guided visual repair: The paired Tools guide directs the VLM to crop and re-query facing edges (
door facing edgevs.telephone cord), returning0.29 m(status="ok", score1.0)—vs.0.10 m(0.0) bare,0.21 m(0.5) unguarded, and0.0SpatialClaw.
Q4: Quantitative & Qualitative Analysis of Solved vs. Unsolved Errors (§4.5 & Table 4, p. 10; Appendix F, pp. 27–32)
Table 4 (p. 10) and Appendix F (Figures 13–19, pp. 27–32) audit every triggered question and report intermediate 3D accuracy and limits:
- Full cohort & intermediate 3D repair (
§4.5, p. 10;Appendix F.3, p. 29): On the exact same1984-benchmark items flagged by , paired skills add+8.23 ppover identical code without skills (89.65%vs.81.41%;+13.75 ppSPBench;+0.78 ppacross all4,929items, ). Across187retry-groundingitems (100%acted), achieves290->1repairs vs.51->0regressions over (+12.83 pp, ;20:3vs. -tools). Intermediate 3D readouts directly improve: crop re-grounding stabilizes90/171MindCube triggers tostatus="ok"at96.7%(87/90) GT sector accuracy, and lifts SPBench intermediate distance MRA from21.05%to33.16%(+12.11 pp) while cutting<0.10 mcollapse2.8×. - Self-contained solved vs. unsolved cases (
Appendix F, pp. 27–32):- Solved: (1) Camera turn vs. path (
Appendix F.1, Figure 13, p. 27;Figure 15, p. 30): camera moves right around a bottle (+71.8°bearing) while turning left (-42.3°yaw); bare VLM answers"left"(0.0), whilecamera_motionseparates path ("right") from yaw (1.0). (2) Mixed-surface depth & box collapse (Appendix F.3, Figure 16a–b, p. 31): a coarse"table"box mixing background (mode 0.49 < 0.50) triggers crop re-anchoring on cleantabletop(1.27 m) vs.door panel(2.14 m), flippingright-back/left-back(0.0) toright-front(1.0); on chair-to-trash-can vs. window (Figure 16b), clearing collapsed box distances (<0.4×center) and querying facing-edge pixels recovers0.24 m < 0.47 m(1.0vs.0.0no-skills). (3) Cross-view occlusion &<15 cmdepth ties (Appendix F.3, Figure 17a–b, p. 31;Appendix D, Figure 12b, p. 24): rejecting0.00 moccluded printer–table collapse in view 8 re-anchors view 5 at1.39 mvs.1.30 mGT (MRA 0.9vs.0.19 m / 0.0no-skills), and gating a0.13 mkeyboard–tower depth tie flipsfront(0.0) toback(1.0). - Partial / Unsolved (
Appendix F.3, Figure 19a–b, p. 32): (4) Close contact + monocular depth drift (Figure 19a): touching laptop–lamp boxes (0.02 mvs.0.88 mcenters) re-anchor via edge pixels from0.06 m(0.0no-skills) to0.21 mvs.0.30 mGT, but single-view depth drift (z = 1.94 / 2.08 m) caps score at partial0.5; (5) Heavy multi-view occlusion (Figure 19b): when target surfaces stay hidden across views even after cropping, recovery still scores0.0(18/187acted items on ;35/124on ).
- Solved: (1) Camera turn vs. path (
Response to Reviewer Sxis (1/2): Core Distinction & Counterfactuals (Q1, Q2)
We thank Reviewer
Sxisfor recognizing the motivation behind dynamic tool/workflow optimization and our consistent gains across models and benchmarks—strengths also highlighted by Reviewersfrwc(clean modular decomposition and 7-benchmark breadth) andkvbd(offline-explore / deployment-retire efficiency). Taking your feedback on terminology density seriously, we rewrote§1 (pp. 1–3)and§3 (pp. 4–6)in plain, concrete English around visual examples (Figures 1–3) and added direct counterfactual controls in the revised PDF (changes in blue; full changelog inFully Revised Draft).Q1: Stripped of Jargon, How Does SpatialWeave Differ from §2 Prior Work? (
§1–§2, pp. 1–3;Table 1, p. 7)Prior spatial VLM methods improve only one side of the boundary between the VLM and its 3D vision tools:
pySpatial,SpatialClaw) hand the VLM static 3D tools. When a detector draws a sloppy box that bleeds onto the wall or a 3D reconstructor confuses camera rotation with movement, the tool returns a raw number silently—neither blocking the bad number nor telling the VLM how to fix its visual input.Meta-Harness, skill-only search) rewrite instructions, but text prompts cannot run 3D coordinate math in Python or block corrupted floats at runtime (on GPT-5.6 Sol, tool-freeMeta-Harnessscores72.13%vs. SpatialWeave's80.75%,Table 1a, p. 7).SpatialWeave updates both sides together in a closed loop: from development failures, it compiles repeated multi-tool chains into single-call Python functions, adds code checks that clear unreliable numbers (
value=None,status="retry-grounding"), and writes matching Markdown guides telling the VLM how to re-crop or switch views when a check fires. This unlocks concrete capability jumps (§1, p. 2;§4.2, pp. 7–8):5.83% -> 90.00%, 4-view camera-frame left/right jumps52.22% -> 88.89%, and turn-then-move bearing jumps69.44% -> 89.81%.Table 1b–c, p. 7): On369unseen ScanNet 3D scan questions, seed tools (H0) add only+1.60 pp(p=0.26) over bare GPT-6.1 Sol, whereas co-evolution (H0→H2) adds+8.21 ± 1.78 pp(p=5.5×10−6,+52.54 ppon unseen-scan viewpoint)—and gains+20.45 pp/+12.89 ppzero-shot across807questions from unseenViewSpatialroom families.Q2: Why Combining Code, Workflows, & Skills—and Retiring the Playbook—Is Non-Trivial (
§4.3–4.4, pp. 8–10)Three controlled counterfactual experiments prove that single-sided or naive integration actively fails:
Table 2a, p. 8;Appendix H.2, Figure 20c, p. 37): In Round 1 of our 6-round zero-human run (H6.1), H1 compiled 3 verified Python workflows without a matching skill update (Table 16, row r1/c5, p. 38). On the4,929held-out test questions, accuracy dropped-1.11 ppbelow seed H0 (81.74% -> 80.63%, p=0.0017) because the VLM misread coordinate frames and status flags. On the160-item target slice (Figure 20c, p. 37), adding the new Python code+check alone dropped accuracy90.00% -> 88.75%, whereas pairing that exact code with its matching skill guide jumped to98.12%(lifting full4,929-item accuracy by+4.68 ppto85.31%).Appendix H.2, Table 17a, p. 38): From the identical 11-tool seed (74.08%,N=620dev items) under exact970episodes per slot, joint code+skill revision (+6.83 ± 1.17 pp) significantly beats standalone skill-only revision (+4.98 ± 1.27 pp, p=0.0255) and code-compile-only revision (+4.26 ± 1.23 pp, p=0.0032).§4.4, p. 9;Table 18, p. 39): Running 3 rounds of evolution without the offline playbook leaves the VLM skipping tools on57%–70%of MindCube/BLINK questions, yielding0surviving workflows and+0.00 pptest gain (81.85% -> 81.85%). Once workflows are compiled into Python, retiring the playbook cuts test tokens by40.9%(18.59k -> 10.99k/q) and tool calls by65.6%(3.85 -> 1.32/q) at+0.70 pphigher accuracy (Figure 4, p. 9).