acceptodds
Withdrawn from ICLR 2027

SpatialWeave: Co-Evolving Tools, Workflows, and Skills for Spatial Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Tool-augmented vision-language models can perceive complex spatial environments, but they struggle to calculate the exact answers that questions require. Existing methods either recalculate everything from scratch for each question and repeat the same mistakes, or they use fixed steps that fail on new problems. We introduce SpatialWeave, which improves spatial reasoning without modifying the underlying language model. Instead, it automatically optimizes the external tools, workflows, and usage rules the model relies on. During evolution, a temporary procedure scaffold guides the model to explore various tool combinations. Successful tool sequences are saved as reusable workflows, and common mistakes are corrected by adding specific skills. After evolution, this scaffold is completely removed. Across seven spatial reasoning benchmarks, SpatialWeave consistently improves performance on three different models. Notably, it raises Qwen3.8-27B to 80.73%, beating the base GPT-6 Astra model at 80.28%. It also improves Gemma-4-31B to 73.58%, outperforming the specialized SpatialClaw agent on the same model. Furthermore, removing the scaffold during inference makes the process much more efficient. It reduces token generation by up to 34.9% and tool calls by up to 46.0% without losing accuracy. Ultimately, SpatialWeave shows that automatically building a lightweight set of external tools and rules is a practical way to achieve strong spatial reasoning while keeping inference fast and simple.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
spatial reasoning, tool-augmented agents, vision-language models, harness co-evolution, executable workflows, procedural skills

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SxisOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces SpatialWeave, which evolves tools, skills and workflows in a harness for solving spatial reasoning questions. Once the harness has evolved, it can be removed during inference, leading to both efficient and performant spatial reasoning systems, as demonstrated on multiple base models and benchmarks.

Critical strengths

(1) The motivation of dynamically optimizing the tools and workflows of VLM tool-use systems, which have been largely reliant on static procedures, is intuitive and insightful.

(2) The performance of SpatialWeave looks strong across diverse base models and benchmarks. And by splitting harness evolution from inference, SpatialWeave also becomes very efficient at inference, retaining only the needed skills.

Critical weaknesses

(1) The distinction between SpatialWeave and the related work on evolving skills or programs in Sec. 2 (last paragraph) remains unclear to me. At the end of the day, the so-called "one harness" of skills, tools and workflows is still just code and a program. We might as well call it a "tool-calling program" (and a "skill" is really just another orchestration of tools and code). Even if the "harness" is removed during inference (which is more efficient and admittedly a benefit), that’s very likely just because the tools and skills have been updated and consolidated during optimization, which is common in related work on tool use.

(2) The writing throughout this paper (especially the method description in Sec. 3, which should have been very clear) is full of jargon and very hard to understand.

Questions for authors

(1) Beyond all the jargon, how does the method differ at its core from the related work in Sec. 2?

(2) What perception and reconstruction errors (Conclusion, last sentence) remain unsolved? Since the method optimizes around so many available tools and skills, which of these errors are addressed and which remain? An analysis and qualitative examples of this would be very valuable for understanding the method in depth.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Sxis (1/2): Core Distinction & Counterfactuals (Q1, Q2)

We thank Reviewer Sxis for recognizing the motivation behind dynamic tool/workflow optimization and our consistent gains across models and benchmarks—strengths also highlighted by Reviewers frwc (clean modular decomposition and 7-benchmark breadth) and kvbd (offline-explore / deployment-retire efficiency). Taking your feedback on terminology density seriously, we rewrote §1 (pp. 1–3) and §3 (pp. 4–6) in plain, concrete English around visual examples (Figures 1–3) and added direct counterfactual controls in the revised PDF (changes in blue; full changelog in Fully Revised Draft).

Q1: Stripped of Jargon, How Does SpatialWeave Differ from §2 Prior Work? (§1–§2, pp. 1–3; Table 1, p. 7)

Prior spatial VLM methods improve only one side of the boundary between the VLM and its 3D vision tools:

  • Fixed-tool agents (pySpatial, SpatialClaw) hand the VLM static 3D tools. When a detector draws a sloppy box that bleeds onto the wall or a 3D reconstructor confuses camera rotation with movement, the tool returns a raw number silently—neither blocking the bad number nor telling the VLM how to fix its visual input.
  • Prompt / text-only optimizers (Meta-Harness, skill-only search) rewrite instructions, but text prompts cannot run 3D coordinate math in Python or block corrupted floats at runtime (on GPT-5.6 Sol, tool-free Meta-Harness scores 72.13% vs. SpatialWeave's 80.75%, Table 1a, p. 7).

SpatialWeave updates both sides together in a closed loop: from development failures, it compiles repeated multi-tool chains into single-call Python functions, adds code checks that clear unreliable numbers (value=None, status="retry-grounding"), and writes matching Markdown guides telling the VLM how to re-crop or switch views when a check fires. This unlocks concrete capability jumps (§1, p. 2; §4.2, pp. 7–8):

  • Subtask breakthroughs (GPT-5.6 Sol): Camera-quadrant relations jump 5.83% -> 90.00%, 4-view camera-frame left/right jumps 52.22% -> 88.89%, and turn-then-move bearing jumps 69.44% -> 89.81%.
  • Unseen 3D room generalization (Table 1b–c, p. 7): On 369 unseen ScanNet 3D scan questions, seed tools (H0H_0) add only +1.60 pp (p=0.26p=0.26) over bare GPT-6.1 Sol, whereas co-evolution (H0→H2H_0 \to H_2) adds +8.21 ± 1.78 pp (p=5.5×10−6p=5.5\times 10^{-6}, +52.54 pp on unseen-scan viewpoint)—and gains +20.45 pp / +12.89 pp zero-shot across 807 questions from unseen ViewSpatial room families.

Q2: Why Combining Code, Workflows, & Skills—and Retiring the Playbook—Is Non-Trivial (§4.3–4.4, pp. 8–10)

Three controlled counterfactual experiments prove that single-sided or naive integration actively fails:

  1. Updating Python workflows without matching skill updates hurts held-out accuracy (Table 2a, p. 8; Appendix H.2, Figure 20c, p. 37): In Round 1 of our 6-round zero-human run (H6.1H^{6.1}), H1H_1 compiled 3 verified Python workflows without a matching skill update (Table 16, row r1/c5, p. 38). On the 4,929 held-out test questions, accuracy dropped -1.11 pp below seed H0H_0 (81.74% -> 80.63%, p=0.0017p=0.0017) because the VLM misread coordinate frames and status flags. On the 160-item target slice (Figure 20c, p. 37), adding the new Python code+check alone dropped accuracy 90.00% -> 88.75%, whereas pairing that exact code with its matching skill guide jumped to 98.12% (lifting full 4,929-item accuracy by +4.68 pp to 85.31%).
  2. Joint search beats equal-budget single-sided search (Appendix H.2, Table 17a, p. 38): From the identical 11-tool seed (74.08%, N=620 dev items) under exact 970 episodes per slot, joint code+skill revision (+6.83 ± 1.17 pp) significantly beats standalone skill-only revision (+4.98 ± 1.27 pp, p=0.0255p=0.0255) and code-compile-only revision (+4.26 ± 1.23 pp, p=0.0032p=0.0032).
  3. The exploration playbook is indispensable offline yet wasteful at test time (§4.4, p. 9; Table 18, p. 39): Running 3 rounds of evolution without the offline playbook leaves the VLM skipping tools on 57%–70% of MindCube/BLINK questions, yielding 0 surviving workflows and +0.00 pp test gain (81.85% -> 81.85%). Once workflows are compiled into Python, retiring the playbook cuts test tokens by 40.9% (18.59k -> 10.99k/q) and tool calls by 65.6% (3.85 -> 1.32/q) at +0.70 pp higher accuracy (Figure 4, p. 9).
AuthorsOpenReview ↗

Response to Reviewer Sxis (2/2): Plain Walkthrough & Error Audit (Q3, Q4)

Q3: Rewritten Section 3 Structure & Plain-English Walkthrough (§3, pp. 4–6; Figures 2–3)

We reorganized Section 3 (pp. 4–6) into three direct subsections without dense notation:

  • §3.1 Explore-to-Compile Lifecycle (p. 4, Figure 2a): Defines base vision tools, single-call Python workflows, and short Markdown guides (Tools + Lessons), and shows how offline exploration reveals repeated tool chains to compile before retiring the playbook.
  • §3.2 Coupled Co-Revision & Validation (p. 5, Figure 2b): Shows how failure clusters yield paired (Python code, check, Markdown guide) updates screened on held-aside dev items before freeze.
  • §3.3 Self-Repairing Inference (pp. 5–6, Figure 3): Shows how the frozen VLM calls tools selectively (~28% answered directly with 0 tool calls) and recovers when a check fires.

Plain-English walkthrough (Appendix F.2, Case 2, Figure 14, pp. 28–29; Figure 3b, p. 6; GT = 0.30 m):

  1. Question: "Minimum distance in meters between the door and wall telephone?" across 8 room views.
  2. Perception error: GroundingDINO's "door" box in frame 7 bleeds onto the wall by the phone, yielding a false 0.06 m distance.
  3. Python check blocks bad float: Seeing closest-point (0.06 m) collapse against center distance (0.93 m), code clears value = None and returns retry-grounding (value=None adds +11.4 pp, p=0.0029p=0.0029, over text warnings alone; §4.5, p. 10).
  4. Guided visual repair: The paired Tools guide directs the VLM to crop and re-query facing edges (door facing edge vs. telephone cord), returning 0.29 m (status="ok", score 1.0)—vs. 0.10 m (0.0) bare, 0.21 m (0.5) unguarded, and 0.0 SpatialClaw.

Q4: Quantitative & Qualitative Analysis of Solved vs. Unsolved Errors (§4.5 & Table 4, p. 10; Appendix F, pp. 27–32)

Table 4 (p. 10) and Appendix F (Figures 13–19, pp. 27–32) audit every triggered question and report intermediate 3D accuracy and limits:

  • Full cohort & intermediate 3D repair (§4.5, p. 10; Appendix F.3, p. 29): On the exact same 198 4-benchmark items flagged by H2H_2, paired skills add +8.23 pp over identical code without skills (89.65% vs. 81.41%; +13.75 pp SPBench; +0.78 pp across all 4,929 items, p=0.024p=0.024). Across 187 retry-grounding items (100% acted), H2H_2 achieves 29 0->1 repairs vs. 5 1->0 regressions over H0H_0 (+12.83 pp, p=3.9×10−5p=3.9\times 10^{-5}; 20:3 vs. H2H_2-tools). Intermediate 3D readouts directly improve: crop re-grounding stabilizes 90/171 MindCube triggers to status="ok" at 96.7% (87/90) GT sector accuracy, and lifts SPBench intermediate distance MRA from 21.05% to 33.16% (+12.11 pp) while cutting <0.10 m collapse 2.8×.
  • Self-contained solved vs. unsolved cases (Appendix F, pp. 27–32):
    • Solved: (1) Camera turn vs. path (Appendix F.1, Figure 13, p. 27; Figure 15, p. 30): camera moves right around a bottle (+71.8° bearing) while turning left (-42.3° yaw); bare VLM answers "left" (0.0), while camera_motion separates path ("right") from yaw (1.0). (2) Mixed-surface depth & box collapse (Appendix F.3, Figure 16a–b, p. 31): a coarse "table" box mixing background (mode 0.49 < 0.50) triggers crop re-anchoring on clean tabletop (1.27 m) vs. door panel (2.14 m), flipping right-back/left-back (0.0) to right-front (1.0); on chair-to-trash-can vs. window (Figure 16b), clearing collapsed box distances (<0.4× center) and querying facing-edge pixels recovers 0.24 m < 0.47 m (1.0 vs. 0.0 no-skills). (3) Cross-view occlusion & <15 cm depth ties (Appendix F.3, Figure 17a–b, p. 31; Appendix D, Figure 12b, p. 24): rejecting 0.00 m occluded printer–table collapse in view 8 re-anchors view 5 at 1.39 m vs. 1.30 m GT (MRA 0.9 vs. 0.19 m / 0.0 no-skills), and gating a 0.13 m keyboard–tower depth tie flips front (0.0) to back (1.0).
    • Partial / Unsolved (Appendix F.3, Figure 19a–b, p. 32): (4) Close contact + monocular depth drift (Figure 19a): touching laptop–lamp boxes (0.02 m vs. 0.88 m centers) re-anchor via edge pixels from 0.06 m (0.0 no-skills) to 0.21 m vs. 0.30 m GT, but single-view depth drift (z = 1.94 / 2.08 m) caps score at partial 0.5; (5) Heavy multi-view occlusion (Figure 19b): when target surfaces stay hidden across views even after cropping, recovery still scores 0.0 (18/187 acted items on H2H_2; 35/124 on H5.6H^{5.6}).
Reviewer_kvbdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Vision-language models often struggle with exact spatial calculations despite being able to perceive complex environments. To solve this, the authors introduce SpatialWeave, an external system of tools, workflows, and rules that improves a frozen model without changing its internal weights. During a training phase, a temporary guide helps the model explore different tool combinations and saves the best sequences as reusable workflows. When answering real questions, this temporary guide is removed to save computing power while the model relies on the saved workflows and rules. The authors tested it across seven benchmarks, and three different models show huge accuracy improvements, with smaller models even beating larger base models.

Critical strengths

I think the automatic discovery and subsequent removal of the temporary training guide is a very intuitive way to explore complex tool chains without paying the cost during actual use.

Critical weaknesses

The most critical weaknesses of this paper come from four main problems.

First, the system is completely blind without perfect external perception tools. The entire reasoning process depends on getting flawless data from 3D reconstruction and depth estimation. If those tools make a mistake or fail to see an object, the reasoning model has no way to recover or correct the error.

Second, the rules the model learns are too rigid and brittle. The system uses exact word triggers to decide when to apply special spatial lessons. If a human asks a question using slightly different words, the triggers fail to activate, and the model loses its special guidance.

Third, the hidden cost of reading huge amounts of text is a major bottleneck. Even though the method reduces the amount of text the model generates, it forces the model to read massive amounts of cached data and tool receipts at every step. This requires enormous computing power and makes the system too slow for real-time use.

Finally, the system shows poor true generalization because it only managed to compile three reusable workflows. Despite being tested on many different types of spatial problems, it only found a few specific patterns to reuse. This suggests the method might just be memorizing narrow solutions rather than learning flexible reasoning skills that work on brand new problems.

Questions for authors
  1. How much does the final accuracy drop if you intentionally inject noise into the 3D reconstruction tools, and what percentage of the benchmark errors can be directly traced back to perception failures?

  2. Could you provide a preliminary experiment or theoretical discussion on how the compiled workflows would adapt if the input were changed from static images to a continuous video feed?

  3. What is the actual latency and hardware requirement for processing these massive context windows, and how does the performance degrade if you force the model to use a strict token limit?

  4. Can you share the specific system prompts and verification constraints given to the coding agents during the training phase, along with examples of rejected code they generated?

  5. Why did the mining process only yield three distinct workflows despite the diverse nature of the benchmarks, and did you experiment with lowering the support threshold to find more patterns? How sensitive is the system to semantic paraphrasing of the questions, and could you implement a meaning based trigger system to ensure the rules are applied even when the exact wording changes?

  6. What specific characteristics of the weaker benchmarks cause the external harness to underperform, and is there a way to make the model dynamically choose whether to use the harness or rely on its internal knowledge?

  7. What was the total compute time, number of application programming interface calls, and financial cost required to complete the entire training process from start to finish?

2 comments
AuthorsOpenReview ↗

Response to Reviewer kvbd (1/2): Perception Noise, Latency & Cost (Q1, Q3, Q8)

We thank Reviewer kvbd for recognizing our design of guiding tool exploration offline and retiring the playbook at deployment to cut inference overhead—complementing Reviewers frwc and Sxis on SpatialWeave's modular decomposition and consistent cross-backbone gains. All new audits are in the revised PDF (changelog in Fully Revised Draft).

Q1: Robustness When 2D Grounding, Depth, or 3D Reconstruction Fails (§4.5, p. 10; Table 4, p. 10; Appendix F, pp. 27–32)

SpatialWeave never blindly trusts GroundingDINO, UniDepthV2, or Pi3X outputs. Evolved workflows embed geometric validity checks (box overlap IoU >= 0.85, 0.8x inner-box depth stability, ±3° / 12° angle deadbands, closest-to-center distance collapse < 0.40, degenerate pose checks) that clear unreliable numbers (value=None) and return status="retry-grounding" or "not-applicable", paired with visual recovery steps in Tools and Lessons (§3.3, p. 5; Figure 3, p. 6):

  • Same-question cohort audit (Table 4a–c, p. 10): Scoring all variants on the exact same questions flagged by H2H_2 avoids selection bias: skills add +0.78 pp across all 4,929 test items (p=0.024p=0.024) and +8.23 pp across all 198 flagged items on the 4-benchmark panel (89.65% vs. 81.41% on same-code H2H_2-tools; +13.75 pp SPBench; 197/198 = 99.5% acted). Across 187 retry-grounding items (100% acted), H2H_2 fixes 69.0% (29/42) of seed H0H_0 errors while regressing on 3.4% (5/145) of correct items—29:5 repairs vs. regressions (Δ=+12.83 pp\Delta=+12.83\text{ pp}, p=3.9×10−5p=3.9\times 10^{-5}; 20:3 vs. H2H_2-tools, p=4.9×10−4p=4.9\times 10^{-4}). On H5.6H^{5.6} (Table 4c), skills raise recovery actions from 29.4% to 77.5% (124/160).
  • Direct repair of intermediate 3D measurements (§4.5, p. 10; Appendix F, p. 29): On MindCube's 171 retry-grounding triggers, crop re-grounding stabilizes 90/171 (52.6%) directly to status="ok" with 96.7% (87/90) ground-truth sector accuracy; on SPBench's 38 close-range distance triggers, re-anchoring lifts intermediate 3D distance accuracy (MRA) from 21.05% (suppressed pre-check float) to 33.16% (+12.11 pp) and cuts <0.10 m box collapse 2.8× (36.8% -> 13.2%).
  • Isolating Python value=None suppression & concrete cases (§4.5, p. 10; Appendix F, pp. 27–32): Printing a warning without clearing the bad float fails (58.5% -> 57.1%, p=0.742p=0.742), whereas value=None suppression jumps target accuracy to 69.9% (+11.4 pp, p=0.0029p=0.0029) and beats a prompt-only rule by +5.00 pp (+4.73 pp held-out from stage-v4 guards; Table 13a, p. 26). Concretely, when chair occlusion collapses printer–table distance to 0.00 m in view 8, clearing value=None triggers cross-view re-anchoring in view 5 to 1.39 m vs. 1.30 m GT (MRA 0.9 vs. 0.19 m / 0.0 without skills; Appendix F.3, Figure 17a, p. 31; see also Figures 13–16 & 18, pp. 27–32); conversely, monocular depth drift on touching laptop–lamp boxes (0.02 m -> 0.21 m vs. 0.30 m GT, partial MRA 0.5) and multi-view occlusion (score 0.0; 18/187 acted items on H2H_2) bound recovery (Figure 19a–b, p. 32).

Q3 & Q8: Serving Latency/VRAM, Token Budgets, & Evolution Dollar Cost (§4.4, pp. 9–10; Table 8, p. 21; Table 11, p. 25)

  • GPU serving latency & VRAM (2× RTX 4090, Table 8, p. 21): Separating Pi3X GPU forward passes from CPU rendering in Ray and caching .npy 3D point maps across turns (<50 ms workflow reads) cuts 60-call wall-clock from 1,532.3 s (25.5 s/call) to 286.1 s (4.77 s/call, 5.36× throughput) and reduces peak VRAM by -24.0% (13,150 -> 9,994 MiB). End-to-end on identical Gemma-4-31B weights, SpatialWeave is 2.56× faster than SpatialClaw (471.1 s vs. 1,206.5 s/q; §4.2, p. 7).
  • Token trade-offs (Figure 4, p. 9; Table 2a, p. 8): Relative to bare VLM (78.02%, 2.46k tok/q), H2H_2 (85.31%, 10.99k tok/q) trades compute for +7.29 pp accuracy; under tighter budgets, code-only H2H_2-tools (84.53%, 7.87k tok/q) beats seed H0H_0 (81.74%, 8.90k tok/q) by +2.79 pp using 11.6% fewer tokens; and retiring the playbook cuts tokens by 40.9% and calls by 65.6% vs. keeping it (18.59k tok/q, 3.85 tools/q).
  • Offline evolution cost (Table 11, p. 25): Freezing H2H_2 in Rounds 1–2 took 9.15 h on 2×RTX 4090 (333.0M uncached, 11.1M output, 2,081M cached tokens; 86.2% cache hit) = $985 standard API ($493 at 50% Batch API; full R1–R6: 20.8 h, $2,222 standard / $1,111 Batch), breaking even vs. keeping the playbook in ~38k–44k queries and transferring zero-shot across 5 backbones at $0 re-evolution cost.
AuthorsOpenReview ↗

Response to Reviewer kvbd (2/2): Video, Reproducibility & Routing (Q2, Q4-Q7)

Q2: Multi-Frame Video vs. Continuous Streams (§4.2, p. 8; §5, p. 10; Appendix B, p. 19)

Evaluating frozen H5.6H^{5.6} zero-shot on debiased VSI-Bench (2,362 egocentric video questions) with 16 sampled frames per video lifts accuracy by +2.35 pp. As noted in §5 (p. 10), SpatialWeave processes bounded sampled frame sets via batch multi-view reconstruction (Pi3X), rather than continuous real-time streams requiring online SLAM and dynamic-object tracking.

Q4: Reviser Prompts, Validation Rules, & Rejected Code Examples (§3.2, p. 5; Appendix H–I, pp. 34–42)

Appendix H–I provides the exact reviser prompt templates, Python AST whitelist (<= 40 statements, helper primitives sector, hsign, depth_label, convert), SHA-256 package hashes, explicit attribution of pre-specified priors vs. discovered artifacts (Table 18, p. 39), the complete 32-candidate validation log (Table 16, p. 38), and visual code/skill diff traces (Figure 20a–b, p. 37). Concrete rejected rows in Table 16 (p. 38) include:

  • Blocked before any VLM calls (0 LLM cost): Row r1/c2 was rejected by the static Python AST check for accessing restricted dunder attributes (Figure 20b, p. 37); row r2/c5 had all 21 candidate compiled chains shelved when offline replay on saved DdevD_{\text{dev}} .npy 3D point maps failed parity/net-repair checks.
  • Stopped early on target tasks (15 rows in Table 16, p. 38): Candidates failing to improve their target failure cluster on pass 1 (after excluding the ≤30\le 30 inspected diagnosis items) were halted automatically (r3/c1–c5, r4/c4–c5, r5/c1,c3–c5, r6/c4–c5), saving ~60% of verification calls.
  • Rejected at full 3-pass validation (Table 16, p. 38; Figure 20b, p. 37): Marginal or token-heavy patches such as rows r4/c2 (+0.65 ± 0.42 pp < 0.81), r4/c3 (+0.65 ± 0.50 pp < 0.98), and their R6 merge r6/m1 (+0.24 ± 0.52 pp < 1.01 at +12.9k extra tok/q) were automatically blocked, preventing late-round bloat.

Q5: Why 3 Compiled Workflows & Sensitivity to Support Threshold nmin⁡n_{\min} (Appendix H.2, p. 37; Appendix I, Table 21, p. 42)

Three is the number of retained cross-benchmark workflows, not the number of explored candidates (15+ candidate modules were synthesized across runs; in 6-round H6.1H^{6.1}, R1 compiled 3 workflows and R2 added 1 expanded workflow—Table 21, p. 42). Sweeping nmin⁡∈{5,10,15,20,30}n_{\min} \in \{5, 10, 15, 20, 30\} across 237 R1 and 219 R2 raw tool-call chains (Appendix H.2, p. 37) shows:

  • Raw clusters ≥nmin⁡\ge n_{\min} drop from 75 -> 40 -> 27 -> 21 -> 9 in R1 (59 -> 27 -> 14 -> 14 -> 9 in R2), and combining nmin⁡n_{\min} with offline 3D replay filters >85% of noisy or task-idiosyncratic chains, forming a stable plateau across n_min in [15, 20] (2 surviving families in R1, 10 in R2).
  • Lowering nmin⁡n_{\min} to 5 admits rare chains that fail to generalize across benchmarks, whereas the promoted motifs (camera_motion, measure_pair, observer-relocated relation) capture core 3D compositions shared across datasets, while rarer questions still compose the atomic tools freely.

Q6: Semantic Preconditions vs. Surface Keyword Matching (§3.3, p. 5; Appendix G, p. 33)

We respectfully clarify a misunderstanding from our earlier wording "trigger matches": SpatialWeave uses zero keyword, regex, or string-matching routers at runtime. Skill files (SKILL.md + references/*.md, Table 14, p. 33) state geometric and visual preconditions read directly by the VLM—e.g., "four ordered views around a shared center object with no camera translation" or "single-view depth margin under 10 cm". We updated §3.3 (p. 5) to "when all geometric preconditions hold". Crucially, the exact same frozen harness transfers zero-shot to external ViewSpatial (+20.45 pp), VSI-Bench (+2.35 pp), and OmniSpatial (+2.94 pp), whose prompt templates never appeared during development.

Q7: Smaller Gains on ERQA / Omni3D & Dynamic Zero-Tool Routing (§3.3, p. 5; Figure 10, p. 23)

Deployed SpatialWeave already lets the VLM decide dynamically whether to call tools or answer from visual knowledge: once the playbook is retired, GPT-5.6 Sol and GPT-6.1 Sol answer ~28% of held-out test questions with zero spatial tool calls (Figure 10, p. 23). On items testing abstract affordances/world knowledge (ERQA) or clear direct counts, skipping 3D tools avoids latency and noise while keeping gains positive across every backbone on all 7 benchmarks (Table 1a, p. 7). Explicit confidence-calibrated abstention is noted as future work (§5, p. 10).

Reviewer_frwcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents SpatialWeave, a spatial reasoning system that improves a frozen vision-language model through an external harness of tools, executable workflows, and textual skills. During development, a temporary procedure scaffold encourages exploration of tool-use trajectories, which inform revisions to the harness; the scaffold is then removed for inference. The authors evaluate the resulting system on seven spatial reasoning benchmarks and three main backbones, reporting improvements over the corresponding base models. They also compare against other systems and analyze cumulative revisions, component removals, and inference costs.

Critical strengths
  1. A practically motivated decomposition of spatial reasoning. The paper separates visual evidence acquisition, reusable geometric computation, and guidance for binding and interpreting results. This directly addresses recurring errors involving entities, quantities, and reference frames, and makes the resulting system inspectable.

  2. Broad empirical evaluation and informative component analysis. The authors evaluate multiple backbones across seven benchmarks and report both cumulative revisions and component removals. They also disclose an important efficiency tradeoff: fewer turns and tool calls than SpatialClaw on the Gemma comparison, but higher input-token usage. These results make the system’s benefits and costs easier to assess.

Critical weaknesses
  1. The evidence for generalization beyond the development distribution is limited. The split is by question ID rather than by scene, and the harness is repeatedly revised and selected using development questions. This does not establish leakage, but it leaves open how much the learned tools and lessons transfer to unseen scenes or task families. Since transfer is central to the value of a reusable harness, scene-disjoint or leave-one-benchmark-out evaluation would materially strengthen the paper.

  2. The contribution of co-evolution is not fully isolated. The v0–v6 comparison evaluates one cumulative sequence of revisions, while the final-system ablations remove components after that sequence. These experiments show that the resulting components can be useful, but do not by themselves establish how much coordinated evolution contributes compared with independently evolving tools, workflows, or skills under a comparable development budget. The role of researcher acceptance decisions and rejected candidates also needs clarification.

Questions for authors
  1. What is the extent of scene overlap between development and evaluation, and are scene-disjoint or leave-one-benchmark-out results available? How do those results compare with the reported question-disjoint evaluation?

  2. How many candidate revisions were proposed, rejected, and accepted, and what researcher decisions were involved? Were any final harness or version-selection decisions informed by evaluation-set results?

  3. Can the authors compare coordinated evolution against single-component alternatives under a comparable development budget, or otherwise quantify how much of the gain depends on jointly revising tools, workflows, and skills?

  4. What is the paired uncertainty for the 1.35-point comparison with SpatialClaw, and how do the systems compare under matched inference budgets? This comparison would affect my assessment of the claimed practical advantage.

  5. General-purpose agent baseline. Have the authors compared SpatialWeave with a general-purpose agent harness, such as Claude Code or Codex, directly solving the same tasks with access to the same perception and geometry tools? A comparison using the same backbone and matched inference budget would help distinguish gains from the proposed co-evolution procedure from gains due to ordinary tool access and agentic execution.

  6. Transfer to unseen benchmarks and scenes. The paper shows transfer from the Sol-developed harness to other backbones, but the development and evaluation sets are split by question ID within the evaluated benchmarks. Can the authors report leave-one-benchmark-out and scene-disjoint results, ideally including transfer to a different backbone? How much of the reported gain remains on questions whose images and scenes do not overlap with development?

2 comments
AuthorsOpenReview ↗

Response to Reviewer frwc (1/2): Unseen Scenes & Self-Evolution (Q1, Q2, Q6)

We thank Reviewer frwc for recognizing our modular decomposition, 7-benchmark / multi-backbone breadth, and strong gains—strengths echoed by Reviewers kvbd (explore-then-retire efficiency) and Sxis (cross-model gains). All new experiments are in the revised PDF (changelog in Fully Revised Draft).

Q1 & Q6: Scene Overlap, Unseen Indoor Rooms, & Pure Evolution Gains (§4.2, pp. 7–8; Table 1b–c, p. 7; Table 6, p. 19)

Question-ID splits alone cannot rule out room familiarity, so we added three scene-disjoint analyses:

  1. External room-disjoint OOD (9,607 items; Table 1b, p. 7): On ViewSpatial (5,712 items), matching all 2,878 ScanNet items against every DdevD_{\text{dev}} scan (1,242 items) by exact scan (scene####_##) and room family (scene####) yields +19.74 pp on GPT-5.6 Sol (43.07% -> 63.53% overall) and +13.08 pp ([10.5, 15.7]) on GPT-6.1 Sol (H2H_2) across 1,697 unseen scans, and +20.45 pp / +12.89 pp ([9.3, 16.5]) across 807 questions from unseen room families—matching Seen-room gains (+19.56 / +12.07 pp). Zero-shot H5.6H^{5.6} on GPT-6.1 Sol also gains +9.61 pp on ViewSpatial, +2.35 pp on 16-frame video VSI-Bench (2,362 items), and +2.94 pp ([0.71, 5.17]) on OmniSpatial (1,533 items) (§4.2, p. 8).
  2. Upstream scene audit on 4,929 internal items (Table 1c, p. 7; Table 6, p. 19): Splitting all 4,929 test items by source image/scan keys into 1,896 Seen-source and 3,033 Unseen-source items shows gains across all 3,033 Unseen items on every backbone (+4.43 pp Sol 6.1, +5.81 pp Sol 5.6, +8.27 pp Qwen, +7.25 pp Gemma; BLINK Unseen N=554: +8.12 to +12.82 pp; MindCube Unseen N=155: +18.06 Sol 5.6, +12.90 Qwen, +17.42 pp Gemma).
  3. Pure evolution (H0→H2H_0 \to H_2) vs. base tools on unseen scenes (§4.2, p. 8; Appendix B, p. 19): Comparing Bare →\to Seed H0H_0 (11 base tools + seed skills) →\to Evolved H2H_2 on GPT-6.1 Sol shows pure evolution (H0→H2H_0 \to H_2) adds +1.09 ± 0.48 pp (p=0.022p=0.022) across all 3,033 unseen items (+3.09 ± 0.85 pp, p=2.8×10−4p=2.8\times 10^{-4} across unseen BLINK/MindCube/SPBench). Crucially, on 369 unseen ScanNet 3D scan questions, seed H0H_0 adds only +1.60 ± 1.41 pp (p=0.26p=0.26) over bare, whereas co-evolution (H0→H2H_0 \to H_2) adds +8.21 ± 1.78 pp (p=5.5×10−6p=5.5\times 10^{-6})—83.7% of the total unseen-scan gain. Under scene#### room-family merging (201 unseen items), overall SPBench deltas narrow (+0.70 pp vs. bare, +2.44 pp vs. H0H_0) solely because camera-viewpoint items shrink from 31.7% of Seen scans to 8.0% (16/201) while monocular-scale measurement rises to 83.1%; on camera viewpoint itself, H0→H2H_0 \to H_2 adds +52.54 pp on unseen scans (N=59) and +37.50 pp on unseen families (N=16, where H0H_0 adds +0.00 pp).

Q2: Evolution Transparency, Zero Human Curation, & Test Independence (§3.2, p. 5; §4.1, p. 6; Appendix H.2, Tables 16–17, p. 38)

We cleanly separate (§4.1, p. 6): (1) 6-round zero-human H6.1H^{6.1} on GPT-6.1 Sol (30 candidate slots + 2 merges = 32 dev evaluations, zero human edits between seed H0H_0 and post-R6 freeze), and (2) cross-backbone H5.6H^{5.6} (evolved on GPT-5.6 Sol via 32,071 dev rollouts, coding-agent synthesis, 3D replay, and 23,361 paired validation rollouts rejecting 7/15 candidates before freeze, then transferred frozen across 5 backbones):

  • Test independence: Selection used only DdevD_{\text{dev}} (1,242 items), excluding the ≤30\le 30 error examples read during diagnosis and re-running the parent fresh on target tasks. All 4,929 held-out test items were evaluated once after R6 freeze.
  • Full 32-candidate log (Appendix H.2, Figure 20, p. 37; Table 16, p. 38): 2 failed static syntax / offline 3D replay (0 VLM calls), 15 stopped early when pass-1 target scores did not improve, and 15 completed 3-pass validation. In Table 16 (p. 38), R1 accepted code-only row r1/c5 (+2.40 ± 0.72 pp, Holm p=0.0059p=0.0059) →H1\to H_1; R2 accepted coupled rows r2/c3 (+1.23 ± 0.61 pp) and r2/c4 (target +1.06 ± 0.28 pp, Holm p=0.0011p=0.0011), merging into r2/m1 (H2H_2: +1.69 ± 0.57 pp, -1,752 tok/q, Holm p=0.0215p=0.0215; cumulative H0→H2H_0 \to H_2 dev +5.21 pp). R3–R6 rejected all 20 candidates plus R6 merge (+0.24 ± 0.52 pp at +12.9k tok/q), preventing drift.
  • 4-run reproducibility (Table 17b, p. 38): Four independent runs from the identical 11-tool seed (including 5-round unattended 74.08% -> 83.78%, +9.70 pp dev, and +6.18 ± 0.67 pp on 2,450 held-out items, p=5.4×10−20p=5.4\times 10^{-20}) consistently rediscover the same 4 geometric/guard families.
AuthorsOpenReview ↗

Response to Reviewer frwc (2/2): Co-Evolution, SpatialClaw & Codex (Q3-Q5)

Q3: Coupled Co-Evolution vs. Equal-Budget Single-Sided Search (§4.3, pp. 8–9; Table 2, p. 8; Table 17a, p. 38)

We verify the benefit of joint code–skill evolution both in equal-budget independent search and in component splits:

  1. Equal-budget independent search (Appendix H.2, Table 17a, p. 38): From the exact same 11-tool seed H0H_0 (74.08%), dev split (N=620), mined chains, and 970 paired validation episodes per slot, coupled code+skill search (80.91%, +6.83 ± 1.17 pp) significantly beats standalone skill-only search (79.05%, +4.98 ± 1.27 pp; head-to-head +1.85 ± 0.83 pp, p=0.0255p=0.0255) and code-compile-only search (78.34%, +4.26 ± 1.23 pp; head-to-head +2.57 ± 0.87 pp, p=0.0032p=0.0032), while uniquely avoiding non-target regressions (+1.11 pp vs. -0.82 / -1.26 pp).
  2. Uncoupled compilation hurts held-out accuracy (Table 2a, p. 8; Appendix H.2, Figure 20c, p. 37): R1 code-only H1H_1 dropped -1.11 pp below H0H_0 on the 4,929 test set (81.74% -> 80.63%, p=0.0017p=0.0017), whereas coupled H2H_2 rose to 85.31% (+3.57 pp over H0H_0, p=1.4×10−17p=1.4\times 10^{-17}; +4.68 pp over H1H_1). On the 160-item target slice (Figure 20c, p. 37), code-only dropped 90.00% -> 88.75%, skill-only reached 95.62%, and coupled reached 98.12%.
  3. Code–skill splits & workflow routing (Table 2b, p. 8; Table 10b, p. 25): Across N=2,770 under matched settings, evolved code with no skills (H2H_2-tools) gains +5.34 pp over H0H_0 (80.20% -> 85.54%, p=7.4×10−19p=7.4\times 10^{-19}) using 7.0% fewer tokens, and coupled skills lift H2H_2 to 86.58% (+1.04 pp, p=0.0227p=0.0227). Removing workflows post-hoc (86.03%) retains skills co-discovered during R2 coupled search; its pooled gap to H2H_2 is +0.55 ± 0.42 pp solely because 81.3% (2,253/2,770) of items invoke zero workflows (-0.02 ± 0.43 pp). On the 517 workflow-routed items, H2H_2 gains +3.02 ± 1.23 pp (93.60% vs. 90.58%, p=0.014p=0.014; cluster p=0.030p=0.030); on H5.6H^{5.6}, workflows add +2.00 ± 0.57 pp overall (p=4.6×10−4p=4.6\times 10^{-4}) and +4.62 ± 1.14 pp on 556 routed items (Table 3a, p. 9).

Q4: Statistical Significance & Inference Budget vs. SpatialClaw (§4.2, p. 7; Appendix B, Table 7, p. 20)

Aligning all 4,929 test questions on identical Gemma-4-31B-it-FP8 weights (enable_thinking=True, 32,768 max tokens) with 50,000 bootstrap resamples and 100,000 permutations across 2,925 scene/image clusters (Table 7, p. 20):

  • Accuracy & significance: SpatialWeave scores 73.58% vs. 72.23% (+1.35 pp), with item-paired 95% CI [+0.12, +2.58] pp (p=0.0304p=0.0304) and cluster-paired 95% CI [+0.02, +2.67] pp (p=0.0497p=0.0497). Gains come from multi-view BLINK (90.65% vs. 84.35%, +6.29 pp, Holm p=0.0027p=0.0027) and MindCube (84.17% vs. 78.21%, +5.95 pp, Holm p=0.0194p=0.0194), plus small edges on ERQA (+1.56), Omni3D (+0.83), and CV-Bench3D (+0.10), while trailing slightly on SPBench (-0.91) and MMSI (-2.24). §4.2 (p. 7) and Figure 9 (p. 23) report this accurately as modestly higher overall accuracy.
  • Tokens & latency: SpatialWeave cuts output tokens by 27.9% (4,691 vs. 6,507/q), turns by 17.3% (5.00 vs. 6.04/q), and mean wall-clock latency by 2.56× (471.1 s vs. 1,206.5 s/q). We also note transparently (Appendix B, p. 20) that cumulative input tokens (including cached prefix reads) are +37.8% higher (104.3k vs. 75.7k/q; SpatialClaw's log omits sub-VLM kernel calls), so we claim matched-weight accuracy and latency/output efficiency rather than lower cumulative input tokens.

Q5: General Coding-Agent Baseline (OpenAI Codex CLI; Appendix B, p. 20)

SpatialWeave exports standard MCP endpoints and Markdown SKILL.md files, requiring no custom loop. Evaluated directly inside the unmodified OpenAI Codex CLI (GPT-5.4) across all 7 benchmarks (5,550 questions; Appendix B, p. 20):

  • Bare VLM scores 64.00% macro; Codex + raw spatial MCP tools (standard agent tool use) scores 70.91%; and Codex + SpatialWeave (MCP + evolved skills) reaches 75.45% (+4.54 pp over raw agent tool use; +14.15 pp on SPBench, +8.25 pp on MindCube).
  • On matched BLINK (624 items, GPT-5.6 Sol), frozen H5.6H^{5.6} scores 91.16% in pi and 89.12% in Codex (vs. 84.71% bare). We use pi during evolution solely for clean permission sandboxing without subagent/memory confounders.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Fully Revised Draft

Dear Reviewers and ACs:

Thank you very much for your constructive comments and insightful reviews, which have greatly helped improve our manuscript. We have carefully addressed every suggestion and uploaded a fully revised 10-page draft, with the main changes marked in blue.

Specifically, we have made the following changes:

In the main paper:

  1. Rewrote the Introduction and Method in plain, concrete terms around explicit code/skill interfaces and step-by-step visual diagrams, removing dense notation (§1, pp. 1–3; §3.1–3.3, pp. 4–6; Figures 1–3).
  2. Clarified the distinct scopes of our 6-round zero-human self-evolved harness (H6.1\mathcal{H}^{6.1} on GPT-6.1 Sol) and our cross-backbone self-evolved harness (H5.6\mathcal{H}^{5.6} transferred frozen across five VLM backbones and external OOD suites) (§3.2, p. 5; §4.1, p. 6).
  3. Added external room-disjoint evaluations on ViewSpatial (5,712 items), 16-frame video VSI-Bench (2,362 items), and OmniSpatial (1,533 items), plus an internal Seen vs. Unseen scene audit and a Bare -> Seed H0 -> Evolved H2 breakdown isolating pure evolution gains on unseen 3D rooms (§4.2, pp. 7–8; Table 1b–c, p. 7).
  4. Added equal-budget coupled vs. single-sided search results, multi-run structural convergence across four independent runs, and a routing analysis on the 517 workflow-invoking test items explaining post-hoc workflow removal (§4.3, pp. 8–9; Table 2a–b, p. 8).
  5. Added a 3-round evolution control without the offline exploration playbook, two-baseline deployment token/call comparisons, and end-to-end evolution API dollar costs (§4.4, pp. 9–10; Figure 4, p. 9).
  6. Added a fixed-cohort outcome audit on diagnostic-triggered items, direct accuracy measurements of repaired intermediate 3D geometric readouts, and controlled experiments isolating Python value=None suppression from text warnings (§4.5, p. 10; Table 4a–c, p. 10).

In the appendix:

  1. Added per-benchmark source-scene Seen vs. Unseen tables and unseen-room H0 -> H2 decomposition details across all seven main benchmarks (Appendix B, Table 6, p. 19).
  2. Added the 4,929-item cluster-paired significance, token, and wall-clock latency audit against SpatialClaw on identical Gemma-4-31B weights (Appendix B, Table 7, p. 20; Figure 9, p. 23).
  3. Added portability experiments in the unmodified OpenAI Codex CLI coding-agent runtime across 5,550 spatial questions (Appendix B, p. 20).
  4. Added steady-state Ray/Pi3X GPU serving latency, throughput (5.36×), and peak VRAM (-24.0%) benchmarks on 2× RTX 4090 (Appendix B, Table 8, p. 21).
  5. Added the round-by-round offline evolution token, GPU-hour, API dollar cost ($985 to freeze H2\mathcal{H}_2), and deployment break-even table (Appendix E, Table 11, p. 25).
  6. Added intermediate 3D readout repair audits and verbatim solved, partial, and occlusion-bounded failure trajectories (Appendix F, Figures 13–19, pp. 27–32).
  7. Added the complete 32-candidate adaptive validation ledger (Appendix H.2, Table 16, p. 38), the matched 970-episode coupled vs. single-sided search and 4-run convergence tables (Table 17, p. 38), and an explicit attribution of pre-specified priors vs. offline-discovered artifacts (Table 18, p. 39).

Detailed point-by-point responses are posted directly under each review. Thank you very much again!

Best regards, Submission 12141 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.