acceptodds
Under review as a conference paper at ICLR 2027

The Spatial Harness: From Fallible Visual Tools to Reliable Agentic Spatial Reasoning

Abstract

While vision-language models (VLMs) excel at semantic understanding, they struggle to locate evidence from videos or multi-view images, and express it as precise spatial measurements, such as object sizes and distances. Visual tools can supply these locations and measurements, yet their outputs remain fallible and may introduce plausible but incorrect observations into downstream reasoning. Reliable tool-assisted reasoning therefore requires verifying these observations before accepting them as evidence and seeking alternatives when they are rejected or insufficient. Building on this principle, we introduce **Spatial Harness**, which models visual tools as stochastic observation channels and formalizes evidence acquisition as an output-feedback control process. Implemented as a planner–executor–verifier loop, the planner is the acquisition controller, proposing primary and backup routes for the executor to instantiate, while verification and memory updates jointly provide observer-like interpretation of tool outputs. Updated memory guides replanning for evidence still missing, while routes retaining no evidence trigger backup execution. To train the roles for distinct responsibilities while reducing reward conflicts, we first optimize specialized teachers on offline role-local contexts containing successful and failed tool outputs. We consolidate their expertise into one role-conditioned agent through multi-teacher on-policy distillation, combining teacher feedback on student rollouts for capability acquisition with task-reward reinforcement learning for refinement. On ReVSI and Ego3D-Bench, the harness improves the accuracy of open-weight models and closed-source APIs, outperforming spatially specialized models. Post-training yields further gains by improving collaborations between agents and imperfect tools.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tool Use, Agent, Vision-Lanugage Models, Spatial Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_h74JOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Spatial Harness, which aims to improve the reliability of tool-assisted spatial reasoning. At its core is a closed evidence-acquisition loop composed of a planner, an executor, and a verifier: the planner schedules primary and backup routes for missing evidence, the executor generates programs that invoke visual tools, and the verifier applies semantic and spatial checks to the returned results, writing them into committed memory or unresolved memory accordingly. When no evidence is retained, the system executes backup routes; when evidence is still missing, it replans based on the updated memory. The method is evaluated on multiple benchmarks.

Critical strengths
  1. The research is innovative in its focus on the problem that visual tools may "return plausible but incorrect results." Distinguishing tool outputs from trustworthy evidence helps prevent perceptual errors from being amplified by downstream reasoning, which is of practical significance.

  2. Verification results determine how evidence enters memory and influence the triggering of backup execution and replanning. This makes verification an explicit execution mechanism and clarifies the responsibilities and training objectives of each role.

  3. The paper establishes a mechanism for actively seeking alternative evidence after errors are detected and integrates it into the method, enabling more flexible evidence accumulation across viewpoints and across steps.

Critical weaknesses
  1. I think the paper already has sufficient novelty, but the introduction still frequently overclaims its contributions. The paper's innovation lies in verification-controlled memory writes, evidence states, and recovery rules. However, existing methods (e.g., SpatialClaw[1]) already include strategies for checking tool outputs and cross-validation, so the incremental novelty at the base-method level needs to be delineated more accurately, and the related text should be revised accordingly.

[1] SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning (arXiv:2606.13673).

  1. Trying tools offline to form a reference ordering, and using this ordering to determine primary and backup routes, may cause misunderstanding and errors in OOM environments, and it inherently restricts generalization to environments with different tool sets.

  2. Unresolved memory also participates in the evidence-coverage determination and can terminate the backup-route search of the current round. However, retaining a candidate does not imply that the ambiguity affecting the answer has been resolved. The current description does not sufficiently explain when the system will continue disambiguation and when it considers ambiguous evidence adequate to support the final answer.

  3. I do not consider the training pipeline and algorithm to be a distinctive contribution: role-specialized teachers, DAPO, and MOPD are all well-established techniques, and the authors merely adopt them without introducing innovations on top.

Questions for authors
  1. The paper reports inference-cost comparisons, but the planning and recovery strategies do not explicitly account for tool costs, latency, or the expected benefit of additional evidence.

  2. Could this framework, conversely, be used to distinguish the reliability and importance of different tools? Going further, could tools be pruned to improve accuracy and reduce cost?

  3. Modeling visual tools as stochastic observation channels and interpreting evidence acquisition as an output-feedback process suggests that more systematic and quantitative control could be achieved based on observation-probability estimation, budgets, and related factors.

5 comments
AuthorsOpenReview ↗

Appreciation for the Encouraging Feedbacks and Summary of Revisions and New Experiments (1 of 4)

We sincerely thank the reviewer for the encouraging feedback and constructive suggestions. We particularly appreciate the recognition of our focus on silent failures of visual tools and the importance of distinguishing tool outputs from trustworthy evidence. Your comments have helped us clarify our contributions and motivated a more systematic analysis of tool reliability, marginal contribution, and acquisition cost, opening promising directions for more adaptive, resource-aware planning.

We have made substantial updates to the paper pdf and conducted new experiments to address all points:

  • W1: Updated the introduction part to distinguish between our contribution and the output checking mechanism used by SpatialClaw.
  • W2: Performs an experiment which randomly removes the accuracy of certain tools and set it as a default value to showcase the robustness of new tool introduction.
  • W3: Added explanation about the usage of unresolved memory in Section 3.2, and provided more qualitative examples to show how the unresolved memory works in the Appendix F.
  • W4: Updated the narratives of training section to emphasize the the selective competence acquisition strategy from distinct training signals, and added another ablation which applies DAPO and MOPD without gating mechanisms to showcase the effectiveness of our proposed Mastery-gated acquisition of teacher signals.
  • Q1: Performed a small scale controlled experiment with limited tool call budget explicitly told to the planner, and discovers that the planner is internally aware of the tool call budget and is secretly cost efficient.
  • Q2: Performed another controlled experiment on the same data with some of the tools removed and reported the accuracy-tool cost trade off.
  • Q3: Proposed a PoC of estimating the tool result distribution and proved its effectiveness over the same data, showcasing the promise of leveraging observation probability distribution.

Details are provided below. We are greateful to your insightful comments an look forward to any further feedback.


W1: About the contribution between Spatial Harness and SpatialClaw.

We thank the reivewer for mentioning the contribution comparison between Spatial Harness and SpatialClaw. We acknowledge that the tool result checking and cross reference have been established as prompts of SpatialClaw. However, a prompt to inspect the evidence is not equivalent to a verdict of its correctness, as prompts may not be faithfully executed by the agent. In Spatial Harness, we made the verification process mandatory and explicit by introducing a separate role and explicit verification stages on both the semantic intergrity and spatial consistency.

We further inspected the paper, code and running logs of both SpatialClaw and Spatial Harness, and discovered that a mandatory checking procedure is vital to guarantee the execution of the checking process. Spatialclaw assigned their checking process merely in the prompts, leading to the checking procedure becoming an occasionally triggered activity depending on the moment-to-moment capability of the agents capabilities. We used GPT-5.6-sol to analyze the agent trajectories of both methods. The table below shows the proportion of the checks actually fires when it is prompted to do so, and the proportion of the checks that finishes without errors.

MethodCheck Fired RateCheck Finished Rate
SpatialClaw35.98%24.01%
Spatial Harness100.00% (by design)99.9%

From the table we can discover that prompting the check alone does not guarantee faithful checking behaviors, which lets the silent failures slip through, thereby showcasing the importance of the role-level mandatory checking procedures.

We have revised our Introduction Section to highlight the difference of verification mechanisms between Spatial Harness and SpatialClaw.

AuthorsOpenReview ↗

Appreciation for the Encouraging Feedbacks and Summary of Revisions and New Experiments (2 of 4)

W2: Discussion about reference ordering on offline scenarios and newly arrived tools.

We thank the reviewer for opening a discussion about the tool reference ordering. The accuracy of Spatial Harness is highly robust to the tool ordering and can transfer well when new environments or new tools arrive.

1. For new environments. Our experiment results on Ego3D dataset have well illustrated the generalization capabilities to new enviroments. The offline tool accuracies are collected at our training set, which does not contain outdoor road environments, as featured by Ego3D datasets. However, Spatial Harness still performs overall the best among all methods listed, showcasing the strong robustness of our harness against environment changes.

2. For new tools. We made an extra ablation study to imitate the circumstances when new tools arrive. We randomly remove the measured accuracies for 50% of the tools in the functional tool sets, and set them as the average of the remaining tools. We measure the changes of their performances and costs at ReVSI datasets, as shown below.

MethodReVSI (Macro Avg.)Avg. Num. of Tool Calls
Spatial Harness (GPT-5.2)67.4%9.37
Tool Prior Randomly Removed (GPT-5.2)65.7%10.82

The results illustrate that some tools with no known accuracy priors does not heavily affect the overall performance and toool costs, since the tool itself is still capable of performing their tasks even without knowing its success rates, while the verifier is still capable of gating the tool errors, which is agnostic to tool ordering.

W3: About the unresolved memory.

In the verification process, each observation who has passed the semantic checks go into the memory. If all the passed observations can be merged to one entity in space, it goes into the verified memory. Otherwise, all distinct and unmergeable pieces go into the unresolved memory together, as the harness cannot judge which piece the question refers to. A typical usage of the unresolved memory is the counting task, as each piece is not negligible to perform the correct countings. We have also added a demo about appying unresolved memory to reasoning over moving objects in dynamic scenes in newly added Appendix G. , where unresolved memory serves as the register for locations of the moving entity at different timesteps.

When the memory acquision ends, all the unresolved memory, as well as the frames and questions are taken into the conclusion stage. The planning agent check both the memory and the question, and decides how to leverage the memory to make subsequent plans, or to reason over the memory and answer. The newly added qualitative examples in Appendix F provides some illustrative results (Figure 5, Figure 11 and Figure 13), where unresolved memory serves as the evidence of counting, or the conflicts to be resolved.

In all, the disambiguation of the unresolved memory happens at the concluding stage, and the usefulness of the unresolved memory depends on the task type and the actual config of the scene.

AuthorsOpenReview ↗

Appreciation for the Encouraging Feedbacks and Summary of Revisions and New Experiments (3 of 4)

W4: The innovation of the training strategy.

Instead of combining MOPD and DAPO, what we propose is a training signal gating mechanism named Mastery-guided Acquisition. The recipe answers an important questions the harness raises: what a compact model should take from teachers that are imperfect themselves due to dataset bias.

Why role teachers. Although the three roles share the context and solves the spatial reasoning tasks in common, their required capabilities differ. The planner reasons about meaning and missing evidence, the executor writes programs, and the verifier recognizes and matches objects across views. When one compact backbone learns all three at once from mixed rewards, these skills compete and the training collapses when the students try to learn these skills together. We therefore let each skill mature in its own teacher, then consolidate the teachers into one student with multi-teacher on-policy distillation (MOPD), the established way to merge specialist capabilities into one model (e.g., MiMo-V2-Flash) without collapsing.

What a teacher should offer, and what it should not. A role teacher should offer the student its procedure, and nothing more. The prodedure includes which tool answers an evidence gap, how to set up the call, and how to tell when a return is unusable. This know-how is largely structural, and it is exactly what the student lacks when every rollout fails. There the task reward is too sparse to teach it, so only a teacher can do so. What a teacher should not offer is its spatial verdicts, whose correctness is defined by labels derived automatically from 3D annotations, noisy enough that ReVSI re-annotates VSI-Bench. A noisy label misleads the reward only when it is included, but a teacher that has fit the noise carries it into every rollout it guides. This also explains a result in the table below: deployed alone, the teachers fall below the untrained harness, yet distilled through the gate they lift the student well above it. Competence-gated guidance puts this into practice: the teacher speaks only where the student still lacks the procedure. In each rollout group:

  • Every response fails: the student cannot yet work the harness here, so the teacher shows how, alongside task reward.
  • Some succeed: the student already has the skill, so task reward refines it with the teacher off, and the teacher's bias stays out.
  • All succeed: mastered, skipped.

Concurrent work (RetireOPD, arXiv:2609.20784) likewise finds that teacher guidance should give way as the student becomes competent. Our gate acts per rollout group, so each role keeps its teacher wherever it still fails. The training thus mirrors the framework: the harness admits a tool output only after the verifier accepts it, and the training admits teacher guidance only where the student needs it.

Evidence and contributions. To showcase the effectiveness of our confidence-gated acquisition mechanism, we perform another ablation where the teacher signal is always on during the MOPD process. Applying DAPO, MOPD or both to every group leaves the harness below its untrained level. Competence-gated guidance raises it, and is the only one to surpass all three teachers on held-out role contexts. The updated ablation study in the paper shows that simply assembling DAPO and MOPD together also hurts the training dynamics.

TrainingReVSI (Macro Average)Validation Accuracy (in ablation study)
None (untrained harness)63.7–
Separate role teachers61.984.3
Multi-target DAPO51.074.1
MOPD from role teachers52.374.1
DAPO + MOPD, teacher always on (new)55.871.7
Ours: DAPO + MOPD, competence-gated71.687.7

The framework alone lifts Qwen3-VL-8B from 49.1 to 63.7, and training adds 7.9 more. Our training contribution is therefore not DAPO or MOPD. It is the gate that lets a teacher transfer tool use without its bias. We have revised the introduction, Contribution 3, Section 3.3, and Fig. 4 accordingly.

Reviewer_K8HZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Spatial Harness, a planner–executor–verifier framework for spatial reasoning with unreliable visual tools. The framework uses verification, memory, and backup tool routes to check observations and recover from failures. The authors also train separate role-specific teachers using DAPO and combine them into a single Qwen3-VL-8B model through multi-teacher distillation. Experiments on ReVSI and Ego3D-Bench show strong results, and ablation studies support the benefits of the proposed components.

Critical strengths
  • Well-motivated framework for reliable spatial reasoning. The paper addresses the practical issue of unreliable visual tool outputs through explicit verification, evidence memory, and alternative acquisition routes. The component ablations provide meaningful evidence for the effectiveness of this design.

  • Strong empirical results. Spatial Harness achieves substantial improvements on ReVSI and Ego3D-Bench, including in the inference-only setting. The role-specific training strategy further improves performance with an 8B backbone.

Critical weaknesses
  • Insufficient reproducibility of the method. My main concern is that the paper provides only a high-level algorithm and training objectives, leaving several essential details unclear. In particular, the criteria for classifying evidence as committed, unresolved, or rejected are insufficiently specified, as are the memory update and invalidation policies. It is also unclear how malformed tool calls and execution failures are handled. These are central to the paper's claimed reliability improvements.
    • Also, the paper lacks raw execution traces showing actual planner outputs, generated programs, tool observations, verifier decisions, and memory updates. Without such artifacts, it is difficult to verify whether the proposed evidence-control and recovery mechanisms operate as described.
  • Limited experimental control and concerns about tool-call reliability. SpatialClaw is evaluated using Qwen3.5-Plus, while Spatial Harness uses Qwen3-VL-8B or GPT-5.2, making it difficult to isolate framework-level improvements. Furthermore, based on my experience with Qwen3-VL-8B, obtaining consistently well-structured outputs for tool calling through prompting alone can be challenging. This raises questions about how the inference-only Spatial Harness reliably generates executable traces as "harness". The paper provides insufficient details on the prompting strategy and how malformed outputs are handled, making it difficult to assess the reproducibility of the reported results.

Minor

  • The robot's speech bubble at the top of the Figure 3 extends beyond its intended boundary.
Questions for authors
  • Could the authors provide actual execution traces showing tool calls, verification decisions, memory updates, and recovery from failures?

  • How does the verifier decide whether evidence is accepted, uncertain, or rejected? Can previously stored evidence be corrected or removed?

  • How does the inference-only Qwen3-VL-8B handle invalid tool-call formats or execution errors? Could the authors share the prompts used?

  • Why was Qwen3.5-Plus used for SpatialClaw? Could the authors compare both methods using the same backbone?

8 comments
AuthorsOpenReview ↗

Appreciation for the Thoughtful Suggestions and Summary of Revisions and New Experiments (1 of 3)

We sincerely thank the reviewer for the careful inspection and thoughtful feedback. We greatly appreciate the recognition of the well motivated framework and the strong empirical results. We also appreciate the constructive suggestions provided, which have guided us to further improve the clarity and reproducibility of our method.

We have provided an anonymous repository for inference code, made substantial updates to the paper pdf and conducted new experiments to address all points:

  • W1, Q1, Q2 and Q3: We provided the exact prompts and the json schema for constraint decoding in the Appendix and the updated supplementary material. Moreover, we have provided illustrative qualitative experiment results of each ReVSI category in the appendix, and provided the raw agent outputs of a test case to better visualize the dynamics of the Spatial Harness.
  • W2 and Q4: We have made an extra ablation study over the same Qwen3.5-Plus backbone and reports the superior performance.

Details are provided below. We are greateful to your insightful comments an look forward to any further feedback.


General responses to weakness and questions.

Spatial Harness keeps its agents within a prescribed workflow and requires every result to be handed over in a designated JSON schema (Lopopolo; Young), where prompting alone cannot guarantee well-formed tool calls ("Structured Outputs"). The harness enforces the schema at two levels: grammar-constrained decoding makes every output syntactically valid by construction (Willard and Louf), and execution level retries, within a bounded budget, for any step whose output is still malformed or does not finish execution (see supplementarhy meterials for the schemas, Appendix H for prompts, and Appendix F for raw traces).

W1a: On the reproducibility of Spatial Harness.

Spatial Harness is highly reproducible. We provide an anonymous repository (https://anonymous.4open.science/r/spatial-harness-4F22/) showcasing the core inference trajectory while removing identity leaking components, such as configs and backend hosts to protect anonymity. We also provided the prompts and the decoding json schema in the Appendix H and the supplementary material. We will open source the full code and model upon acceptance.

AuthorsOpenReview ↗

Appreciation for the Thoughtful Suggestions and Summary of Revisions and New Experiments (2 of 3)

W1a, W2b & Q2, Q3: About the sepcification of the evidence classification, memory update, format control, and handling of the malformed outputs.

Thanks for pointing the abmiguity out. We modified the paper to clarify the illustration of evidence classification and memory update (Section 3.2). For avoiding malformed outputs, this is conducted via enforcing a json schema during VLM decoding.

We summarize the clarification as follows:

(W1a-1) A. Evidence classification. The evidence from tool outputs go through two levels of checking inside the verifier. The first level is the semantic checking, focusing on whether the tool result fits into the concept of required by the planner. (e.g. The sofa is inside the bounding box. / The depth map does not contain outdoor area when required to calculate the room area.) Evidences fail the semantic checks are rejected.

All the remaining evidences pass and go into the spatial checking, where evidences are tested whether they can be spatially projected into one place. If all projected evidences can be spatially projected into one place, the union of these evidences are regarded as committed. If the evidences have separate clusters in space, all these clusters are regarded as unresolved, since the agents are unsure which of the clusters the questions are referring to (e.g. For counting tasks).

(W1a-2, Q2) B. Memory update and invalidation policies Memory update follows the rules of evidence calssification. For a single tool call results, committed evidence unions go into committed memory, while unresolved clusters go into unresolved memory. Once the evidences enter into memory, they are preserved and no longer invalidated. The judgement and the usage of the memory are conducted by the final concluding and answering phase.

(W1a-3, Q3) C. Format control, error and execution failure handling. Our initial experiments have the same findings that prompting Qwen3-VL alone cannot guarantee 100% legitimate outputs of its action choice at the format level. To prevent such errors from interrputing the agent loop, Spatial Harness explicitly proposes a schema of constraint decoding into the VLM calls in the calls of the agent roles, while at most 2 retries for the agent and tool outputs if execution fails (See Figure 6 in Appendix F). Constraint decoding strongly keeps the model to give out formatted outputs under the schema. We inspected our logs and find that, under our harness design, no outputs are still malformed or suffers from tool execution errors in the end of answering, which we largely attribute the effect to the schema provided to VLM decoding process, as well as the memory-based replans.

A typical illustration of schema of the executor and the corresponding VLM call (hosted by VLLM) are appended at the end of this reply. For full json schemas, please refer to the updated supplementary material.

W1b & Q1: The real runtime traces of agents.

We have updated the appendix to show the actual runtime traces of the agents, as well as more qualitative examples spanning through 7 ReVSI categories. Please refer to the Appendix F in the updated paper PDF, the runtime traces are appended after the illustrative examples.

W2 & Q4: Extra Qwen-3.5 ablation.

We chose Qwen3.5-Plus for SpatialClaw in the hope of faithfully reproduce its capabilities under its original configs.

To isolate the framework improvements from backbone improvements, we perform an extra ablation study which runs Spatial Harness over the same Qwen3.5-Plus API. The results are listed as follows.

Model / MethodMacro Avg.Obj. CountAbs. Dist.Obj. SizeRoom SizeRel. Dist.Rel. Dir.Route Plan
SpatialClaw (Qwen3.5-Plus)49.352.656.459.236.652.547.640.1
Spatial Harness (Qwen3-VL)63.756.775.759.754.067.572.459.9
Spatial Harness (Qwen3.5-Plus)66.553.377.761.752.271.976.871.7
Spatial Harness (GPT-5.2)67.461.576.771.055.770.272.364.5
Spatial Harness (Trained)71.666.176.980.066.970.772.368.5

From the table we can discover that Spatial Harness substantially outperforms SpatialClaw under the same backbone, showcasing the superiority and wide applicability of our harness design, as well as emphasizing our claim that harness design should rely on explicit roles and procedures, instead of moment-to-moment agent capabilities.

Minor 1: Layout issues in Figure 3.

Thanks a lot for pointing out, we have redrawn this figure to show better clarity.

AuthorsOpenReview ↗

Appreciation for the Thoughtful Suggestions and Summary of Revisions and New Experiments

References:

  • Lopopolo, Ryan. "Harness Engineering: Leveraging Codex in an Agent-First World." OpenAI, 11 Feb. 2026, openai.com/index/harness-engineering/.
  • "Structured Outputs." Claude Platform Docs, Anthropic, platform.claude.com/docs/en/build-with-claude/structured-outputs. Accessed 13 Nov. 2026.
  • Willard, Brandon T., and Rémi Louf. "Efficient Guided Generation for Large Language Models." arXiv, 2023, arxiv.org/abs/2307.09702.
  • Young, Justin. "Effective Harnesses for Long-Running Agents." Anthropic, 26 Nov. 2025, www.anthropic.com/engineering/effective-harnesses-for-long-running-agents.

Example schemas and vlm constraint decoding calls.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "Executor contracts across implemented and planned functional tool sets",
  "description": "Design-space schema covering four functional families. Existing tool branches reflect registered contracts.",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "tool_id",
    "call_id",
    "arguments"
  ],
  "properties": {
    "tool_id": {
      "type": "string"
    },
    "call_id": {
      "type": "string"
    },
    "arguments": {
      "type": "object"
    }
  },
  "oneOf": [
    // Here all the specifications of functional tool sets will appear.
    // Omitted due to space constraint in both the rebuttal area and the main paper.
    // Full specifications please see the supplementary materials package.
  ],
}

schema = exec_schema()
messages = build_exec_messages(paths, target, sheet, args.max_aliases)

response = client.chat.completions.create(
    model=args.vlm_model,
    messages=messages,
    max_tokens=args.vlm_max_tokens,
    temperature=0.0,
    top_p=1.0,
    extra_body={"guided_json": schema},
)
Reviewer_nQJZOpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+1)
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper presents Spatial Harness, a tool-assisted framework for spatial reasoning over videos and multi-view images that aims to reduce the impact of visual tool errors on final answers. The framework follows a planning–execution–verification loop. Given a question and the evidence collected so far, the planner proposes primary and backup tool-use routes, the executor translates each route into an executable program, and the verifier checks the semantic correctness and spatial consistency of the tool outputs. The system stores confirmed evidence separately from unresolved candidates. It tries backup routes when no usable evidence is retained and replans to fill remaining evidence gaps, before producing an answer from the visual input and accumulated memory.

For training, the authors first use DAPO to train separate teachers for planning, execution, and verification on offline contexts containing both successful and failed tool outputs. They then combine multi-teacher on-policy distillation (MOPD) with DAPO to transfer these capabilities into a single role-conditioned model. Experiments on ReVSI and Ego3D-Bench show that the framework improves spatial reasoning across the tested open-weight and proprietary models without task-specific training, with further gains from the proposed training procedure.

Critical strengths
  1. The paper brings tool use, evidence verification, and failure recovery into a coherent workflow for spatial reasoning. It addresses both how models acquire spatial information and how they proceed when that information is unreliable or incomplete. The approach provides a clear path from evidence collection to answer generation and has practical value.

  2. The experiments evaluate both the framework itself and the additional benefits of training. They cover video and multi-view spatial reasoning with different backbone models, while ablations examine the roles of tools, memory, verification, and training strategies. These analyses help explain which components contribute to the results, beyond the overall performance comparisons.

Critical weaknesses
  1. ReVSI covers seven task categories, but the qualitative examples focus exclusively on relative direction. This makes it difficult to understand how the pipeline handles tasks with different evidence requirements, such as room-size estimation. In particular, the paper does not walk through which tools are called, what measurements they return, or how those measurements are verified. A complete example for each task would help establish the framework’s applicability beyond directional reasoning.

  2. The pipeline appears closely tailored to object grounding and relative spatial relationships. It is unclear which parts are shared across spatial tasks and which require task-specific procedures. The motivation for the particular training recipe is also not sufficiently convincing. Although the paper motivates role specialization and combines DAPO with MOPD, it does not clearly establish why this combination is needed for the proposed framework or what it contributes beyond applying existing training methods. The authors should distinguish more clearly between the contributions of the framework, its task-specific components, and the training procedure.

  3. Figures 1–3 leave several aspects of the method unclear. In Figure 1, the rocking chair appears to change position or orientation between the scene image and the tool-output image. It is unclear whether these are crops from different video frames or illustrative images, possibly generated with AI. This also raises questions about whether the videos contain moving objects in addition to camera motion. More generally, the paper does not explain how frames are selected across rounds: are the 64 frames sampled uniformly, and are tools applied to all sampled frames or only a selected subset? Figure 2 contains too much text to make the control flow easy to follow, while Figure 3’s layout and annotations make it difficult to trace the example from the initial detections to the final answer.

  4. The runtime cost of iterative inference is unclear. The framework involves repeated planning, tool execution, and verification, but the reported token usage and API costs do not establish its end-to-end latency. Please report the average latency per sample, the average number of inference rounds, and the time spent in each stage, including visual-tool execution. A comparison with existing tool-assisted methods would help clarify the trade-off between accuracy and inference time.

Questions for authors

Please see weaknesses part.

8 comments
AuthorsOpenReview ↗

Appreciation for the Thoughtful Suggestions and Summary of Revisions and New Experiments (1 of 4)

We sincerely thank the reviewer for the thoughtful feedback. We greatly appreciate the recognition of the coherent workflow and strong empirical results. We also appreciate the constructive suggestions provided, which have guided us to further clarify the flexibility of our proposed method, how our harness design and training strategy complement the demands of spatial reasoning, as well as the efficiency of our method.

We have made substantial updates to the paper pdf and conducted new experiments to address all points:

  • W1: We have provided extra qualitative experiment results of each ReVSI category in Appendix F, and provided the raw agent outputs of a test case to better visualize the dynamics of the Spatial Harness.
  • W2: We have updated the paper to highlight the rationale behind our mastery-gated acquisition of teacher signals.
  • W3: We have redrawn all figures to calrify the inference procedure, and adds another demo to showcase the adaptation of spatial harness in dynamic environments.
  • W4: We have performed an extra experiment on the breakdown of the inference time cost.

Details are provided below. We are greateful to your insightful comments an look forward to any further feedback.


For weaknesses.

We thank the reviewer for raising the questions of the flexibility of the Spatial Harness, as well as the rationale of fitting Spatial Harness into our mastery-gated acquisition of teacher signals. The main points are as follows.

  • Spatial Harness aims at finding spatial evidence under the physical space, which does not limit themselves to objects. In practice, besides semantic consistency of objects, Spatial Harness also checks other semantic conformity like whether a depth map faithfullly depicts the boundary of a room.
  • Spatial Harness requires distinct capabilities to run, i.e. genreral reasoning, coding, multiview correspondence. Directly training on them not only witnesses reward collapse, but also introduces the bias of noisy spatial datasets. In this sense, our training strategy first trains role local teachers, and Mastery-guided Acquisition is proposed to distill the tool mastery while gating the learned teacher bias.

W1: On extra illustrative examples.

We thank the reviewer for requiring more illustrative examples of different tasks to better showcase the effectiveness of Spatial Harness. We provided extra illustrations for all seven ReVSI categories, as well as the full raw output of an example in Appendix F to better illustrate the dynamics of Spatial Harness.

The inference dynamics of all examples follows a high level workflow in each round as follows.

  • Plan for the evidences to obtain. (The evidence can be a specific object, or it can also be a general concept without specific object mentioned).
  • Translate the plan into code and obtain the results via executing the code.
  • Check if each tool execution results fits into the demand of the planner. Assemble all the results passed the check by their spatial locations, and put them into the memory accordingly.

For your specifically mentioned room size example, the flow is as follows, as is shown in Figure 9 in Appendix F of the updated paper.

  • The planner plans for a DA3 reconstruction through all frames, and then appends a contour calculation over at the ground level afterwards.
  • The executor calls tools to finish the two steps and the tools are executed with results returned.
  • The verifier first inspects all frames about whether it resembles the scene in the room. In this step, 9 of 64 results are discarded because they include the depth outside the room. The other 55 frames are spatially clustered and their results are saved in the unresolved memory. The subsequent contour calculation result is rejected because some of the sources does not fit into the demand.
  • In the second round, with the reconstruction information, the planner plan again for the contour calculation. The verifier approves the evidence, since the data residing in the memory conforms to the task demand now.

In this sense, the planner, executor and verifier jointly produce a high quality spatial reasoning results, with the planner makes dynamic tool call plans to obtain different spatial evidences, while the verifier faithfully checks whether the tool results fit into the planner's need, going beyond the simple semantic corretness of the object.

For full tool resutls walkthrough all ReVSI examples, please refer to Appendix F.

AuthorsOpenReview ↗

Appreciation for the Thoughtful Suggestions and Summary of Revisions and New Experiments (2 of 4)

W2a: On the flexibility of Spatial Harness.

We thank the reviewer for pointing out the flexibility issue of Spatial Harness. As the beginning and the example in Weakness 1 reply shows, our harness is highly flexible and can be adapted to different spatial demands even without specific objects mentioned. For the applicability of different tools, a table for different functionalities and task types in ReVSI is listed as follows.

Task NameSAM3YOLOEDA3VGGT-ΩDist.AngleContourBEVNVS
Obj. Cnt.✓✓✓✓✓
Abs. Dist.✓✓✓✓✓
Obj. Size✓✓✓✓✓✓
Room Size✓✓✓✓
Rel. Dist.✓✓✓✓✓
Rel. Dir.✓✓✓✓✓✓
Route Plan✓✓✓✓✓✓

From the table we can observe that object grounding functionality is not shared across all tasks, while the 3D reconstruction capabilities are shared, as it is the core of spatial reasoning in 3D scenes. This showcases that our harness base itself in proposing tool guided spatial information acquisition for VLMs, and is highly flexible to spatial reasoning tasks.

AuthorsOpenReview ↗

Appreciation for the Thoughtful Suggestions and Summary of Revisions and New Experiments (3 of 4)

W2b: The rationale of the training recipe.

The recipe answers two questions the harness raises: how one compact model can learn three very different roles, and what it should take from teachers that are imperfect themselves due to dataset bias.

Why role teachers. Although the three roles share the context and solves the spatial reasoning tasks in common, their required capabilities differ. The planner reasons about meaning and missing evidence, the executor writes programs, and the verifier recognizes and matches objects across views. When one compact backbone learns all three at once from mixed rewards, these skills compete and the training collapses when the students try to learn these skills together. We therefore let each skill mature in its own teacher, then consolidate the teachers into one student with multi-teacher on-policy distillation (MOPD), the established way to merge specialist capabilities into one model (e.g., MiMo-V2-Flash) without collapsing.

What a teacher should offer, and what it should not. A role teacher should offer the student its procedure, and nothing more. The prodedure includes which tool answers an evidence gap, how to set up the call, and how to tell when a return is unusable. This know-how is largely structural, and it is exactly what the student lacks when every rollout fails. There the task reward is too sparse to teach it, so only a teacher can do so. What a teacher should not offer is its spatial verdicts, whose correctness is defined by labels derived automatically from 3D annotations, noisy enough that ReVSI re-annotates VSI-Bench. A noisy label misleads the reward only when it is included, but a teacher that has fit the noise carries it into every rollout it guides. This also explains a result in the table below: deployed alone, the teachers fall below the untrained harness, yet distilled through the gate they lift the student well above it. Competence-gated guidance puts this into practice: the teacher speaks only where the student still lacks the procedure. In each rollout group:

  • Every response fails: the student cannot yet work the harness here, so the teacher shows how, alongside task reward.
  • Some succeed: the student already has the skill, so task reward refines it with the teacher off, and the teacher's bias stays out.
  • All succeed: mastered, skipped.

Concurrent work (RetireOPD, arXiv:2609.20784) likewise finds that teacher guidance should give way as the student becomes competent. Our gate acts per rollout group, so each role keeps its teacher wherever it still fails. The training thus mirrors the framework: the harness admits a tool output only after the verifier accepts it, and the training admits teacher guidance only where the student needs it.

Evidence and contributions. To showcase the effectiveness of our confidence-gated acquisition mechanism, we perform another ablation where the teacher signal is always on during the MOPD process. Applying DAPO, MOPD or both to every group leaves the harness below its untrained level. Competence-gated guidance raises it, and is the only one to surpass all three teachers on held-out role contexts. The updated ablation study in the paper shows that simply assembling DAPO and MOPD together also hurts the training dynamics.

TrainingReVSIValidation Accuracy (in ablation study)
None (untrained harness)63.7–
Separate role teachers61.984.3
Multi-target DAPO51.074.1
MOPD from role teachers52.374.1
DAPO + MOPD, teacher always on (new)55.871.7
Ours: DAPO + MOPD, competence-gated71.687.7

The framework alone lifts Qwen3-VL-8B from 49.1 to 63.7, and training adds 7.9 more. Our training contribution is therefore not DAPO or MOPD. It is the gate that lets a teacher transfer tool use without its bias. We have revised the introduction, Contribution 3, Section 3.3, and Fig. 4 accordingly.

Comments on the paper

2 comments

Dear reviewers, the authors have posted their responses to your reviews. Please have a look and discuss ASAP as the author-reviewer discussion period ends at November 18 at 11:59pm AoE

Best, AC

AuthorsOpenReview ↗

Summary of Recognized Strengths, New Experiments, Analysis, and Discussion about Further Extension

We sincerely thank the reviewers for their time and insightful feedback. We are encouraged that all reviewers recognized our contributions to reliable, tool-assisted agentic spatial reasoning, the feedbacks for polishing the paper, as well as valuable discussions on possible extensions.

Recognized Strengths and Contributions

1. Novel, well-motivated verification-driven framework Reviewers praised our focus on fallible visual tools that "return plausible but incorrect results" (h74J), describing Spatial Harness as a "well-motivated framework for reliable spatial reasoning" (K8HZ) that unites tool use, verification, and failure recovery in a "coherent workflow" (nQJZ) via an "explicit execution mechanism" (h74J, nQJZ).

2. Strong empirical results across benchmarks and backbones Reviewers highlighted "strong empirical results" and "substantial improvements on ReVSI and Ego3D-Bench, including in the inference-only setting" (K8HZ) across video/multi-view tasks and diverse backbones, with further gains from role-specific training and mastery-gated teacher signal guidance (nQJZ, K8HZ).

3. Comprehensive ablations and practical value Reviewers valued our component ablations as "meaningful evidence for the effectiveness of this design" (K8HZ, nQJZ) and recognized the framework's strong "practical value" (nQJZ, h74J).

Summary of New Experiments and Analysis

1. Same-backbone comparison & verification analysis (vs. SpatialClaw)

Addressing backbone control (K8HZ) over SpatialClaw, we evaluated Spatial Harness on Qwen3.5-Plus (66.1% vs. 52.1% on ReVSI). Trajectory analysis further confirms our mandatory verification fires in 100% of cases (99.9% finished) vs. SpatialClaw's prompt-based checks (35.98% fired, 24.01% finished).

2. Mastery-Guided Acquisition ablation

Addressing training novelty beyond DAPO/MOPD (nQJZ, h74J), we reframed our contribution and proposed an always-on DAPO + MOPD ablation drops to 55.8% ReVSI —below the untrained harness (63.7%)—whereas our competence-gated guidance reaches 71.6%, transferring tool-use procedures without teacher bias.

3. End-to-end runtime latency breakdown

Addressing iterative overhead (nQJZ), stage-level timing shows Spatial Harness uses less than half of SpatialClaw's latency (582.45s vs. 1311.5s) and far fewer inference rounds (1.37 vs. 8.69).

4. Robustness to unseen tools (missing accuracy priors)

Addressing offline tool-ordering concerns (h74J), randomly removing 50% of tool accuracy priors causes minimal shift (67.6% →\rightarrow 65.9% ReVSI; 9.37 →\rightarrow 10.82 calls), showing the verifier reliably gates errors regardless of priors.

5. Reproducibility, constrained decoding & full-category walkthroughs

Addressing reproducibility (K8HZ), we put forward exact prompts (Appendix H), JSON schemas for grammar-constrained decoding (Supplementary), raw traces, an anonymous repository for running demos, as well as walkthroughs across all seven ReVSI categories (Appendix F & H) (nQJZ).

Discussion about Further Extension

1. Resource-aware budgeting, tool pruning & reliability estimation

Following insightful discussion provided by Reviewer h74J about tool reliability audit and budgeting, we ran three proof-of-concept studies on a held-out ReVSI subset (GPT-5.2):

  • Budget-aware planning: Explicit budgets (0–20 calls) scale smoothly from 50.9% (0 calls) to 68.7% (10 calls) before plateauing, with >95%>95\% of 1-call runs prioritizing 3D reconstruction (DA3).

  • Tool pruning: Removing YOLOE (63.1%, 6.29 calls), VGGT-Ω\Omega (65.4%, 8.12 calls), or both (62.2%, 6.01 calls) quantifies the accuracy–cost tradeoff and confirms the marginal value of functional tool sets.

  • Observation-probability estimation: Contextual reliability statistics raise accuracy from 68.7% to 70.2% while reducing calls (9.37 →\rightarrow 8.99; ∼6.2%\sim 6.2\% first-tool changes), motivating a dedicated posterior estimator in future work.

We sincerely thank Reviewer h74J for raising this discussion as an idea of future extension of Spatial Harness!

2. Extension to 4D dynamic scenes Addressing moving-object scenarios beyond static camera motion (nQJZ), we added a DSR-Bench demo (Appendix G, Fig. 14), showing unresolved memory tracks moving entities across timesteps while replanning isolates target frame ranges.

Updated Paper and Final Version Plan

We revised the PDF for clarity—redrawing Figures 1–4, clarifying evidence classification and unresolved memory (Sec. 3.2), refining our SpatialClaw positioning (Intro), and updating Mastery-Guided Acquisition (Sec. 3.3, Fig. 5). New results and analyses will be integrated into the final version, with full code and models released upon acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.