acceptodds
Withdrawn from ICLR 2027

SpatialCanvas: A Visual Canvas Harness for Explicit Spatial Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models can recognize local spatial relations yet still fail on compositional spatial questions that require reasoning across viewpoints or reference frames. We argue that a key failure mode is not missing visual evidence itself, but losing the association between an observation and the entity, source view, and reference frame under which it is valid. We introduce SpatialCanvas, a training-free inference framework that makes these bindings explicit for a frozen VLM. Given a spatial question, SpatialCanvas compiles the required entities, reference frames, spatial operations, and evidence dependencies into a typed program. The VLM supplies source-bound observations, while SpatialCanvas instantiates the corresponding frames, stores observations and derived relations in an image-bound state, and executes only frame-compatible spatial transformations. A structural validator checks whether the evidence required by the program is supported before producing the final answer. With frozen Qwen3-VL-4B, SpatialCanvas improves accuracy over direct inference from 28.1 to 30.5 on MMSI-Bench, from 34.9 to 44.4 on MindCube-tiny, and from 45.7 to 48.1 on OmniSpatial. These results suggest that explicitly preserving reference-frame and source-view bindings can improve compositional spatial reasoning without updating model parameters. Our code will be made publicly available soon.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLM, Spatial Reasoning, Spatial Intelligence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NijyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SpatialCanvas, a training-free inference framework that maintains an explicit typed spatial program and an image-bound state for a frozen VLM. The program specifies entity roles, source views, reference frames, operations, and evidence obligations. The VLM supplies local observations, while deterministic operators perform transformations and derivations only when the required role, source, type, unit, and frame conditions are satisfied.

The framework renders intermediate state onto source images and supports targeted re-observation and revision. When the final evidence structure is complete and the derived relation uniquely identifies an answer, selection is deterministic; otherwise, the frozen VLM selects an option using the canvas and state readout. With Qwen3-VL-4B, the paper reports improvements in the “All” score on three benchmarks, together with cumulative ablations, structural-completeness diagnostics, paired correctness transitions, and execution statistics.

Critical strengths
  1. The framework separately records grounded marks, frames, derived relations, and evidence dependencies. Compatibility checks reject unsupported combinations of roles, types, sources, or frames, while upstream revisions invalidate and recompute only affected downstream records. The distinction between deterministic answer mapping and VLM fallback makes the execution interface inspectable. Its value lies in exposing the assumptions and provenance of spatial inference, rather than guaranteeing factual correctness.

  2. The reported overall gains are 2.4, 9.5, and 2.4 percentage points on MMSI-Bench, MindCube-tiny, and OmniSpatial, respectively. Cumulative ablations show an initial decline after grounding, followed by recovery and gains with the complete pipeline. The paper also reports regressions in PR., Ard., and Pers., alongside 540 harmful revisions in the diagnostic snapshot. Proof-conditioned accuracy varies substantially across datasets, appropriately showing that structural completeness is not a uniform confidence signal.

Critical weaknesses

The observer-frame construction prompts appear to conflate a planar heading convention with the projection of a physical body frame. Appendix B, prompts 2.4 and 2.5, specify SOURCE-IMAGE vectors and map a left-facing profile to physical LEFT=image-down and FRONT=image-left, with the reverse mapping for a right-facing profile. Candidate LEFT and FRONT axes are also required to be approximately perpendicular.

These rules are not generally valid for upright people in ordinary photographs. In a side-on view, the physical left–right axis can lie largely along the viewing direction and have a shortened or degenerate image projection. Moreover, projected orthogonal 3D axes need not remain orthogonal in 2D. A structurally complete frame could therefore encode an inappropriate geometric convention, or the consistency check could reject a valid projected frame. This concern targets the prescribed frame-construction semantics, beyond the already acknowledged limitation that structural validation cannot certify visual correctness.

Questions for authors
  1. In which geometric domain are the mappings and orthogonality check in Appendix B, prompts 2.4/2.5, intended to hold? Please distinguish image-plane, observer/body, camera, and world frames, and clarify whether these prompts are restricted to planar tasks. How are depth-aligned or degenerate body-axis projections and nonorthogonal projected axes handled? Controlled examples, the affected task coverage, and the provenance and dimensionality of the transformations in Eq. (5) would help establish the method’s scope.

  2. Could you provide a comparison at approximately matched total call budgets to assess the specific contribution of binding checks and targeted refinement? The cumulative ablations control several execution settings, but add components and permit additional observation rounds in Full. A generic decomposition/self-refinement baseline or a Full variant removing source/frame-binding checks separately would strengthen attribution. Without such controls, conclusions should emphasize the benefit of the complete pipeline rather than isolate the proposed checks.

  3. Could you reconcile the main tables, diagnostic snapshot, and token statistics through per-question execution records? Please provide evaluator aggregation formulas, run/snapshot identifiers, and paired correctness counts. The diagnostic transitions imply 1,106 Direct and 1,238 Full correct answers, whereas the rounded main-table scores would imply 1,095 and 1,243 if “All” were unweighted per-example accuracy. Please clarify whether weighting or different snapshots explain this difference.

Reviewer_TNu6OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work proposes a training-free/tool-free method for multi-perspective spatial reasoning. The authors claim that compositional spatial reasoning errors in VLMs come from losing entity, source-view, and frame bindings. They propose an execution framework in which a router converts each question into a structured plan that composes VLM observations using fixed deterministic operators. With this structure, their method checks inputs before each operation and can refine observations when checks fail. When evidence is complete, it selects the answer through deterministic execution. Otherwise, it asks the VLM to choose an answer using the canvas and recorded observations. They test their method on three commonly used benchmarks and show improvements from 28.1 to 30.5 on MMSI-Bench, 34.9 to 44.4 on MindCube-tiny, and 45.7 to 48.1 on OmniSpatial with Qwen3-VL-4B.

Critical strengths

The paper provides a very detailed analysis. It reports corrected and introduced errors, proof-conditioned accuracy, and breakdowns by call count and image count, making its benefits and limitations clear. Authors also provide Table 2, which adds grounding, coordinate reasoning, visual feedback, and refinement sequentially under a consistent evaluation protocol, making the incremental gains easy to follow.

Critical weaknesses
  1. There are a few closely related works that do explicit spatial reasoning that are missing from the comparison: pySpatial [1], VADAR [2], GCA [3], and Think3D [4]. Notably, several rows of Table 1 match GCA's Table 1, while that table's GCA row and TIGeR row are omitted.
  2. The novelty of this work is unclear to me. From the prior works, GCA [3] already uses a query-conditioned reference frame, Allocentric Perceiver [5] constructs observer frames, and TRACE [6] builds a scene state using the VLM without external tools. The main difference is unclear.
  3. The source of the gains seems to be perception not explicit reasoning. As shown in Table 2, much of the gain arrives with Refinement, which basically re-observes, corrects boxes, and re-estimates orientation. This suggests that improved perception/grounding may drive the results, but its effect is not separated from frame reasoning.
  4. MindCube gains seem to come from Rotation subset. Rotation questions state the camera rotations in text and can be solved by text reasoning plus image matching (Qwen3-VL-Thinking reaches the same 87.0), so this result does not demonstrate the claimed frame-binding mechanism.
  5. Only one backbone is tested, which limits the generalizability evidence.
  6. No ablation is provided for the execution layer. What happens if the same extracted scene data is given to the backbone VLM to reason and answer with Chain of Thought or thinking tokens? Without this comparison, the benefit of the deterministic execution layer is difficult to establish. In addition, the compatibility checks and the validator are not ablated in isolation, and the paper does not report how often compatibility checks fail.
Questions for authors

Below are my questions in order of importance.

  1. Can you please clarify your contribution and the experimental effect of that contribution compared to GCA, Allocentric Perceiver, and TRACE?
  2. Do the gains carry over to backbones tested in pySpatial and GCA compared to them?
  3. How does performance change when the same extracted scene data is given to the backbone VLM to reason and answer with CoT, without deterministic execution?
  4. How is the router implemented? Can you provide a complete example including the router?
  5. What is the end-to-end latency compared with direct prompting and relevant baselines?
  6. Are the numbers in Table 1 from the GCA paper or from your own experiment?
  7. Which protocol produces the Direct numbers (Answer-only or CoT, Table 3), and with which prompt and Qwen3-VL-4B variant (Instruct or Thinking)?
  8. Why are the results described as "preliminary," and which numbers are final?
Reviewer_MWL3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SpatialCanvas, a training-free framework for spatial reasoning with vision-language models. The method compiles questions into typed spatial programs, explicitly records entities, source views, reference frames, and evidence dependencies, and visualizes intermediate states on an image-bound canvas. Compatibility checks and evidence refinement support the final answer generation.

Using frozen Qwen3-VL-4B, the method improves overall accuracy over direct inference by 2.4, 9.5, and 2.4 percentage points on MMSI-Bench, MindCube-tiny, and OmniSpatial, respectively. The paper also provides cumulative ablations, error-transition analysis, and execution diagnostics.

Critical strengths

S1. The explicit spatial state directly addresses a relevant failure mode.

The method preserves entity identities, source views, and reference frames for intermediate observations, and checks compatibility before composing spatial relations. This provides a concrete mechanism for addressing the conflation of information across views and perspectives.

S2. Canvas feedback and targeted refinement form a coherent pipeline.

The method renders entity locations, directions, and derived relations back onto the source images, providing inspectable intermediate states for subsequent reasoning. Missing or conflicting evidence triggers re-observation and correction, making the reasoning process easier to trace and diagnose.

Critical weaknesses

W1. The contributions of the spatial mechanisms and additional observation and verification rounds are not sufficiently disentangled.

Direct uses a single model call, whereas the full method averages 11.52 calls and allows re-observation and correction. The current results therefore demonstrate that the complete system outperforms single-call inference, but do not sufficiently distinguish the benefits of explicit reference frames and evidence checks from those of additional inference rounds.

Comparisons with multi-round verification methods that do not use these spatial mechanisms, under comparable inference budgets, or ablations that disable the relevant checks while retaining the rest of the pipeline, would more directly establish the contribution of the core mechanisms.

W2. The accounting of generated tokens requires clarification.

Table 3 reports an average of only 2.08 generated tokens per call, whereas Appendix B requests structured records containing multiple fields. It is unclear how this figure corresponds to the complete outputs. Please clarify whether all outputs from observation, refinement, and answer-selection calls are included, so that the efficiency comparison can be interpreted accurately.

Questions for authors
  1. Under comparable inference budgets, are there controlled results showing additional benefits from explicit reference frames and evidence checks over ordinary multi-round observation or verification?

  2. How is the average of 2.08 generated tokens per call in Table 3 calculated? Does it include the complete structured outputs from all observation, refinement, and answer-selection calls?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.