acceptodds
Under review as a conference paper at ICLR 2027

From Reasoning Failures to Composable Video Spatial Intelligence

Abstract

Spatial reasoning benchmarks evaluate vision-language models across diverse tasks, but task-level scores do not reveal which underlying capabilities account for success or failure. Each task requires recovering spatial evidence, representing geometry, and reasoning over it. We disentangle these capabilities by comparing predicted and ground-truth spatial context under a shared schema and coordinate contract. This comparison reveals four recurring sources of error: inaccurate perception, missing information in the spatial context, selection of the wrong measurement, and errors in reference frames or in tracking position and orientation. Guided by this diagnosis, we develop CROSS, a training-free library of typed geometric operators and spatial skills that check for available evidence to support reliable video spatial reasoning. The resulting library supplies verified context to non-coding VLMs or callable skills to a SpatialClaw agent. We evaluate on five benchmarks. raises the average score from 55.9% to 60.2% on ReVSI and improves the SpatialClaw result from 62.8% to 66.3% on DSI-Bench. These gains demonstrate that explicit handling of spatial conventions can repair systematic reasoning failures without additional training.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Spatial Reasoning, Failure Diagnosis, Composable Spatial Skills

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9C3GOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper diagnoses the failure cases in video spatial reasoning, from perception gap, wrong measurement, wrong frame/state to missing information. The authors proposed composable skills/tools for 2 types of interfaces (vlm, and coding agent) to address the diagnosed failure cases and showed improved performance on several benchmarks covering both 2 interfaces.

Critical strengths
  1. The paper is well motivated to diagnose the failure cases in spatial understanding and address those failure cases, because spatial understanding is a high-level task that requires different kinds of perception and reasoning capabilities of the VLM model/agent.
  2. The results are evaluated on multiple benchmarks and include 2 types of interfaces (VLM and coding agent).
Critical weaknesses
  1. Failure taxonomy includes 4 failure types, but they are not clearly separated: perception gap, wrong measurement, wrong frame or state, missing information. As a diagnostic contribution, a better and clearer separation and explanation of what each type represents is needed.
  2. The main paper only has a high-level overview of atomic operator families, with details left in the appendix, this hinders the understanding of how those skills are composed and how these skills actually contribute to spatial video understanding. Those skills are not well presented in Figure 4, where the example gives the perfect chance to showcase the composable skills.
Questions for authors

Clarification requests:

  1. Can you clarify the frame/memory budget used for the different methods being compared?
  2. How do you make sure the subtask skills and operators are not over-engineered?

Minor suggestion: SAM3 and DA3 in lines 168 and 169, where they first appear, have no explanation or proper citation.

2 comments

W1: Separation of the four failure types

We thank the reviewer for this comment. The four types are separated by where the error arises: inaccurate evidence (perception gap), accurate but insufficient evidence (missing information), and sufficient evidence used to compute the wrong quantity (wrong measurement) or in the wrong reference frame or time step (wrong frame or state). Appendix A.5 already assigns each failure by explicit rules (Table 8), but Section 3.3 introduced the types mainly through examples. We will move the following definitions and rules into Section 3.3:

TypeWhere the error arisesHow a failure is assignedExample
Perception gapThe predicted evidence is inaccurate.Accuracy drop from GT to predicted context on the same questions.ReVSI-Tiny average: 76.2 with GT context vs. 50.0 with predicted context.
Missing informationAccurate evidence lacks a variable the answer needs.The target cannot be recovered from the GT context.Room size: no floor footprint; 25 of 27 errors equal scene length × width.
Wrong measurementThe evidence suffices, but the model computes a different quantity.The prediction matches an alternative quantity computed from the same context.Absolute distance: 225 of 227 errors equal the center-to-center distance.
Wrong frame or stateThe evidence suffices, but the model uses the wrong reference frame or time step.The prediction matches the target after a frame or state transformation.Relative direction: 23 of 25 errors are left–right mirrors, reversed facing, or fixed scene axes.
  • Assignment and overlap. The perception gap is measured between the predicted- and GT-context arms; the other three types classify individual failures under GT context, where perception is held fixed. For each such failure, we first test whether a deterministic computation over the GT context reproduces the target. Failures whose target cannot be reproduced are missing information; for the rest, we evaluate the subtask's measurement or frame/state rules and assign the first match. A match indicates consistency with a mechanism, not a unique cause, so the types are not mutually exclusive; overlapping matches are reported (e.g., the three pose-estimation rules match 22, 15 and 13 errors).
  • The separation predicts which failures computation can repair. Under the same GT context, CROSS raises absolute distance from 58.4 to 100.0 and relative direction from 52.8 to 100.0, but leaves room size (56.7 → 56.0) and 3D grounding (74.6 → 75.1) essentially unchanged (Table 2).

We will also use these four names consistently throughout the paper.

W2: How skills are composed and what they contribute

We thank the reviewer for this suggestion. The full composition is specified in Appendix C (Tables 9 and 10), but the main paper showed only operator families. We will move the composition into Section 4 and redraw Figure 4 around one skill, so that each step is tied to the failure it repairs.

Composition (Section 4). Eleven skills reuse one basis of 22 operators, and each targets a diagnosed failure:

SkillOperator graphRepairs
Absolute distanceclean_geometry → aggregate_geometry → surface_distance → reduce_set(min)Wrong measurement
Relative distanceabsolute-distance graph per candidate → rank_with_marginWrong measurement
Relative directionrelative_vector → build_ego_basis → project_local → classify_sectorWrong frame or state
Route planningrelative-direction graph + advance_agent at each waypoint → match_candidateWrong frame or state
Pose estimationrelative_se3 → compose_se3 → pose_distance → rank_with_marginWrong frame or state
Room areafloor_polygon → polygon_area; abstains when only a global envelope is availableMissing information (detected)

Worked example (Figure 4, relative direction). "Facing the TV from the nightstand, is the cooktop left, right, or behind?" Figure 4 will trace this skill step by step, with each step's value from the run and the baseline error it prevents:

StepValue in the runBaseline error it prevents
Role bindinganchor = nightstand, facing = TV, target = cooktop; masks verified before measuringSpatialClaw propagates empty masks as NaN angles and still answers "right"
relative_vector → build_ego_basisforward = TV − nightstand, up = (0, 0, 1)Direct video uses room-level instead of observer-relative directions
project_local → classify_sectorcooktop 2.27 m forward and 3.71 m left; bearing +58.49° → left (positive = left)Left–right mirroring
Gate (context interface)relative uncertainty 0.034 ≤ 0.06 → applyAnswering from unstable geometry

The same graph runs in both interfaces: SpatialClaw calls it as one tool, and the context interface receives its result as a derived block.

Q1: Frame and memory budgets

We thank the reviewer for this question. Every CROSS gain we report compares a baseline with its CROSS counterpart under the same backbone, frames, perception tools and limits; CROSS adds no frames. The two interfaces use different protocols, so we compare runs only within each interface (Appendix A.1).

Context augmentationCode writing (SpatialClaw)
Frames shown to the VLMUniform: 32 (ReVSI-Tiny), 30 (ReSTI-Tiny, STI-Bench), 64 (ReVSI, VSI-Bench)32 key frames (64 on MMSI-Video-Bench, SpatialClaw's released configuration), plus frames the agent requests: up to 48 per session on ReVSI-Tiny and ReSTI-Tiny, and 250 (SpatialClaw's default) on ReVSI, DSI-Bench and MMSI-Video-Bench
Frames given to perceptionDA3: every decoded frame of the video (median 700–2,300 per scene); SAM3: at most 300 uniformly sampled frames; computed once per scene and cachedDA3: at most 64 frames; SAM3: the video at 1 fps on ReVSI-Tiny and ReSTI-Tiny, and at its native frame rate on ReVSI and MMSI-Video-Bench
Prompt length and token limitsMean prompt on full ReVSI: 9.8k tokens (direct), 37.3k (+ pred. context), 10.2k (+ CROSS); up to 40,960 generated tokensUp to 32,768 (Qwen) or 131,072 (Gemma) tokens per call; up to 30 code steps
Shared by baseline and + CROSSBackbone, frames, decoding, and the same perception cacheThe entire agent configuration
Added by CROSSA derived text block, with unbound entities filtered outCROSS functions in the tool library
  • CROSS adds no perception calls and shortens the prompt. Its operators run deterministic geometry over the cached perception that "+ pred. context" also uses. When the gate declines, the question uses the video-only prompt; when it accepts, the context is filtered to the entities the question refers to (7.6% of the full context on average). The CROSS prompt therefore averages 10.2k tokens, versus 37.3k for predicted context alone and 9.8k for video only (Qwen3-VL tokenizer).
  • Published methods use their own backbones and frame budgets (Frames column of Table 3), so we treat their scores as descriptive context rather than matched comparisons (Appendix E). We will add a Frames column to Tables 4–6, replace SpatialClaw's "64" in Table 3 with "32 shown, up to 64 for reconstruction", and add this table to Appendix A.2.

Q2: Are the skills and operators over-engineered?

We thank the reviewer for this question. Three properties guard against over-engineering: the library is small and derived from the diagnosis rather than from benchmark scores, it transfers to new benchmarks without changing the operators, and its composed skills add measurable value over the operators alone.

  • Small, diagnosis-derived basis. 22 operators with single responsibilities compose into 11 skills, and skills share subgraphs: relative distance reuses the absolute-distance graph, and route planning reuses the relative-direction graph (see the table in W2). Each operator addresses a diagnosed failure rather than patching a benchmark question (Section 4.2).
  • Fixed before transfer. The operators were frozen after the ReVSI-Tiny and ReSTI-Tiny diagnosis. For DSI-Bench, we composed four new skills from them (camera motion, object motion, distance trend, observer bearing change), raising SpatialClaw from 62.8 to 66.3. On MMSI-Video-Bench, the library was used unchanged, with no new skills, prompts or tuning. The code-writing interface tunes no hyperparameters, and the context-interface gate keeps its gains on scenes not used for tuning (see our response to Reviewer dfpX).
  • Composed skills are needed. On MMSI-Video-Bench, supplying the composed skills scores higher than supplying the atomic operators alone (43.0 vs. 39.3).

Minor: SAM3 and DA3

We thank the reviewer. SAM3 (Carion et al., 2026) is an open-vocabulary segmentation and tracking model, which we prompt with the object categories in the question to obtain entity masks and tracks. DA3 (Lin et al., 2026) predicts metric depth and camera poses from video frames. We will introduce both, with citations, where they first appear in Section 3.2.

Reviewer_CwHEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CROSS, a training-free library of geometric operators and spatial skills designed to improve spatial reasoning in vision-language models. The authors systematically analyze spatial reasoning failures by comparing predicted and ground-truth spatial contexts, identifying four major sources of errors related to perception, missing information, measurement selection, and spatial reference conventions. Based on these findings, CROSS provides explicit geometric operations and verified spatial context to support reliable spatial reasoning in both non-coding VLMs and agent-based systems. Experiments across several VLM spatial reasoning benchmarks demonstrate consistent improvements.

Critical strengths
  1. The paper provides a systematic analysis of spatial reasoning failures in VLMs, identifying several meaningful sources of errors and offering useful insights into their underlying limitations.
  2. The proposed CROSS introduces a training-free library of geometric operators and spatial skills, providing a practical way to enhance spatial reasoning capabilities in VLMs without additional training.
  3. CROSS demonstrates consistent performance improvements across multiple spatial reasoning benchmarks, supporting the effectiveness and general applicability of the proposed approach.
Critical weaknesses
  1. Although CROSS achieves consistent improvements across several benchmarks, its gain on VSI-Bench is relatively modest (+1.6 points), and the resulting performance remains below several existing models. Moreover, the context-augmentation experiments are conducted only on Qwen3-VL-32B-Thinking, leaving its effectiveness on other VLM backbones, particularly non-thinking models, insufficiently explored.
  2. The reliability and generalizability of CROSS may be limited by its heuristic gating mechanism, which cannot detect all incorrect operator outputs and relies on room-scale assumptions that may not generalize well to outdoor scenes.
Questions for authors

Please refer to the weaknesses.

4 comments

W1: Modest VSI-Bench gain and other VLM backbones

We thank the reviewer for this comment. The concerns are the modest gain on VSI-Bench and the use of a single backbone for context augmentation. On VSI-Bench, the gain is confined to the subtasks CROSS covers, and the models ranked above ours there score below ours on ReVSI. On ReVSI, CROSS improves all four backbones we now test, including three non-thinking models and a second model family.

VSI-Bench.

  • The gain comes from the subtasks CROSS covers. CROSS improves absolute distance (+3.2), room size (+4.2) and relative direction (+5.1), and leaves four of the eight subtasks unchanged, including appearance order, which no operator covers (Table 11). Relative direction also has less headroom: direct inference already scores 57.5 on VSI-Bench versus 41.6 on ReVSI, while CROSS reaches similar accuracy on both (62.6 and 64.3).
  • Models above ours on VSI-Bench score below ours on ReVSI. The spatially fine-tuned VST-7B-SFT and Cambrian-S-7B reach 65.2 and 67.5 on VSI-Bench but 46.4 and 49.1 on ReVSI, where our training-free pipeline reaches 60.2 (Tables 3, 4). ReVSI re-annotates the object and geometry errors of VSI-Bench and checks answerability, so we use it as our primary benchmark.

Other backbones. Full ReVSI (6,517 questions, 64 frames, the same gate for all backbones):

BackboneSettingCnt.Abs. dist.Obj. sizeRoom sizeRel. dist.Rel. dir.RouteAvg.
Qwen3-VL-32B-Thinking (Table 3)Direct63.558.668.750.656.641.651.655.9
+ pred. context45.550.155.869.259.242.942.352.2
+ pred. context + CROSS64.563.667.852.659.364.349.160.2
Qwen3-VL-32B-InstructDirect60.761.267.160.856.138.846.655.9
+ pred. context49.852.349.472.558.837.938.451.3
+ pred. context + CROSS60.765.467.160.856.852.844.458.3
Qwen3.6-27B (non-thinking)Direct60.248.164.445.656.536.331.548.9
+ pred. context50.745.253.469.158.956.832.652.4
+ pred. context + CROSS60.257.264.445.656.961.140.955.2
Gemma-4-31B (non-thinking)Direct61.342.871.557.551.931.338.050.6
+ pred. context47.744.843.068.759.562.445.953.1
+ pred. context + CROSS61.353.871.557.552.561.545.957.7
  • CROSS improves every backbone, most on the diagnosed failures. It raises the average by 2.4 to 7.1 points over direct inference and exceeds predicted context alone on every backbone. The largest gains are on absolute distance (+4.2 to +11.0) and relative direction (+14.0 to +30.2), the tasks most affected by wrong measurement and wrong frame or state in our diagnosis.
  • Where predicted context alone scores higher, the gate falls back to video by design. On room size, the context provides no complete floor, so the gate abstains rather than estimate an unsupported area; on relative distance, it falls back when a candidate object is missing from the reconstruction (Table 10). The same fallback protects counting and object size, where predicted context alone loses 9.5 to 28.5 points. Route planning gains on Qwen3.6 and Gemma but not on the Qwen3-VL models, consistent with the gate's false accepts discussed in W2.

We will add these results to the revision.

W2: Reliability and generality of the heuristic gate

We thank the reviewer for this comment. The concern is whether the heuristic gate limits the reliability and generality of CROSS. The code-writing interface uses no gate and improves on every benchmark we test, including outdoor videos; in the context-augmentation interface, the gate reliably separates correct from incorrect derivations on held-out data.

  • CROSS in the code-writing interface does not depend on the gate. The agent reads the validity codes and decides itself whether to use a result. It improves SpatialClaw on ReVSI (55.6 → 58.9), DSI-Bench (62.8 → 66.3) and MMSI-Video-Bench (42.0 → 43.0). On the 236 outdoor questions of MMSI-Video-Bench, it raises SpatialClaw from 35.2 to 39.8, while calling CROSS less often than indoors (39% vs. 65% of sessions).
  • The gate keeps accurate derivations and filters out unreliable ones. On held-out ReVSI (4,966 questions from scenes not used for tuning), derivations the gate accepts are correct 79.8% of the time, versus 44.1% for those it rejects; for absolute distance, 85.4% versus 10.4%. On accepted questions, CROSS improves over direct inference by +15.4 on absolute distance and +35.8 on relative direction.
  • Limits. The gate checks the derivation (role binding, degeneracy, frame consistency, boundary and uncertainty margins; Appendix C.3), not the reconstruction, so systematic reconstruction errors can pass; 20.2% of accepted derivations are wrong. The gate's limits, calibrated on room-scale scenes, still help on tabletop scenes (+3.2 on STI-Bench) but do not transfer to driving scenes. We will state this in Appendix E.

Thanks for your response. My concerns have been resolved. I decide to maintain my rating.

Thank you for maintaining your positive rating and for confirming that your concerns are resolved. Your questions led to new results that strengthen the paper, and we will add them to the revision.

Reviewer_dfpXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes the spatial reasoning failures of Vision-Language Models (VLMs) in video domains and proposes CROSS, a training-free framework featuring a composable library of geometric operators and spatial skills. By introducing context augmentation and tool-use interfaces, the framework resolves key geometric ambiguities and coordinate confusion. Its effectiveness is empirically validated across multiple benchmarks (including ReVSI, RESTI, and DSI-Bench), demonstrating consistent performance gains.

Critical strengths

Clear disentanglement of perception vs. reasoning: Many existing works broadly attribute failures in visual-spatial tasks to ``insufficient spatial reasoning.'' In contrast, this paper conducts controlled comparative experiments using cpredc_{\text{pred}} and cgtc_{\text{gt}}, clearly disentangling the perception ceiling from the reasoning residual. This is an aspect that I find lacking in current literature; the authors' approach is structured, meticulous, and meaningful.

Critical weaknesses

Rule-based routing mechanism: The routing mechanism is rule-based and appears to rely on benchmark task labels and fixed pattern matching. In realistic, open-ended scenarios where question forms vary widely, such rule-based routing is prone to failure. This issue limits whether the method is practically meaningful in real-world settings.

Questionable novelty and research contribution: Is this work evaluating and enhancing the spatial cognition of VLMs, or is it essentially evaluating an external 3D geometric processing library? The proposed approach resembles an engineering solution rather than a research methodology, lacking novelty and broader inspiration for the community.

Questions for authors

How does the method perform on MMSI-Video-Bench? This benchmark lacks fixed question templates and would test the generalization ability of the method.

Is there any overlap between the hyperparameter-tuning data and the evaluation data?

4 comments

W1 & Q1: Rule-based routing and generalization to MMSI-Video-Bench

We thank the reviewer for suggesting MMSI-Video-Bench. The concern is whether CROSS depends on benchmark-specific routing and therefore fails on open-ended questions. This dependence applies to our context-augmentation interface, but not to the code-writing interface, where the agent reads the raw question and decides itself whether and how to use CROSS. On MMSI-Video-Bench, the code-writing interface uses the CROSS library unchanged and gives a modest overall gain over SpatialClaw (+1.0), with gains on spatial-construction questions and losses on cross-video and planning questions.

Context augmentation relies on task labels. We agree on this interface: its router selects a skill from the task label and extracts semantic roles with fixed patterns (Appendix C.3). It serves as a controlled setting for testing which diagnosed failures explicit computations repair, not as an interface for free-form questions, and it does not transfer to them: on MMSI-Video-Bench, our frozen patterns parse only 1 of 934 single-video questions. We will state this limitation in Section 4.1 and Appendix E.

The code-writing interface has no router. The agent receives only the raw question and decides whether to call a composed skill, an atomic operator, or neither, and with which arguments (Appendix C.4). The code-writing results in the paper already use this setting (Tables 2, 3, 6; e.g., ReVSI 55.6 → 58.9, DSI-Bench 62.8 → 66.3).

MMSI-Video-Bench. We use Gemma-4-31B on all 1,106 questions, with the CROSS library unchanged (no MMSI-specific skills, prompts or tuning) and the same agent configuration for both arms. On SpatialClaw's 1,000-question subset, our baselines reproduce its reported numbers (36.3 vs. 36.9 without tools; 42.1 vs. 41.6 with SpatialClaw).

Gemma-4-31B, all 1,106 questionsAcc.
Backbone only (no tools)36.8
SpatialClaw42.0
SpatialClaw + atomic operators only39.3
SpatialClaw + CROSS (operators + skills)43.0
Category (#Q)BackboneSpatialClawSpatialClaw + CROSS
Spatial Construction (464)39.748.752.6
Motion Understanding (264)41.745.145.1
Planning (124)32.336.333.1
Prediction (82)24.422.030.5
Cross-Video (172)30.833.127.3
  • The agent uses CROSS selectively, deciding from the question alone: in 59.7% of all sessions, 79% of spatial-construction questions, and 26% of cross-video questions.
  • Gains appear where the operators apply. Spatial construction rises from 48.7 to 52.6, mainly on instance–instance (+14.5) and camera–instance (+10.1) relations, which require the frame conversions and metric computations in our direction and distance skills.
  • Composed skills outperform atomic operators alone: 43.0 vs. 39.3 on the same questions.
  • Limits. Cross-video accuracy drops from 33.1 to 27.3 and planning from 36.3 to 33.1. Cross-video questions require correspondences across two videos, which no current operator provides; this is a plausible cause that we have not yet confirmed by error analysis.

We will add these results, the per-category breakdown, the invocation statistics, and a failure analysis of the two declining categories to the revision.

W2: Novelty and research contribution

We thank the reviewer for recognizing our disentanglement of perception and reasoning. The concern is whether the paper studies VLM spatial reasoning or mainly evaluates an external geometry library. Our contribution is the diagnosis of VLM failures; CROSS is the intervention that tests it, showing which failures are computational and can be repaired by explicit, reusable procedures. We claim neither new geometric algorithms nor an improved internal spatial ability of the model (Appendix E).

  • Accurate evidence is not enough. With GT context, Qwen3-VL-32B-Thinking scores only 58.4 on absolute distance and 52.8 on relative direction; 225 of 227 absolute-distance errors use the center-to-center instead of the closest-surface distance. Supplying the correct computation over the same context fixes both tasks (100.0) but not room size or 3D grounding, where the context lacks the needed information (Table 2).
  • Geometric tools alone do not fix these failures. SpatialClaw already has DA3 reconstruction, SAM3 segmentation and code execution. Adding CROSS keeps these tools fixed and changes only the procedures, raising ReVSI from 55.6 to 58.9 and DSI-Bench from 62.8 to 66.3.
  • The computations transfer. The operators, derived from the ReVSI-Tiny and ReSTI-Tiny diagnosis, stayed frozen; for DSI-Bench we only composed four new skills from them, and on MMSI-Video-Bench the library was used unchanged (W1).

The broader insight is that video spatial reasoning involves two separable problems, recovering evidence and computing with it under the correct measurement, frame and state, and that VLMs should not have to rediscover the second for every question. We will state this framing in the introduction and contributions.

Q2: Overlap between tuning and evaluation data

We thank the reviewer for this question. For the context-augmentation interface, there was overlap: the gate thresholds were tuned on development scenes that are also part of the reported evaluation. After these scenes are removed, the gains remain; the code-writing interface tunes no hyperparameters.

  • What was tuned. Only the gate's skill-specific limits in the context-augmentation interface, selected by grid search on 30% of the scenes of the diagnostic subsets (76 of 253 ReVSI-Tiny scenes and 45 of 148 ReSTI-Tiny scenes). These scenes are included in Table 2, as stated in Appendix C.3, and also in the full ReVSI, VSI-Bench and STI-Bench evaluations (Tables 3, 4 and 5).
  • Held-out results. We rescored the same runs, with the same thresholds, after removing every question from the development scenes. The CROSS gain over direct inference is +4.5 on held-out ReVSI (+4.3 on all questions), +1.8 on held-out VSI-Bench (+1.6), and +3.4 on held-out STI-Bench (+3.7).
BenchmarkEvaluation set#QDirect+ pred. context+ CROSS
ReVSIAll (Table 3)6,51755.952.260.2
Development scenes removed4,96655.852.760.3
VSI-BenchAll (Tables 4, 11)5,05159.851.261.4
Development scenes removed3,94360.151.961.9
STI-BenchAll (Table 5)2,05641.637.245.3
Development scenes removed1,80441.737.845.1
  • Code-writing interface. No hyperparameters are tuned: SpatialClaw + CROSS keeps SpatialClaw's configuration, and the agent uses no gate. The DSI-Bench and MMSI-Video-Bench results therefore involve no tuning.

In the revision, we will report the held-out results as the primary context-augmentation numbers and describe the development split in the main text.

Thank you to the authors for the additional experimental results and clarifications.

I believe the rule-based routing remains a clear limitation of this method. Furthermore, the benchmarks used in the paper are highly templated, and the development and evaluation scenes clearly share the same distribution; such a hyperparameter tuning setting may not be viable in the real world.

Therefore, I maintain my score.

Thank you for the follow-up. CROSS requires neither rule-based routing nor hyperparameter tuning, and its evaluation is not limited to templated benchmarks; our responses show this with experiments, including on MMSI-Video-Bench, the template-free benchmark you proposed to test generalization.

  • CROSS does not depend on rule-based routing (our response to W1 & Q1). In the code-writing interface, the agent reads the raw question and decides itself whether to call CROSS, and it improves SpatialClaw on MMSI-Video-Bench. Only the context-augmentation interface uses the router. We use this interface both for the diagnosis and as the evaluation protocol for context augmentation.
  • Our evaluation is not limited to templated benchmarks (our response to W1 & Q1). We evaluate on five benchmarks, chosen for both annotation quality and comparability with published work, and two of them are not template-based: STI-Bench, whose questions are generated by MLLMs from annotations and filtered by humans, and MMSI-Video-Bench, whose questions are designed by experts. CROSS improves results on both. The other three are VSI-Bench, the most widely used benchmark for video spatial reasoning; ReVSI, which rebuilds its annotations and questions with human verification and checks answerability under each frame budget; and DSI-Bench, which covers dynamic scenes.
  • CROSS does not depend on hyperparameter tuning (our responses to W1 & Q1 and Q2). The code-writing interface tunes nothing, so its results on MMSI-Video-Bench use no development data. Only the context-interface gate was tuned, and its gains hold on held-out scenes.

None of these points identifies a specific gap in our results or proposes a further test, whereas our responses answer each with direct experimental evidence. We hope this clarifies that CROSS requires neither rule-based routing nor hyperparameter tuning, and that its evaluation extends beyond templated benchmarks. We would appreciate it if you would reconsider your assessment of our contribution. Thanks.

Comments on the paper

2 comments

Dear Reviewers,

Thank you for your time and effort in reviewing this paper.

As the discussion deadline is approaching, I would appreciate it if you could take a moment to review the authors' responses and let they know whether any concerns remain or require further clarification.

Best,

AC

AuthorsOpenReview ↗

Summary of the discussion

We thank the AC and all reviewers for their time and constructive feedback. Reviewer CwHE maintains a positive rating and has confirmed that our responses resolved their concerns. Reviewers dfpX and 9C3G, as the discussion period closes in less than a day, we would be grateful if you could let us know whether our responses address your concerns, or whether any question remains that we can still clarify.

Reviewers noted that our controlled disentanglement of perception and reasoning addresses an aspect "lacking in current literature" (dfpX), that the failure analysis offers "useful insights" into the limitations of VLMs (CwHE) and is well motivated (9C3G), and that CROSS is a practical, training-free way to improve spatial reasoning (CwHE), evaluated across multiple benchmarks and two interfaces (CwHE, 9C3G).

Three new experiments substantially strengthen our central claim. The paper shows that VLMs fail at video spatial reasoning not only because of perception: given accurate evidence, they still compute the wrong quantity or use the wrong frame or state, and explicit, reusable procedures repair these failures. The new experiments test this claim on a new benchmark, on new backbones, and on held-out scenes.

  • New benchmark: MMSI-Video-Bench (dfpX W1, Q1; CwHE W2). On open-ended questions without fixed templates, the code-writing interface uses the CROSS library unchanged, with no router or gate, and improves SpatialClaw from 42.0 to 43.0 overall, from 48.7 to 52.6 on spatial construction, and from 35.2 to 39.8 on outdoor videos. Composed skills outperform atomic operators alone (43.0 vs. 39.3). The procedures therefore transfer beyond the benchmarks they were derived from.
  • New backbones (CwHE W1). On full ReVSI, CROSS improves all four backbones we test, including three non-thinking models and a second model family, by 2.4 to 7.1 points. The largest gains are on absolute distance (+4.2 to +11.0) and relative direction (+14.0 to +30.2), the tasks most affected by wrong measurement and wrong frame or state in our diagnosis, so the repair of the diagnosed failures is not specific to one model.
  • Held-out scenes (dfpX Q2, 9C3G Q2). After removing every scene used to tune the context-interface gate, CROSS still improves over direct inference by +4.5 on ReVSI, +1.8 on VSI-Bench and +3.4 on STI-Bench. The code-writing interface tunes no hyperparameters, and the 22 operators stayed frozen on new benchmarks. The gains therefore do not come from tuning on the evaluation data.

Other concerns.

  • CROSS does not require the router or the gate (dfpX W1, CwHE W2). Both belong to the context-augmentation interface. The code-writing interface uses neither and improves SpatialClaw on ReVSI (55.6 → 58.9), DSI-Bench (62.8 → 66.3) and MMSI-Video-Bench. Within the context interface, derivations the gate accepts on held-out ReVSI are correct 79.8% of the time, versus 44.1% for those it rejects.
  • The gains come from the procedures, not from the geometric tools (dfpX W2). SpatialClaw already has the same reconstruction, segmentation and code execution, and adding CROSS changes only the procedures. With GT context, the backbone still scores only 58.4 on absolute distance; the correct computation over the same context reaches 100.0.
  • VSI-Bench (CwHE W1). The gain is confined to the subtasks CROSS covers, and the models ranked above ours on VSI-Bench score below ours on ReVSI.

Presentation and clarification (9C3G W1, W2, Q1, minor). The remaining points ask for clearer definitions of the four failure types, the skill composition in Section 4 and Figure 4, the frame and memory budgets, and citations for SAM3 and DA3. Most of this material is already in the appendix (Tables 7–10), and our individual replies answer each point; for example, every CROSS gain compares runs with the same backbone, frames, perception tools and limits. Addressing these points changes the presentation, not the method or the results.

We will add the new results to the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.