- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper diagnoses the failure cases in video spatial reasoning, from perception gap, wrong measurement, wrong frame/state to missing information. The authors proposed composable skills/tools for 2 types of interfaces (vlm, and coding agent) to address the diagnosed failure cases and showed improved performance on several benchmarks covering both 2 interfaces.
Critical strengths
- The paper is well motivated to diagnose the failure cases in spatial understanding and address those failure cases, because spatial understanding is a high-level task that requires different kinds of perception and reasoning capabilities of the VLM model/agent.
- The results are evaluated on multiple benchmarks and include 2 types of interfaces (VLM and coding agent).
Critical weaknesses
- Failure taxonomy includes 4 failure types, but they are not clearly separated: perception gap, wrong measurement, wrong frame or state, missing information. As a diagnostic contribution, a better and clearer separation and explanation of what each type represents is needed.
- The main paper only has a high-level overview of atomic operator families, with details left in the appendix, this hinders the understanding of how those skills are composed and how these skills actually contribute to spatial video understanding. Those skills are not well presented in Figure 4, where the example gives the perfect chance to showcase the composable skills.
Questions for authors
Clarification requests:
- Can you clarify the frame/memory budget used for the different methods being compared?
- How do you make sure the subtask skills and operators are not over-engineered?
Minor suggestion: SAM3 and DA3 in lines 168 and 169, where they first appear, have no explanation or proper citation.
2 comments
Q1: Frame and memory budgets
We thank the reviewer for this question. Every CROSS gain we report compares a baseline with its CROSS counterpart under the same backbone, frames, perception tools and limits; CROSS adds no frames. The two interfaces use different protocols, so we compare runs only within each interface (Appendix A.1).
| Context augmentation | Code writing (SpatialClaw) | |
|---|---|---|
| Frames shown to the VLM | Uniform: 32 (ReVSI-Tiny), 30 (ReSTI-Tiny, STI-Bench), 64 (ReVSI, VSI-Bench) | 32 key frames (64 on MMSI-Video-Bench, SpatialClaw's released configuration), plus frames the agent requests: up to 48 per session on ReVSI-Tiny and ReSTI-Tiny, and 250 (SpatialClaw's default) on ReVSI, DSI-Bench and MMSI-Video-Bench |
| Frames given to perception | DA3: every decoded frame of the video (median 700–2,300 per scene); SAM3: at most 300 uniformly sampled frames; computed once per scene and cached | DA3: at most 64 frames; SAM3: the video at 1 fps on ReVSI-Tiny and ReSTI-Tiny, and at its native frame rate on ReVSI and MMSI-Video-Bench |
| Prompt length and token limits | Mean prompt on full ReVSI: 9.8k tokens (direct), 37.3k (+ pred. context), 10.2k (+ CROSS); up to 40,960 generated tokens | Up to 32,768 (Qwen) or 131,072 (Gemma) tokens per call; up to 30 code steps |
| Shared by baseline and + CROSS | Backbone, frames, decoding, and the same perception cache | The entire agent configuration |
| Added by CROSS | A derived text block, with unbound entities filtered out | CROSS functions in the tool library |
- CROSS adds no perception calls and shortens the prompt. Its operators run deterministic geometry over the cached perception that "+ pred. context" also uses. When the gate declines, the question uses the video-only prompt; when it accepts, the context is filtered to the entities the question refers to (7.6% of the full context on average). The CROSS prompt therefore averages 10.2k tokens, versus 37.3k for predicted context alone and 9.8k for video only (Qwen3-VL tokenizer).
- Published methods use their own backbones and frame budgets (Frames column of Table 3), so we treat their scores as descriptive context rather than matched comparisons (Appendix E). We will add a Frames column to Tables 4–6, replace SpatialClaw's "64" in Table 3 with "32 shown, up to 64 for reconstruction", and add this table to Appendix A.2.
Q2: Are the skills and operators over-engineered?
We thank the reviewer for this question. Three properties guard against over-engineering: the library is small and derived from the diagnosis rather than from benchmark scores, it transfers to new benchmarks without changing the operators, and its composed skills add measurable value over the operators alone.
- Small, diagnosis-derived basis. 22 operators with single responsibilities compose into 11 skills, and skills share subgraphs: relative distance reuses the absolute-distance graph, and route planning reuses the relative-direction graph (see the table in W2). Each operator addresses a diagnosed failure rather than patching a benchmark question (Section 4.2).
- Fixed before transfer. The operators were frozen after the ReVSI-Tiny and ReSTI-Tiny diagnosis. For DSI-Bench, we composed four new skills from them (camera motion, object motion, distance trend, observer bearing change), raising SpatialClaw from 62.8 to 66.3. On MMSI-Video-Bench, the library was used unchanged, with no new skills, prompts or tuning. The code-writing interface tunes no hyperparameters, and the context-interface gate keeps its gains on scenes not used for tuning (see our response to Reviewer dfpX).
- Composed skills are needed. On MMSI-Video-Bench, supplying the composed skills scores higher than supplying the atomic operators alone (43.0 vs. 39.3).
Minor: SAM3 and DA3
We thank the reviewer. SAM3 (Carion et al., 2026) is an open-vocabulary segmentation and tracking model, which we prompt with the object categories in the question to obtain entity masks and tracks. DA3 (Lin et al., 2026) predicts metric depth and camera poses from video frames. We will introduce both, with citations, where they first appear in Section 3.2.
W1: Separation of the four failure types
We thank the reviewer for this comment. The four types are separated by where the error arises: inaccurate evidence (perception gap), accurate but insufficient evidence (missing information), and sufficient evidence used to compute the wrong quantity (wrong measurement) or in the wrong reference frame or time step (wrong frame or state). Appendix A.5 already assigns each failure by explicit rules (Table 8), but Section 3.3 introduced the types mainly through examples. We will move the following definitions and rules into Section 3.3:
We will also use these four names consistently throughout the paper.
W2: How skills are composed and what they contribute
We thank the reviewer for this suggestion. The full composition is specified in Appendix C (Tables 9 and 10), but the main paper showed only operator families. We will move the composition into Section 4 and redraw Figure 4 around one skill, so that each step is tied to the failure it repairs.
Composition (Section 4). Eleven skills reuse one basis of 22 operators, and each targets a diagnosed failure:
Worked example (Figure 4, relative direction). "Facing the TV from the nightstand, is the cooktop left, right, or behind?" Figure 4 will trace this skill step by step, with each step's value from the run and the baseline error it prevents:
The same graph runs in both interfaces: SpatialClaw calls it as one tool, and the context interface receives its result as a derived block.