- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces SpaceConflict, a benchmark of about 24K spatial claims built from existing datasets. Each claim is labeled Supported, Contradictory or Unknown, and claims are organized into four levels: local fact binding (L1), relational composition (L2), cross-observation consistency (L3), and transformation-conditioned state judgment (L4). Using L4-style state trajectories, the authors query the same world under three conditions and further discover the availability–utilization gap. Finally, they propose Operational State Supervision (OSS), auxiliary fine-tuning examples that target canonical state tuples and trajectory states, and report large PairAcc gains on the benchmark’s test split, particularly on L3/L4.
Critical strengths
- Diagnostic idea: Querying the same trajectory under direct, full and explicit conditions is a clear way to separate reporting a state from using it.
- Diagnostic depth and control: the diagnostics include controls aimed at specific alternative explanations. The controls are informative although partial.
- Scientific Transparency: the paper documents large set of results, including negative and boundary findings.
Critical weaknesses
- Figures are internally inconsistent. Figure 1 is supposed to present observations of one same office, but views disagree with each other, e.g. Chair A and chair B are mirrored in L1 relative to both top views. The desk’s footprint differs between the top-down reconstruction and the L2 top view. The monitor changes position. In L3 the labeled “chair_A” is a shelf. In figure 2, despite it says, “All input views depict the same world state S0”, the two top views and perspective view again disagrees with each other. For a spatial reasoning benchmark, especially whose L3 level is about cross-view consistency, such poor illustration undermines the quality of the paper.
- Labels are inherited from source annotations, and the paper reports no audit of label correctness relative to the media the model sees. The L4 example supplies images without specifying a frame. For viewpoint-dependent relations, such items may be ill-posed. Frame-invariant claims and frame-dependent relational claims are pooled in every reported number.
- The related work is incomplete, making the novelty relative to existing work unsure. Failing to compose individually known facts is well documented in language models. The related work cites only spatial benchmarks. The paper should position the gap as an instance of this broader phenomenon and say what is specific to the multimodal spatial setting.
- Minor issues:
a. The benchmark evaluation (Table 3) is hard to interpret:
-
No trivial baselines are given.
-
The Qwen3-VL-Thinking models are run with thinking disabled and a 512-token cap. The far below chance on L3/L4 score likely reflects output-format failure.
b. There are several inconsistencies:
- The abstract states 23,196 inputs; Table 1 state 24,396.
- Sec. 5.1 refers to metrics “in Sec. 3.1”. in fact, they are defined in Sec. 2.3.
Questions for authors
- What is the label accuracy, per level, on a human-audited random sample judged only against the model-visible media and prompt? What fraction of Supported/Contradictory items would annotators label Unknown?
- For frame-dependent claims, how is the reference frame communicated to the model?
- Do the OSS gains transfer to an external spatial benchmark such as VSI-Bench or SPAR-Bench?
- How does the availability–utilization gap differ from the compositionality gap documented in text-only models?