acceptodds
Withdrawn from ICLR 2027

Seeing, Saying, but Not Using: From Reportable Spatial Facts to Usable States in Multimodal Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A multimodal large language model that correctly reports a spatial fact does not necessarily use that fact in subsequent reasoning. To study this distinction, we introduce SpaceConflict, a benchmark of 24,196 inputs for the construction and use of spatial state. Under a unified Supported/Contradictory/Unknown judgment interface, it covers local fact binding (L1), relational composition (L2), cross-observation consistency (L3), and state judgment under transformation (L4). Posing a direct-state query, a full-transformation query, and an explicit-initial-state query on the same world reveals an availability–utilization gap: models recover the initial state from visual evidence yet fail when that state must drive a transformation. For Qwen3.5-9B, 50 of 100 sequences with a correctly recovered initial state fail the full transformation, and supplying the state explicitly repairs all 50; the gap narrows with scale but does not close. We therefore propose Operational State Supervision (OSS), which supervises task-relevant spatial states and their transformation trajectories and aligns shared facts across contexts. OSS improves paired accuracy on matched judgments most on L3 and L4, the levels that depend on organizing and using state. Evaluating multimodal spatial reasoning thus requires asking not only whether a model can see and state a spatial fact, but whether that fact becomes a usable state in subsequent computation. Code and benchmark artifacts are available at https://anonymous.4open.science/r/spaceconflict-iclr27/.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Spatial Reasoning, Spatial State Utilization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_k21POpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SpaceConflict, a benchmark of about 24K spatial claims built from existing datasets. Each claim is labeled Supported, Contradictory or Unknown, and claims are organized into four levels: local fact binding (L1), relational composition (L2), cross-observation consistency (L3), and transformation-conditioned state judgment (L4). Using L4-style state trajectories, the authors query the same world under three conditions and further discover the availability–utilization gap. Finally, they propose Operational State Supervision (OSS), auxiliary fine-tuning examples that target canonical state tuples and trajectory states, and report large PairAcc gains on the benchmark’s test split, particularly on L3/L4.

Critical strengths
  1. Diagnostic idea: Querying the same trajectory under direct, full and explicit conditions is a clear way to separate reporting a state from using it.
  2. Diagnostic depth and control: the diagnostics include controls aimed at specific alternative explanations. The controls are informative although partial.
  3. Scientific Transparency: the paper documents large set of results, including negative and boundary findings.
Critical weaknesses
  1. Figures are internally inconsistent. Figure 1 is supposed to present observations of one same office, but views disagree with each other, e.g. Chair A and chair B are mirrored in L1 relative to both top views. The desk’s footprint differs between the top-down reconstruction and the L2 top view. The monitor changes position. In L3 the labeled “chair_A” is a shelf. In figure 2, despite it says, “All input views depict the same world state S0”, the two top views and perspective view again disagrees with each other. For a spatial reasoning benchmark, especially whose L3 level is about cross-view consistency, such poor illustration undermines the quality of the paper.
  2. Labels are inherited from source annotations, and the paper reports no audit of label correctness relative to the media the model sees. The L4 example supplies images without specifying a frame. For viewpoint-dependent relations, such items may be ill-posed. Frame-invariant claims and frame-dependent relational claims are pooled in every reported number.
  3. The related work is incomplete, making the novelty relative to existing work unsure. Failing to compose individually known facts is well documented in language models. The related work cites only spatial benchmarks. The paper should position the gap as an instance of this broader phenomenon and say what is specific to the multimodal spatial setting.
  4. Minor issues:

a. The benchmark evaluation (Table 3) is hard to interpret:

  • No trivial baselines are given.

  • The Qwen3-VL-Thinking models are run with thinking disabled and a 512-token cap. The far below chance on L3/L4 score likely reflects output-format failure.

b. There are several inconsistencies:

  • The abstract states 23,196 inputs; Table 1 state 24,396.
  • Sec. 5.1 refers to metrics “in Sec. 3.1”. in fact, they are defined in Sec. 2.3.
Questions for authors
  1. What is the label accuracy, per level, on a human-audited random sample judged only against the model-visible media and prompt? What fraction of Supported/Contradictory items would annotators label Unknown?
  2. For frame-dependent claims, how is the reference frame communicated to the model?
  3. Do the OSS gains transfer to an external spatial benchmark such as VSI-Bench or SPAR-Bench?
  4. How does the availability–utilization gap differ from the compositionality gap documented in text-only models?
Reviewer_TvAoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SpaceConflict, a multimodal benchmark designed to evaluate whether multimodal large language models (MLLMs) can not only recognize and report spatial facts but also use those facts in subsequent reasoning. The benchmark organizes spatial reasoning into four structural levels: local fact binding (L1), relational composition (L2), cross-observation consistency (L3), and transformation-conditioned state judgment (L4). It constructs matched Supported/Contradictory claims under shared visual evidence and task contexts, together with Unknown examples.

A central contribution is the investigation of an availability–utilization gap. The authors compare direct-state queries, full-transformation queries, and explicit-initial-state queries on matched worlds. Their results indicate that models can correctly report an initial spatial state when queried independently but fail to solve subsequent transformations requiring that state. Providing the correct initial state explicitly can recover many of these failures.

Motivated by this observation, the paper proposes Operational State Supervision (OSS). OSS represents spatial facts in a canonical format and introduces two auxiliary supervision objectives: cross-context state alignment and state-trajectory supervision. These objectives are combined with conventional answer supervision during fine-tuning, without changing the inference architecture.

Critical strengths

S1. A meaningful distinction between spatial fact recognition and downstream utilization.

The paper raises an important question that is not adequately captured by conventional end-to-end spatial reasoning accuracy. A model may correctly answer a direct spatial query yet fail when the same information becomes a prerequisite for a subsequent transformation. The distinction between fact reportability and downstream usability provides a useful perspective for understanding multimodal reasoning failures.

S2. A structured and relatively comprehensive diagnostic framework.

SpaceConflict organizes several spatial reasoning requirements under a unified three-way judgment interface. Its matched Supported/Contradictory construction and source-grounded provenance records make it possible to analyze model performance beyond aggregate accuracy. The inclusion of Unknown examples also recognizes that missing visual evidence should not automatically be interpreted as contradiction.

S3. Controlled behavioral experiments provide useful evidence.

The Direct-state, Full-transition, and Explicit-state conditions provide an informative behavioral comparison. Additional symbolic-program controls, prompt-wording experiments, and non-count relation tasks help narrow possible explanations for the observed failures. Importantly, the paper also reports negative findings and acknowledges limitations of its causal intervention experiments.

S4. OSS is a simple and practically plausible training intervention.

Supervising task-relevant states and transformation outcomes is a reasonable way to provide richer learning signals than final-answer supervision alone. OSS uses structured targets derived from source annotations and deterministic transformations, while preserving the original inference interface. Its simplicity and compatibility with conventional fine-tuning are practical advantages.

Critical weaknesses

W1. Serious numerical inconsistencies undermine the reliability of the benchmark results.

Several reported results are inconsistent with their corresponding numerators and denominators. For example, Table 20 reports Qwen3.5-9B ClaimAcc as 59.26%, while the displayed fraction 14,339/24,396 corresponds to approximately 58.78%. Similarly, its Unknown recall is reported as 51.70%, whereas 1,189/2,500 corresponds to 47.56%. In Table 22, the reported L2 ClaimAcc for Qwen3.5-9B is 89.81%, whereas 2,177/2,624 corresponds to 82.96%.

These discrepancies cannot be explained by rounding. Notably, several reported percentages match denominators that are exactly 200 smaller than those printed. This suggests a possible mismatch between evaluation releases or sample inventories. Since these results are also used in the main benchmark comparisons, the inconsistencies raise concerns about the reliability and reproducibility of the reported evaluation.

W2. Dataset statistics and evaluation protocols are not consistently reconciled.

The abstract reports 23,196 benchmark inputs, whereas Table 1 reports 24,396. Table 16 contains additional inconsistencies: the training and development subtotals do not match their component counts, and the three displayed split totals sum to 24,196 rather than 24,396.

Furthermore, Table 1 and Appendix C.2 indicate that L2 includes 200 Unknown examples, whereas Appendix D.2 states that L2 contains no Unknown claims. Although the manuscript distinguishes historical and updated evaluation protocols, the reported statistics are not sufficiently reconciled. The authors should provide a definitive data inventory and explicitly map every reported result to its corresponding benchmark version.

W3. The benchmark may contain shortcuts that weaken its intended reasoning requirements.

The L2 benchmark is described as requiring irreducible two-premise relational composition. However, the actual L2 prompt in Appendix Q.2.2 provides the numerical bounding boxes of two entities and asks whether one is located to the left of the other. Under image-coordinate semantics, the relation can be inferred directly from the supplied coordinates, without visual perception or multi-premise composition.

This example does not establish that the entire benchmark suffers from shortcut leakage. Nevertheless, it demonstrates that irreducibility of the construction-time proof does not necessarily establish that the model-visible input requires the intended reasoning process.

The existing text-only audit does not sufficiently resolve this concern because it does not provide a detailed strong-model evaluation of the complete rendered prompts across levels. Coordinate-removal controls and level-wise text-only evaluations would strengthen the benchmark validity argument.

W4. Source-grounded annotation consistency does not guarantee visual answerability.

The benchmark relies on deterministic adapters, source annotations, proof records, and transition verification. These procedures help validate symbolic consistency, but they do not necessarily establish that all required information is recoverable from the actual images provided to the model.

For example, the appendix distinguishes source-annotation-level checks from pixel-level validation and acknowledges that source-world reference frames may not be visually recoverable in every rendering. This issue is particularly important for multiview relations, reference-frame transformations, and Unknown judgments.

A stratified validation study using only model-visible evidence would be valuable for establishing whether the intended labels and reasoning requirements are actually supported by the input.

W5. The availability–utilization diagnosis does not uniquely identify a failure of state utilization.

The three diagnostic conditions are evaluated through independent requests. Correctly recovering an initial state under Direct-state does not demonstrate that the same state was successfully recovered during the failed Full-transition request.

An alternative explanation is that the additional task instructions and transformations interfere with initial-state extraction, entity binding, or reference-frame interpretation. Explicitly providing the initial state could bypass these difficulties rather than repair a downstream utilization failure.

The authors acknowledge this limitation in Appendix P. However, the main narrative sometimes presents the phenomenon as though correctly recovered spatial facts were demonstrably present but not incorporated into subsequent computation.

The evidence establishes a meaningful behavioral dissociation, but does not uniquely localize the underlying failure mechanism.

W6. OSS is not directly evaluated against the availability–utilization gap that motivates it.

The central motivation for OSS is to improve the transition from reportable spatial facts to usable states. However, the main OSS experiments report ClaimAcc and PairAcc on the benchmark rather than measuring whether fine-tuning reduces the original availability–utilization gap.

In particular, the paper does not provide a direct before-and-after comparison of Direct-state, Full-transition, and Explicit-state performance for the OSS model. Consequently, the reported accuracy improvements do not establish that OSS specifically repairs cases in which the original model could report an initial state but failed to perform the corresponding transformation.

This is a major missing link between the paper's motivating diagnosis and its proposed training intervention.

Questions for authors

Q1. Can the authors reconcile the numerical inconsistencies across the benchmark statistics and evaluation tables?

Please clarify the differences among the reported dataset sizes and explain why the numerators, denominators, and percentages in Tables 20 and 22 do not agree. Which exact evaluation release generated each result? Can the authors provide a unified evaluation manifest and regenerate all affected statistics?

Q2. How do the authors verify that the benchmark genuinely requires the intended spatial reasoning rather than exploiting shortcuts?

In particular, why does the L2 rendered prompt expose numerical bounding-box coordinates for a spatial relation judgment? How does the benchmark ensure that construction-time proof irreducibility translates into reasoning necessity for the model-visible input? Please provide coordinate-removal, text-only, and visual-answerability validation results.

Q3. How can the authors distinguish a failure of state utilization from a failure to recover the state under the more complex Full-transition context?

Since Direct-state and Full-transition are independent requests, a correct response in the former does not establish that the corresponding spatial fact was correctly represented in the latter. What additional controls support the specific utilization interpretation rather than context-dependent perception, binding, or state-initialization failures?

Q4. Does OSS directly reduce the availability–utilization gap, and which part of its supervision is responsible?

Please evaluate the original Direct-state, Full-transition, and Explicit-state diagnostics before and after OSS training, particularly on the originally recoverable-but-failed cases. An alignment-only ablation and information-/budget-matched supervision baselines would also help determine whether the improvement arises from canonical state alignment, trajectory supervision, or additional task-relevant training information.

Q5. How robust are the OSS improvements and the reported gap beyond the current evaluation settings?

Please provide complete class-specific results, especially for Unknown predictions, and clarify the large discrepancy between PairAcc and ClaimAcc. Can the findings be reproduced with nontrivial sequential transformations, another model family, multiple training runs, or a newly frozen evaluation set? How does the claimed scaling trend reconcile with the non-count results?

Reviewer_hznXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a specific failure mode (availability–utilization gap) of MLLMs: a model may be able to observe and report a spatial fact without being able to use that fact reliably in subsequent computation. Therefore, this paper introduces SpaceConflict, a benchmark designed for controlled conflicts among visual evidence, spatial claims, and changing world states. In addition, this paper introduces Operational State Supervision (OSS). Instead of supervising only the final answer, OSS directly supervises task-relevant spatial states and their transformation trajectories, which explicitly improves performance on SpaceConflict.

Critical strengths
  • This paper is well-motivated and studies an important problem of MLLMs. It discovers the availability–utilization gap of MLLMs based on experimental observations as discussed in Sec. 3.

  • This paper introduces a new spatial benchmark, SpaceConflict, that evaluates spatial reasoning in four levels.

  • This paper proposes OSS, which supervises task-relevant states and their transformation trajectories to strengthen the transition from spatial facts to operational states. OSS is shown effective to improve the performance on SpaceConflict.

Critical weaknesses
  • In Table 3, more recent SOTA models should be added for evaluation.

  • The benchmark consists of indoor scenes. However, the evaluation of unstructured outdoor scenes is more challenging but also important for multimodal large language models.

  • Generalization of OSS. OSS is validated by fine-tuning on the SpaceConflict training split and evaluating on its test split.

    • Does an OSS-trained model also improve on other spatial benchmarks?
    • In the paper, only Qwen model is finetuned. Does OSS transfer to other open-sourced models, such as InternVL?
  • The paper writing could be improved. It is a bit difficult to follow.

    • In Sec. 5.4, what is the difference between Answer-SFT Natural and Answer-SFT Balanced? Could the authors explain why their ClaimAcc is worse than the base model after finetuning?
    • Image quality is bad. For example, Figure 1 is generated by AI. There are many wrong words, such as "evidencn" and "choar".
    • Citation format should be unified. In Sec. 6, there exist two formats: (Chen et al., 2024) and Liao et al. (2024).
    • Data visualization of SpaceConflict should be added to better understand the benchmark, such as data distribution.
Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.