acceptodds
Under review as a conference paper at ICLR 2027

Counterfactual Twins and Executable Validity Gates for Diagnosing Spatial Reasoning in Multimodal LLMs

Abstract

Benchmark accuracy reports how often a multimodal LLM answers correctly, not whether it answered by looking. We present TWIN-Diag, a spatial-reasoning benchmark built from counterfactual twins. Each item pairs a procedurally generated indoor scene with two single-object interventions of identical displacement magnitude, one flipping the ground-truth answer and the other preserving it. Across eleven MLLMs from five families, answers track the image but not the truth. On relative distance (T-5), the one template passing every gate, answers change on 21.6% of controls, where the truth is held, and 24.1% of treatments, where it flips, against 5.0% for a repeated image. Pooled over models, counterfactual sensitivity is at most +0.069 (one-sided 95% bound, rooms as the unit of inference), and no model-template cell survives correction. The control's construction matters. In a pre-registered test on real renders of our previous release's pairs, controls sampled to keep the answer changed fewer answers than controls solved to the treatment's exact magnitude in all ten models (room-level p = 0.019, Holm-rejected), inflating measured sensitivity. Ours also match on-screen travel within 10%. Nine executable validity gates read only the data and exit non-zero on failure. All were fixed before any model answered the release (78 pairs, 32 rooms), four after models had answered earlier builds. The gates are binding. An extended gate rejected our previous release, and absolute distance (T-2) fails the question-count gate, so it is descriptive. The plan's confirmatory test, an image ablation on six unseen T-2 items, rejects at its design floor (p = 1/81), in conflict with that gate. On T-5 the same ablation fails to show that the image improves accuracy. Physical-plausibility defects that no gate checks do not detectably separate the arms. We group the twenty-four defects found in our work into seven failure modes of counterfactual-benchmark construction.

Primary area:
datasets and benchmarks
Keywords:
multimodal large language models, spatial reasoning, benchmark validity, counterfactual evaluation, executable validity gates, pre-registration, procedural scene generation, shortcut learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_6brMOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces TWIN-Diag, a diagnostic benchmark for spatial reasoning in multimodal LLMs. Each example contains a base scene and two single-object interventions: a treatment that changes the ground-truth answer and a control that preserves it. The interventions have identical world-space displacement magnitudes and approximately matched on-screen travel. Nine executable gates check properties including label coverage, leakage, visual-change balance, and referential stability. The released benchmark contains 78 matched pairs across 32 rooms and evaluates eleven models. Only relative distance, T-5, passes all nine gates. Its average answer-change rates are 24.1% for treatments and 21.6% for controls, yielding pooled sensitivity of +0.025 and a reported one-sided 95% upper bound of +0.069. A separate experiment on an earlier build finds that sample-and-filter controls underestimate answer instability relative to exact-magnitude controls. The paper also documents construction failures, withdrawn findings, and limitations of its validation protocol.

Critical strengths
  1. The control-construction problem is consequential and supported by a targeted experiment. Prior work such as CLEVR-Change distinguishes semantic changes from visual distractors; this paper focuses on matching the magnitude of answer-changing and answer-preserving interventions. In the D3 experiment, sample-and-filter controls produce fewer answer changes in all ten tested models, with a pooled difference of approximately 0.059 and a room-level test surviving the stated Holm correction (§5.3; Appendix B.2). This demonstrates that a seemingly routine benchmark-construction choice can materially alter the reported diagnostic score.
  2. The explicit audit and executable checks provide useful methodological infrastructure. The gates address concrete failures, including unstable referring expressions and mismatched screen motion, and the authors report cases where these checks invalidate their own earlier results. The use of room-level inference, disclosure of repeated analyses, and separation of exploratory from confirmatory evidence make the final claims easier to scrutinize. These practices and the documented failure taxonomy could be valuable to other benchmark developers.
Critical weaknesses
  1. W1 — The benchmark has not established that its released items reliably measure the intended spatial relations from the images. There is no completed human baseline, and the earlier assistant-model answerability probe was not repeated after the referential-expression repair (Appendix C). Moreover, the prompts leave the distance convention unspecified: nearest-point rather than centre-based distance changes 10 of 156 T-5 state labels, affecting nine pairs (Table 11). Physical defects also remain substantial: only 31 of 78 pairs, including 18 of 52 T-5 pairs, are free of both reported defect categories (Table 21). The authors’ sensitivity analyses are useful, but finding no significant model effect on the much smaller clean subset does not establish that the original benchmark is valid. Likewise, defects occurring in both arms can make both interventions difficult to interpret even if their frequency does not distinguish the arms. The unsuccessful T-5 image ablation is not evidence of unanswerability, but it also supplies no positive validation that resolves this gap. Consequently, low sensitivity cannot yet be confidently attributed to deficient spatial reasoning rather than ambiguity or perceptual difficulty.
  2. W2 — The evidence does not establish that the final matching scheme sufficiently isolates semantic answer changes. G6 passes the released T-5 data, yet a classifier combining its three low-level statistics distinguishes the arms under leave-one-room-out evaluation (AUC 0.587, room-swap p=0.002p=0.002); a held-out changed-pixel rule also achieves positive sensitivity (Table 6). These exploratory results do not prove that the evaluated models exploit such cues, but they show that passing the gates does not eliminate a relevant alternative explanation. Furthermore, D3 evaluates the previous build, whose exact controls moved farther on screen than their treatments. Sample-and-filter controls differ in both world-space and screen-space displacement, so D3 supports a policy-level effect without isolating the benefit of exact world-space matching or validating the final screen-matched controls (Appendix B.2). Together with only 27 T-5 rooms and no independent T-5 replication, this leaves the proposed diagnostic interpretation insufficiently validated. The pooled upper bound remains informative about the reported statistic on this selected benchmark and model set, but does not resolve these validity concerns.
Questions for authors
  • Validation evidence relevant to W1: Are results available from the planned blind human study on the released base, treatment, and control images? In particular, can raters consistently identify the intended objects and recover the prescribed labels, including the nine T-5 pairs affected by alternative distance conventions?

  • Additional analysis relevant to W1–W2: What are the pooled sensitivity estimate and room-level uncertainty bounds on the T-5 subset that is both physically clean and invariant to the alternative distance readings? Table 21 reports subset point estimates and significance counts, but uncertainty for this stricter subset would help assess whether the headline bound remains informative.

  • Clarification of the matching contribution relevant to W2: Is there evidence comparing exact world-space matching with a control policy that matches on-screen travel but allows world-space magnitudes to differ? Such evidence would materially affect whether I interpret exact magnitude matching as an independently validated requirement or as one component of a useful construction policy.

Reviewer_eERjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper looks at whether multimodal models actually use what they see when answering questions about a room. The authors build paired scenes where moving an object either changes the correct answer or leaves it unchanged. The main finding is that models often change their answers even when the correct answer stays the same. The paper also shows that how these test cases are built can affect the results.

Critical strengths

[S1] The idea of matching answer-changing and answer-preserving moves is useful. The comparison across ten models shows that using loosely matched controls can make models appear more sensitive than they really are. This gives good evidence that the proposed construction matters in practice.

Critical weaknesses

[W1] I still have concerns about the validity of the test cases. Only 31 of 78 pairs are free of the reported physical defects, and there is no human validation of whether the questions are reliably answerable from the images. Also, a combined rule using visual-change statistics can still distinguish the two arms on T-5 (AUC 0.587, p = 0.002), despite passing the individual checks. It is unclear that whether the benchmark isolates the intended reasoning ability.

[W2] The empirical efforts are fairly limited for the main conclusion. Only T-5 passes all the checks, with 52 pairs from 27 rooms. Its image ablation does not establish that seeing the actual image improves accuracy, and the per-model tests have only 28% power to detect a difference of 0.20 after correction. The near-zero average result is interesting, but the current evidence is not strong enough to characterize smaller effects or individual model behavior.

Questions for authors
  1. Do you have results from the planned human study showing that people can reliably answer the released T-5 questions?
Reviewer_iWDGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper propose new benchmark called TWIN-Diag, which generates two synthetic images including one preserving the ground truth answer and another flipping it. To be specific, the new images are generated by applying the same displacement of single object in the image following the supporting surface so that the VLM cannot change their answer by simply detecting modification of the images. Based on TWIN-Diag, the authors measure how often models change or maintain their answer and define counterfactual sensitivity to quantify this. During the construction of TWIN-Diag, nine validity gate is performed to validate the quality of generated images. The authors tested on 11 MLLMs of 5 family.

Critical strengths
  1. Propose new spatial reasoning benchmark following strict pipeline.
  2. Extensive evaluation across 11 MLLMs.
Critical weaknesses
  1. All writings seem like typical AI-generated texts (i.e., enumeration of details without key message) which significantly reduce the readability. I had difficult time understanding what the authors want to argue in the paper and it reads more as technical report than academic paper. I am quite suspicious whether the authors did check the writing quality of the paper by themselves. Current writing style is far below the standard of ICLR paper and could be the main reason for rejection. Below are some examples I found strange and awkward: (1) Notations and all numbers in abstract are used without definition (e.g., T-2, G6, counterfactual sensitivity +0.069, etc) (2) The paper does not include the conclusion section and ends with 6.1. Then, why we even need 6.1? (3) L221-231, what’s the point of this paragraph? This seems like artifacts from prompt the authors used. (4) Sec 4.3: CCR and FSR are not defined anywhere in the paper.

  2. The benchmark is aimed to diagnosis shortcut in spatial reasoning but limited to ||delta+||=||delta-|| setting (same displacement magnitude). However, if the MLLM is robust enough, it should be able to answer correctly with variable displacements (e.g, 2*||delta+-||, 3.52||delta+-||, etc ) which is not tested or discussed in the paper.

  3. The question template follows the VSI-bench and it only tests the distance-related questions. Thus, diagnosing spatial intelligence is overclaim due to the limited scope of questions. Moreover, these simple question can create another shortcut. The author should consider include various question types of spatial reasoning [1]

  4. The total size of the benchmark is too small which will result in unreliable evaluation with high variance per model.

Minor) Figure 2 seems like including AI-generated images and are not consistent in each stage which is confusing. Please make it consistent or switch with real example.

[1] Cai, Z., Wang, Y., Sun, Q., Wang, R., Gu, C., Yin, W., ... & Yang, L. (2025). Holistic evaluation of multimodal llms on spatial intelligence. arXiv preprint arXiv:2508.13142.

Questions for authors
  1. How the authors make sure that unrealistic displacement is not happend in the generated image (e.g., chair penetrate the wall, blurred artifacts etc.)?

  2. What is the intuition of the proposed nine gates? Which one is the most critical and the least critical during benchmark construction? Any human validation involved during the process?

I propose the author to significantly improve the writing and extend the benchmark in large scale including various question types while validating thoroughly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.