acceptodds
Under review as a conference paper at ICLR 2027

WHAT DOES SPATIAL ACCURACY CERTIFY? MATCHED AND CROSSED CANDIDATE AUDITS

Abstract

Spatial multiple-choice accuracy can conceal candidate shortcuts and unstable an- swers under controlled layout changes. We test what such scores certify using question-blind pixel rules, crossed scene–instruction cases, matched foil transla- tions and repeated responses across six model configurations and 47,136 formal requests. A fixed question-blind rule solves 245/326 released STARE questions, though candidate-only prompting performs poorly on a matched subset. Our bal- anced construction gives exact 1/4 accuracy ceilings for specified input-omission rules. Yet Qwen2.5-VL-32B (Qwen32) original preprocessed inputs admit a per- fect query-plus-centroid rule; direct-grid rendering restores seven declared source and native bounds. On one aligned 78-cell shape, Qwen32 rejects both omission nulls, but 54.27% of cross-layout response pairs disagree, versus 2.05% of same- input pairs. An exact categorical permutation test rejects response-distribution equality; the mean half-squared categorical distance is 0.5222 (95% interval [0.4586, 0.5859]). Separately, the simultaneous 95% interval for Qwen32’s disk- pair mean-accuracy change is [−3.18, 3.48] points, within the prespecified ±5- point margin despite 397 correctness reversals among 2,048 pairs. These results separate evidence for required inputs, mean tolerance and response stability with- out identifying an internal spatial algorithm.

Primary area:
foundation or frontier models, including LLMs
Keywords:
spatial reasoning, vision-language models, model evaluation, benchmark shortcuts, controlled experiments, robustness evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cUA6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates what spatial multiple-choice accuracy can actually demonstrate about multimodal large language models (MLLMs). It distinguishes three aspects: reliance on necessary input information, sensitivity of mean accuracy to candidate layout changes, and response consistency.

The authors first identify candidate-based shortcuts in STARE using a question-blind pixel rule. They then introduce Crossed and Matched Candidate Audits to examine input dependence and sensitivity to candidate layouts. A Balanced Candidate Construction establishes exact accuracy ceilings for specified restricted-input strategies. Repeated responses and statistical tests are further used to assess changes in answer distributions.

Experiments cover six model configurations and 47,136 formal requests. The results show that simple candidate features can achieve high accuracy on existing spatial tasks, while similar mean accuracies across layouts can conceal substantial differences in model responses.

Critical strengths

1. Rigorous experimental controls and statistical analysis.

The paper provides careful evaluation designs to support its conclusions.

  • The controlled geometric transformations and balanced candidate construction establish exact accuracy ceilings for specified restricted-input strategies.
  • The authors examine shortcuts introduced by image preprocessing and use repeated responses and statistical tests to distinguish mean accuracy changes from response-distribution changes.

These designs strengthen the validity of the findings within the tested settings and provide useful insights for more rigorous spatial reasoning evaluation.

Critical weaknesses

1. Limited novelty beyond prior work and insufficient evidence of broader significance.

The paper builds on several well-established evaluation ideas.

  • Benchmark shortcuts: SugarCrepe [1] and I-RAVEN [2] have already identified candidate-set biases and proposed methods to mitigate them. Brown et al. [3] further introduced a systematic framework for detecting and reducing non-visual shortcuts.
  • Controlled evaluation and consistency: Winoground [4] introduced crossed-input evaluation to test vision-language compositionality, while MM-R³ [5] systematically investigated the relationship between accuracy and response consistency.

Although this paper introduces more rigorous balancing constructions and statistical tests, it does not sufficiently demonstrate what important new findings these techniques provide beyond prior work or how they substantially improve existing evaluation practices.

The experimental scope is also limited.

  • The core experiments focus on controlled 2D geometric multiple-choice tasks, with the main repeated-response analyses covering only a few fixed shapes.
  • Prior benchmarks such as 3DSRBench [6] and Real-3DQA [7] have explored more diverse spatial relationships and 3D scenarios. The paper provides limited evidence that its methods and conclusions generalize to more complex spatial reasoning tasks.
  • The most pronounced response instability is observed in Qwen32, which achieves relatively low accuracy on the corresponding tasks. This further limits the conclusions about the reliability of models with strong spatial reasoning performance.

Together, these limitations weaken the novelty and broader significance of the contribution.

2. Significant weaknesses in presentation and paper organization.

The Introduction and Conclusion are particularly underdeveloped.

  • Introduction: It largely lists experimental findings and technical details without clearly developing the research motivation, gaps in prior work, or the significance of the proposed contributions.
  • Conclusion: It briefly restates the findings and limitations but provides little discussion of their implications or broader relevance.
  • Overall organization: The main text contains dense mathematical definitions, statistical procedures, and experimental details, while some information essential for understanding the methodology is scattered across the extensive appendix.

These issues make the paper unnecessarily difficult to follow and obscure its central contributions. The overall presentation needs substantial improvement to meet the clarity and completeness expected of a conference paper.

Questions for authors

1. What specific problems does the proposed shortcut auditing method address that prior methods cannot?

SugarCrepe [1] and I-RAVEN [2] have already addressed candidate-set biases through improved candidate construction, while Brown et al. [3] proposed systematic shortcut detection and mitigation. The proposed Balanced Candidate Construction provides exact accuracy ceilings for certain restricted-input strategies.

  • Which shortcuts can these guarantees identify or rule out that existing methods cannot adequately handle?
  • Can the authors provide direct experimental evidence demonstrating this additional value?

2. What new findings does the response consistency analysis provide beyond prior work?

MM-R³ [5] has already investigated VLM response consistency under repeated inputs and controlled perturbations, showing that accuracy and consistency do not necessarily correlate. This paper introduces candidate layout interventions and categorical MMD tests, but its main conclusion remains that similar mean accuracy does not imply stable responses.

  • Beyond providing more rigorous statistical evidence, what important new phenomena does this analysis reveal?
  • Does the proposed analysis lead to substantially different conclusions about model reliability compared with existing consistency evaluations?

3. How well do the proposed evaluation methods generalize beyond the current experimental setting?

The core experiments mainly involve 2D geometric multiple-choice tasks, and the final repeated-response analyses cover only a few fixed shapes. In contrast, 3DSRBench [6] and Real-3DQA [7] evaluate more diverse spatial relationships and 3D scenarios.

  • Is there experimental evidence that the proposed auditing framework applies to other types of spatial reasoning tasks beyond the specific candidate constructions studied here?
  • If not, how should the scope of the paper's conclusions be limited?

4. How are the stability and independence assumptions underlying the statistical tests validated?

The paper reports approximately 2% response disagreement for Qwen32 under identical inputs and greedy decoding, with the source of this variation remaining unexplained. Since the statistical tests assume stable and independent response distributions:

  • Can the authors provide further evidence supporting these assumptions?
  • Could unexplained inference variability or dependencies across repeated calls and test groups affect the main statistical conclusions?

References

[1] Hsieh, C.-Y., Zhang, J., Ma, Z., Kembhavi, A., & Krishna, R. (2023). SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality. Advances in Neural Information Processing Systems (NeurIPS), 36, Datasets and Benchmarks Track. Paper.

[2] Hu, S., Ma, Y., Liu, X., Wei, Y., & Bai, S. (2021). Stratified Rule-Aware Network for Abstract Visual Reasoning. Proceedings of the AAAI Conference on Artificial Intelligence, 35(2), 1567–1574. Paper.

[3] Brown, E., Yang, J., Yang, S., Fergus, R., & Xie, S. (2026). Benchmark Designers Should “Train on the Test Set” to Expose Exploitable Non-Visual Shortcuts. Conference on Language Modeling (COLM). Paper.

[4] Thrush, T., Jiang, R., Bartolo, M., Singh, A., Williams, A., Kiela, D., & Ross, C. (2022). Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 5238–5248. Paper.

[5] Chou, S.-H., Chandhok, S., Little, J. J., & Sigal, L. (2025). MM-R³: On (In-)Consistency of Vision-Language Models (VLMs). Findings of the Association for Computational Linguistics: ACL 2025, 4762–4788. Paper.

[6] Ma, W., Chen, H., Zhang, G., Chou, Y.-C., Chen, J., de Melo, C., & Yuille, A. (2025). 3DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 6924–6934. Paper.

[7] Ma, X., Sun, T., Chen, S., Bhalgat, Y., Gu, J., Chang, A. X., Armeni, I., Laina, I., Peng, S., & Prisacariu, V. A. (2026). Do 3D Large Language Models Really Understand 3D Spatial Relationships? International Conference on Learning Representations (ICLR). Paper.

Reviewer_BEENOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks what accuracy on spatial multiple-choice tasks actually certifies. It proposes an audit that separates three questions: whether a model uses the required inputs, whether mean accuracy is stable under controlled layout changes, and whether individual answers are stable. The authors first show that a question-blind rule solves 75% released STARE 2D instruction questions. They then build crossed questions, where the same candidate board is paired with queries whose correct answers differ. In this case, only GPT-5.6 (Sol) clearly exceeds the oracle ceilings on the crossed tasks; Qwen2.5-VL-32B shows almost no mean accuracy change.

Critical strengths
  1. The question of what benchmark accuracy certifies is relevant, and separating input use, mean tolerance and answer stability is a reasonable framing.
  2. The STARE observation is concrete and useful to the community, which shows that a simple question-blind rule reaches 75% accuracy.
  3. The crossed construction, where the same board appears with every answer label, gives exact accuracy ceilings for rules that omit specified inputs.
Critical weaknesses
  1. The writing is very hard to follow, starting with the introduction. The introduction consists of short paragraphs packed with numbers and unexplained terminology that do not help understanding for a human reader as me. There is no clear motivation, problem statement or structured contributions.
  2. The methodology lacks motivation. For example, section 3.2 in particular introduces very sophisticated transformations (transporting foil centroids around a fixed reference, invariance of pairwise distances, a Z4 cyclic phase construction) without stating what they are designed to achieve or which shortcut each step removes. The paragraph "Balance the matched population itself" presents an impossibility result for a fixed-rotation design that is then abandoned, so the reader follows the authors' exploration path rather than the final method. The writing of methodology is more like an experimental log than an orgnaized study.
  3. The results are model-specific, and each experiment uses a different subset of models. The crossed bitmap study uses Gemma, Mistral, Qwen2.5-VL-3B and Sol; the disk study uses Sol and Qwen32; the balanced study uses Sol, Qwen32 and Opus 5.... Why is this the case? It is therefore not possible to tell whether the conclusions hold for other models or for spatial accuracy in general, as the title suggests.
  4. The instability finding mainly concerns a model that is close to chance. The model that actually solves the tasks (Sol) is stable. This limits the practical meaning of "mean accuracy hides unstable answers".
  5. Limited scope and practical takeaways. All controlled experiments use synthetic 2D shapes, a few fixed shapes and one transformation family. The paper does not audit other spatial benchmarks systematically and does not provide a protocol or checklist that benchmark designers could adopt. This limits this paper's contribution to the community.
Questions for authors
  1. Can the authors state clearly in Method, the exact goals of the design choices?
  2. Which design is the final recommended audit, and which results in Tables 1 to 6 use it?
  3. Can all audits be run on the same set of models, including several open models of different capability? Do the instability findings hold for models other than Qwen2.5-VL-32B? Why do different experiments use different set of models?
  4. What concrete recommendations do the authors derive for designing or auditing spatial benchmarks?
  5. How general are these findings to different spatial benchmarks and transformation groups?
Reviewer_YTp1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether high accuracy on spatial multiple-choice tasks actually establishes. It combines question-blind candidate heuristics, crossed scene–instruction tests, matched layout interventions, and repeated responses to examine three distinct questions: whether the relevant inputs are needed, whether layout changes affect mean accuracy, and whether answer distributions remain stable. A fixed question-blind farthest-centroid rule solves 245 of 326 released STARE questions, demonstrating that a simple shortcut is available without showing that the evaluated models use it. For Qwen2.5-VL-32B, an aligned-shape experiment reports 54.27% cross-layout response disagreement versus 2.05% disagreement on repeated identical inputs. Separately, a disk-pair study finds 397 correctness reversals across 2,048 pairs despite a mean-accuracy change of only +0.15 percentage points. The paper also shows that native image preprocessing can introduce exploitable centroid cues. These results clarify the evidential limits of spatial accuracy, without identifying internal reasoning strategies of models.

Critical strengths

1.This paper identifies a concrete weakness in interpreting spatial multiple-choice accuracy. The high answer accuracy does not necessarily provide sufficient evidence of spatial reasoning, as performance may be influenced by shortcuts in candidate construction.This contrast is useful: the failure of a prompted baseline does not establish the absence of an exploitable shortcut.

2.This paper conducts an evaluation from three dimensions: required inputs, mean tolerance and response stability. Compared to the traditional evaluation method that relies solely on a single accuracy rate indicator, this framework can reveal behavioral differences that are difficult to be reflected by average performance indicators, providing a more detailed evaluation approach for analyzing whether the model relies on specific input cues.

3.The paper gives its audit claims a precise statistical scope. The balanced constructions yield exact 25% accuracy ceilings for specified input-omission policies, rather than an unrestricted claim about spatial reasoning. The accompanying permutation tests and confidence intervals make the evidence for layout dependence and mean-accuracy tolerance easier to assess.

Critical weaknesses
  1. The experiments focus largely on controlled planar configurations. These are appropriate for testing specific shortcuts, but they leave open whether the same audits would be informative for natural images with occlusion, depth ambiguity, and viewpoint changes.

  2. The reported response differences establish behavioral sensitivity to layout changes, but provide limited evidence about which cues drive that sensitivity. For example, it remains unclear how much disagreement is associated with candidate position, orientation, or preprocessing-induced visual changes.

  3. The theoretical 25% ceilings apply to specified restricted-input policies; they do not rule out strategies that jointly exploit the query and candidate board. The paper itself notes that, in one balanced construction, the query together with the first candidate’s orientation is sufficient to recover the answer. Thus, passing the omission audits does not establish that a model compares all candidates as intended.

  4. The model configurations use different preprocessing pipelines, decoding settings, and inference budgets. The paper appropriately states that these are configuration-level observations rather than equal-compute model rankings. But this makes cross-model behavioral differences difficult to interpret. Where feasible, a smaller controlled comparison would help distinguish configuration effects from differences in model behavior.

  5. Some layout interventions change more than the intended spatial factor. In the balanced matched construction, candidate distances are not preserved across all conditions, and the input preprocessing analysis does not isolate the effect of any single transformation. Consequently, the observed response changes cannot always be attributed to one geometric or rendering variable. Additional controls that vary candidate spacing and preprocessing separately would make the intervention results easier to interpret.

Questions for authors

Please refer to the weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.