- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates what spatial multiple-choice accuracy can actually demonstrate about multimodal large language models (MLLMs). It distinguishes three aspects: reliance on necessary input information, sensitivity of mean accuracy to candidate layout changes, and response consistency.
The authors first identify candidate-based shortcuts in STARE using a question-blind pixel rule. They then introduce Crossed and Matched Candidate Audits to examine input dependence and sensitivity to candidate layouts. A Balanced Candidate Construction establishes exact accuracy ceilings for specified restricted-input strategies. Repeated responses and statistical tests are further used to assess changes in answer distributions.
Experiments cover six model configurations and 47,136 formal requests. The results show that simple candidate features can achieve high accuracy on existing spatial tasks, while similar mean accuracies across layouts can conceal substantial differences in model responses.
Critical strengths
1. Rigorous experimental controls and statistical analysis.
The paper provides careful evaluation designs to support its conclusions.
- The controlled geometric transformations and balanced candidate construction establish exact accuracy ceilings for specified restricted-input strategies.
- The authors examine shortcuts introduced by image preprocessing and use repeated responses and statistical tests to distinguish mean accuracy changes from response-distribution changes.
These designs strengthen the validity of the findings within the tested settings and provide useful insights for more rigorous spatial reasoning evaluation.
Critical weaknesses
1. Limited novelty beyond prior work and insufficient evidence of broader significance.
The paper builds on several well-established evaluation ideas.
- Benchmark shortcuts: SugarCrepe [1] and I-RAVEN [2] have already identified candidate-set biases and proposed methods to mitigate them. Brown et al. [3] further introduced a systematic framework for detecting and reducing non-visual shortcuts.
- Controlled evaluation and consistency: Winoground [4] introduced crossed-input evaluation to test vision-language compositionality, while MM-R³ [5] systematically investigated the relationship between accuracy and response consistency.
Although this paper introduces more rigorous balancing constructions and statistical tests, it does not sufficiently demonstrate what important new findings these techniques provide beyond prior work or how they substantially improve existing evaluation practices.
The experimental scope is also limited.
- The core experiments focus on controlled 2D geometric multiple-choice tasks, with the main repeated-response analyses covering only a few fixed shapes.
- Prior benchmarks such as 3DSRBench [6] and Real-3DQA [7] have explored more diverse spatial relationships and 3D scenarios. The paper provides limited evidence that its methods and conclusions generalize to more complex spatial reasoning tasks.
- The most pronounced response instability is observed in Qwen32, which achieves relatively low accuracy on the corresponding tasks. This further limits the conclusions about the reliability of models with strong spatial reasoning performance.
Together, these limitations weaken the novelty and broader significance of the contribution.
2. Significant weaknesses in presentation and paper organization.
The Introduction and Conclusion are particularly underdeveloped.
- Introduction: It largely lists experimental findings and technical details without clearly developing the research motivation, gaps in prior work, or the significance of the proposed contributions.
- Conclusion: It briefly restates the findings and limitations but provides little discussion of their implications or broader relevance.
- Overall organization: The main text contains dense mathematical definitions, statistical procedures, and experimental details, while some information essential for understanding the methodology is scattered across the extensive appendix.
These issues make the paper unnecessarily difficult to follow and obscure its central contributions. The overall presentation needs substantial improvement to meet the clarity and completeness expected of a conference paper.
Questions for authors
1. What specific problems does the proposed shortcut auditing method address that prior methods cannot?
SugarCrepe [1] and I-RAVEN [2] have already addressed candidate-set biases through improved candidate construction, while Brown et al. [3] proposed systematic shortcut detection and mitigation. The proposed Balanced Candidate Construction provides exact accuracy ceilings for certain restricted-input strategies.
- Which shortcuts can these guarantees identify or rule out that existing methods cannot adequately handle?
- Can the authors provide direct experimental evidence demonstrating this additional value?
2. What new findings does the response consistency analysis provide beyond prior work?
MM-R³ [5] has already investigated VLM response consistency under repeated inputs and controlled perturbations, showing that accuracy and consistency do not necessarily correlate. This paper introduces candidate layout interventions and categorical MMD tests, but its main conclusion remains that similar mean accuracy does not imply stable responses.
- Beyond providing more rigorous statistical evidence, what important new phenomena does this analysis reveal?
- Does the proposed analysis lead to substantially different conclusions about model reliability compared with existing consistency evaluations?
3. How well do the proposed evaluation methods generalize beyond the current experimental setting?
The core experiments mainly involve 2D geometric multiple-choice tasks, and the final repeated-response analyses cover only a few fixed shapes. In contrast, 3DSRBench [6] and Real-3DQA [7] evaluate more diverse spatial relationships and 3D scenarios.
- Is there experimental evidence that the proposed auditing framework applies to other types of spatial reasoning tasks beyond the specific candidate constructions studied here?
- If not, how should the scope of the paper's conclusions be limited?
4. How are the stability and independence assumptions underlying the statistical tests validated?
The paper reports approximately 2% response disagreement for Qwen32 under identical inputs and greedy decoding, with the source of this variation remaining unexplained. Since the statistical tests assume stable and independent response distributions:
- Can the authors provide further evidence supporting these assumptions?
- Could unexplained inference variability or dependencies across repeated calls and test groups affect the main statistical conclusions?
References
[1] Hsieh, C.-Y., Zhang, J., Ma, Z., Kembhavi, A., & Krishna, R. (2023). SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality. Advances in Neural Information Processing Systems (NeurIPS), 36, Datasets and Benchmarks Track. Paper.
[2] Hu, S., Ma, Y., Liu, X., Wei, Y., & Bai, S. (2021). Stratified Rule-Aware Network for Abstract Visual Reasoning. Proceedings of the AAAI Conference on Artificial Intelligence, 35(2), 1567–1574. Paper.
[3] Brown, E., Yang, J., Yang, S., Fergus, R., & Xie, S. (2026). Benchmark Designers Should “Train on the Test Set” to Expose Exploitable Non-Visual Shortcuts. Conference on Language Modeling (COLM). Paper.
[4] Thrush, T., Jiang, R., Bartolo, M., Singh, A., Williams, A., Kiela, D., & Ross, C. (2022). Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 5238–5248. Paper.
[5] Chou, S.-H., Chandhok, S., Little, J. J., & Sigal, L. (2025). MM-R³: On (In-)Consistency of Vision-Language Models (VLMs). Findings of the Association for Computational Linguistics: ACL 2025, 4762–4788. Paper.
[6] Ma, W., Chen, H., Zhang, G., Chou, Y.-C., Chen, J., de Melo, C., & Yuille, A. (2025). 3DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 6924–6934. Paper.
[7] Ma, X., Sun, T., Chen, S., Bhalgat, Y., Gu, J., Chang, A. X., Armeni, I., Laina, I., Peng, S., & Prisacariu, V. A. (2026). Do 3D Large Language Models Really Understand 3D Spatial Relationships? International Conference on Learning Representations (ICLR). Paper.