acceptodds
Withdrawn from ICLR 2027

One Relation Is Not Enough: Benchmarking Relational Spatial Understanding in Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

27 views

Abstract

Spatial understanding requires more than recognizing isolated spatial relations; it also requires understanding how diverse descriptions involving different reference objects and reference frames jointly characterize the same spatial structure. A correct answer under a single formulation is therefore insufficient to reflect a model's ability to identify multiple valid spatial descriptions completely and accurately. We introduce CoSpatial-Bench, a direct multi-select benchmark that evaluates whether vision-language models can select all valid descriptions of the same target from a fixed candidate set, based on the scene and question, while rejecting unsupported options. The Base variant contains 495 questions and 2,000 human-reviewed ordinary candidate descriptions. Our primary metric, Error-Free Coverage (EFC), measures coverage of valid descriptions under a no-false-selection constraint, while paired variants examine recognition performance under changes in referring expressions, viewpoints, and spatial conditions. Across ten VLM baselines, the best Base EM is only 33.54, while the highest EFC is 42.53. A separate single-candidate intervention study characterizes correction of initially incorrect option decisions under perceptual support and subsequent combined perceptual and reasoning support: across three diagnostic cohorts, perceptual support corrects 58.7–70.7% of initial errors, while 8.6–13.8% remain unresolved after both stages. We further introduce Target-Centered Relational QA Synthesis (TCR-QAS) and train Qwen3.5-4B with whole-question GRPO on the synthesized data, increasing its Base EFC from 28.17 to 33.37. These results highlight the need to evaluate spatial understanding beyond the correctness of a single answer: complete and error-free identification of diverse valid descriptions remains challenging.

Primary area:
datasets and benchmarks
Keywords:
Spatial Benchmarking, Spatial Relation Understanding, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WjGmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a new multi-select benchmark on spatial understanding and evaluates multiple VLMs.

Critical strengths

The key idea that VLMs' spatial understanding of a referent needs more comprehensive evaluation is interesting. The paper is also easy to follow.

Critical weaknesses

The task's distinctiveness needs further justification. Besides, the analysis of the evaluation results, e.g., error types and mechanisms, remains limited.

Questions for authors
  • The paper lacks a clear characterization of description diversity in the benchmark. One of the central distinctions of CoSpatial-Bench from existing spatial QA benchmarks is its emphasis on multiple valid descriptions that differ in spatial relations, reference objects (anchors), reference frames, or levels of specificity. The authors should clarify the construction criteria and provide a quantitative breakdown of the resulting dataset.
  • Section 5.1 mainly summarizes model rankings and metric differences. Could the authors provide more detailed analyses of model performance and failure modes? For example, Doubao Seed 2.0 Pro and Qwen3.8-Max achieve nearly identical EFC scores but substantially different false selection rates. It would be useful to understand whether these differences arise from conservative selection strategies, relation types, or something else.
  • What is the motivation for introducing GRPO training, and what does the improvement actually demonstrate? The connection to the benchmark's central objective is not sufficiently explained.

I think the above points need further clarification. I am happy to raise my score if the authors address them.

Reviewer_Z2cdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces CoSpatial-Bench, a multi-select benchmark of 495 questions and 2,000 human-reviewed candidate descriptions in which a model must select every valid spatial description of the same target, varying in anchor, relation, reference frame, and specificity, while rejecting all unsupported options and the none-of-the-above choice. A GRPO study on image-disjoint synthetic data raises Qwen3.5-4B EFC from 28.17 to 33.37 while FSR worsens from 40.25 to 51.92, showing the gap is partly addressable.

Critical strengths
  • Each question requires accepting all valid descriptions of the same target, which vary in anchor, relation, and reference frame, and rejecting all unsupported ones in a single response, so memorized single-relation captions cannot suffice. The Augmented variants re-score the same images under changed referring expressions and reference frames against their own reviewed gold sets. The multi-validation can better test VLM's capability of spatial relation .
Critical weaknesses
  • The GRPO gain may reflect selection style and label-style distillation rather than relational understanding, since FSR rises from 40.25 to 51.92 alongside the EFC gain from 28.17 to 33.37 and the reward is distilled from three-model consensus labels that match human judgments on only 85% of audited candidates, with no SFT or reward ablation to isolate the cause.

  • The image pool is heavily concentrated in reused public sources, with VSR alone providing 369 of 495 questions (74.5%), which limits scene diversity and leaves image-level familiarity with pretraining corpora unaddressed. The formulation difference over prior multi-answer benchmarks is also narrow, since SpatialVOC2K already annotates multiple applicable relations per object pair and SpaCE-10 already requires selecting both correct options, leaving variable undisclosed cardinality over same-target supplied descriptions as the main new element.

Questions for authors

See weaknesses.

Reviewer_UMt5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CoSpatial-Bench, a benchmark designed to evaluate whether vision-language models (VLMs) can identify all valid spatial descriptions of the same target object while rejecting incorrect ones. Unlike conventional spatial reasoning benchmarks that typically focus on individual relations or single-answer questions, CoSpatial-Bench adopts a multi-select formulation involving multiple spatial relations, reference objects, and levels of description specificity.

The benchmark contains 495 base questions with 2,000 human-reviewed candidate descriptions, together with augmented variants examining referring expressions, viewpoints, and spatial transformations. The authors propose Error-Free Coverage (EFC) to measure the coverage of valid descriptions under a no-false-selection constraint.

Experiments across ten VLMs demonstrate substantial difficulties in identifying complete valid answer sets, with the highest exact-match accuracy reaching 33.54%. The paper further investigates model errors through staged perceptual and reasoning evidence interventions. Finally, the authors introduce Target-Centered Relational QA Synthesis (TCR-QAS) and apply GRPO to Qwen3.5-4B, improving its Base EFC from 28.17% to 33.37%.

Critical strengths
  1. A meaningful evaluation perspective on spatial understanding.

The paper highlights the limitation of evaluating spatial understanding through individual relations or single-answer questions. Recognizing one correct spatial description does not necessarily imply that a model can identify other simultaneously valid descriptions of the same object. The multi-select formulation provides a useful perspective for evaluating the completeness and correctness of relational spatial understanding. The proposed EFC metric also explicitly considers the trade-off between identifying valid descriptions and avoiding incorrect selections.

  1. Relatively comprehensive empirical characterization of model behavior.

The authors evaluate ten VLMs, including both proprietary and open-source models, and report complementary metrics covering exact-set identification, option-level discrimination, and false selections. The evaluation additionally considers transformed questions, option-order sensitivity, and staged evidence interventions. The GRPO experiment further explores whether training can improve performance under the proposed evaluation protocol. These analyses provide useful insights into the behavior and limitations of current VLMs on multi-answer spatial understanding tasks.

Critical weaknesses
  1. Insufficient justification for the proposed evaluation setting as a distinct spatial reasoning capability.

Although the paper highlights the importance of identifying multiple coexisting spatial relations, it remains unclear whether the proposed multi-select formulation evaluates a fundamentally different capability from existing spatial relation verification tasks. In principle, independently verifying each candidate description and aggregating the valid ones could achieve the same objective. The paper lacks a direct comparison between independent relation verification and joint multi-select prediction, making it difficult to determine whether the observed difficulties arise from spatial reasoning limitations or the evaluation formulation itself. Such comparisons are necessary to better establish the novelty and motivation of the benchmark.

  1. Insufficient motivation and empirical justification for using GRPO.

The motivation for adopting GRPO is not sufficiently established. Since the training data already provides complete ground-truth answer sets, the task could naturally be addressed through supervised fine-tuning (SFT) or multi-label classification. Although GRPO allows direct optimization of the proposed EFC metric, the paper does not explain why reinforcement learning is preferable to simpler supervised approaches, nor does it provide corresponding baselines. Furthermore, while GRPO improves EFC, the False Selection Rate also increases substantially, raising questions about whether the improvement reflects better spatial understanding or merely a change in answer selection behavior. Comparisons with SFT and reward-function ablations would help justify the necessity and effectiveness of the proposed training strategy.

Questions for authors
  1. How does the proposed multi-select formulation assess spatial reasoning capabilities beyond independently verifying each candidate relation? Could the authors provide a comparison between independent binary verification and joint multi-select prediction to demonstrate the distinct challenges introduced by CoSpatial-Bench?

  2. Given that the training data already contains complete ground-truth answer sets, why is GRPO preferred over supervised fine-tuning (SFT)? Could the authors provide an SFT baseline and explain why EFC improves while the False Selection Rate increases substantially after GRPO training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.