acceptodds
Withdrawn from ICLR 2027

Recognition Gains, Compositional Gaps: Rethinking Spatial Reasoning in Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Spatial reasoning underpins real-world applications of vision-language models, including robotics, autonomous driving, and navigation. Yet progress in spatial recognition leaves unresolved whether models can reliably combine and reuse the information they recognize. To examine this relationship, we introduce SpaReCo-Bench (Spatial Recognition and Composition Benchmark), which organizes 15 tasks across four families: state, identity, relation, and composition. Centered on object orientation and spatial transformations, the benchmark grounds basic judgments and compositional problems in shared visual contexts, making it possible to assess recognition and the use of recognized information together. Our evaluations reveal a persistent gap between the two: models can correctly identify the spatial states and relations needed for a task, yet fail when these judgments must be combined. This disconnect persists after spatial training, even as recognition and the application of explicit transformations improve, showing that gains in these capabilities do not by themselves resolve difficulties in attribute composition and transformation analogy. SpaReCo-Bench provides a controlled foundation for distinguishing recognition gains from progress in compositional reasoning and for diagnosing where spatial information ceases to support reliable task performance. The benchmark, dataset, and code will be made publicly available.

Primary area:
interpretability and explainable AI
Keywords:
vision-language models, object orientation, spatial reasoning, multimodal benchmarks, consistency, compositional reasoning, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 2 reviews, above 0% of the others.
Reviewer_Mm28OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces SpaReCo-Bench, a benchmark connecting spatial recognition to compositional use through shared object views and linked questions. It organizes 15 task identifiers into state, identity, relation, and composition families, reported as 25 evaluation entries, and combines controlled object-view questions with retained tasks from existing benchmarks. The evaluation includes 11 general-purpose or spatially tuned VLM checkpoints, targeted fine-tuning of Qwen2.5-VL-3B, linked condition-following analyses, and external evaluation. The main reported finding is that source-task recognition gains transfer unevenly to related selection and transformation tasks, whereas direct supervision improves some attribute-composition tasks. The intended contribution is a controlled diagnostic evaluation framework rather than a new model.

Critical strengths

It departs from the isolated scoring paradigm adopted by conventional spatial benchmarks, and couples basic recognition tasks with compositional decision tasks through shared object views. This design enables precise pinpointing of whether model errors stem from inadequate spatial recognition or insufficient compositional reasoning.In particular, its hierarchical task gradient and paired experimental design clearly delineate three distinct levels of compositional ability—attribute recombination, explicit transformation application, and analogical rule transfer—achieving substantially finer diagnostic granularity than standard benchmarks.

Critical weaknesses

Dense task identifiers, experimental groupings, and transfer matrices fragment the narrative. Frequent cross-referencing obscures the main argument and makes the experimental evidence difficult to follow.

Despite 11 checkpoints in Table 2, general-purpose baselines are limited to 2B-7B models, and the key transfer results rely on Qwen2.5-VL-3B alone. Larger models, newer open models such as Qwen3.5 and InternVL-3.5, and proprietary references such as Gemini 3.8 Flash and GPT-6 are absent. This limits the generality of the reported compositional deficits.

The benchmark primarily combines established tasks in viewpoint recognition, cross-view matching, attribute conjunction, rotation, and visual analogy. Explicit task links provide useful organization, but their distinct diagnostic value over existing evaluations is not established. Table 1 demonstrates coverage rather than a new measurement capability.

The evidence does not clearly separate recognition errors from failures to use correctly recognized information. Low compositional accuracy alone cannot establish a distinct reasoning deficit, limiting the strength of the central claim.

Questions for authors

What diagnostic insight is unique to the linked evaluation, and what evidence isolates its benefit? Are unseen attribute combinations or transformation patterns held out alongside object identities?

Does low C2/C3 accuracy persist when all required components are correctly recognized under matched input contexts? Do ground-truth controls or human results rule out presentation issues and task ambiguity?

Do the key deficits and transfer patterns persist across model scales, newer models, and a second backbone?

Reviewer_wJQuOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SpaReCo-Bench is a spatial-reasoning benchmark for VLMs: 15 tasks in four families (State, Identity, Relation, Composition), with about 141K evaluation questions. Its key design feature is that component questions and compositional target questions share images. That lets the authors compare recognizing an attribute with using it. The paper evaluates 11 open 2–7B VLMs and runs 10 controlled SFT experiments from Qwen2.5-VL-3B to measure source learning, cross-task transfer, and direct target learning. It concludes that transfer from spatial training is "task-dependent."

Critical strengths
  1. Linked-question design. Shared visual evidence across component and target questions, plus conjunction-constructed galleries (C1, I2), could separate perception failures from composition failures. Most spatial benchmarks cannot do this.
  2. Source / transfer / direct-target protocol (Table 3). Training directly on the target is a good control. It shows, for example, that I2 is learnable (+39.5 pp) even though transfer to it from S3 or I1 is null.
Critical weaknesses
  1. The core transfer results contradict each other. Figure 4's diagonal cells match the rest of the paper, but its off-diagonal cells contradict Table 3, Figure 5b, and the text:

    PairTable 3 / Fig. 5b / textFig. 4
    I1 → I2≈ −0.2+46.8
    S3 → I2≈ −9.0+28.0
    I2 → I1+20.75−26.2
    C1 → R3≈ +0.5+49.3

    The headline claims ("C1-SFT improves all six targets, mean +26.1"; "R3-SFT mean −15.9") are computed from Figure 4. Recomputed from Figure 5b, they are roughly +4 and roughly −1. The main evidence for task-dependent transfer is therefore unreliable.

  2. Key tasks sit at chance, even after direct training.

    • Across all 11 models, S3 (24-way), R1–R3 (7-way), and C1–C3 (4-way) score within a few points of chance.
    • Direct SFT on 12K C2 or C3 examples gains +0.8 and +0.5 pp, with CIs that include zero.
    • With no headroom on the target, the near-zero transfer to C2/C3 is uninterpretable. The paper reads it as a "compositional gap" without ruling out simpler causes: input format or resolution limits, label ambiguity, or too little training.
    • The family averages in Figure 3 are dominated by chance-level columns, so the model-to-model differences they show are noise.
  3. The transfer experiments are confounded.

    • Token budgets differ across training sources.
    • Each condition uses a single seed, and the CIs reflect question sampling only, not run-to-run variance.
    • There is no non-spatial SFT control, so the large drops on I1 may reflect format collapse rather than spatial interference.
Questions for authors
  1. Which transfer numbers are correct? Please reconcile Figure 4 with Table 3, Figure 5b, and the text, and restate the conclusions using the corrected matrix. (This would most change my assessment.)
  2. Can C2/C3 be learned at all? Please report results from stronger or larger models, from more training data or higher resolution, and human accuracy.
  3. Why does EgoOrient-InternVL3-2B exactly match InternVL3-2B on eight additional-entry columns of Table 2? Please also report per-task n; D-2D appears to have about 23 questions.
  4. Table 4 has nine values per row but eight column labels, and the 3DSRBench overall score cited in the text is missing. Please fix the table.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.