acceptodds
Under review as a conference paper at ICLR 2027

Show, Don't Tell: Evaluating Spatial Cognition in Generative Pixels Rather Than LLM Text

Abstract

Spatial intelligence enables agents to move beyond static semantic understanding and interact with the physical world. Many spatial tasks are grounded in continuous visual scenes, where locations, regions, and paths are more naturally expressed by pointing, marking, or drawing than by precise coordinates or discrete textual symbols. Yet existing spatial benchmarks usually require coordinates, options, or text, creating an answer-interface mismatch that makes it difficult to evaluate image-generation models under the same task semantics as text-output VLMs, despite their ability to externalize spatial judgments in pixel space. To address this problem, we propose ProVisE (Protocolized Visual Evaluation), a benchmark-agnostic framework that elicits protocol-constrained visual answers from image-generation models, parses them into structured predictions for task-specific scoring, and extends to previously unconfigured benchmarks through Agentic construction and validation of executable generation–parser protocols. We further introduce SpatialGen-Bench, a curated diagnostic benchmark comprising 470 samples across 14 spatial subtasks, four capability levels, and diverse answer forms. We evaluate representative text-output VLMs and image-generation models in a unified setting and validate Agentic protocol construction on six external spatial benchmarks. Experimental results show that image-generation models are competitive when spatial answers can be externalized in pixel space, while text-output VLMs retain a clear advantage in compositional spatial reasoning. These findings reveal complementary strengths of pixel-space expression and text-based reasoning, while ProVisE and SpatialGen-Bench establish a unified testbed for generative spatial evaluation and future studies of spatial cognition in image-generation models.

Primary area:
datasets and benchmarks
Keywords:
spatial cognition, spatial reasoning, image generation, vision-language models, visual evaluation, datasets and benchmarks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aMm8OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper mainly solves the problem of how to evaluate spatial abilities in image-generation models when existing benchmarks expect textual answers, coordinates, or option labels. It introduces ProVisE, which pairs constrained visual-answer prompts with parsers that convert generated markings, masks, depth maps, and trajectories into benchmark-compatible predictions. It also propose a benchmark SpatialGen-Bench, containing 470 samples across 14 task types. Through extensive experiments covering 20 text-answering and 11 visual-answering systems, several interesting findings such as the complementarity between visual and text are revealed.

Critical strengths

I think a reasonable way of evaluating the spatial intelligence of visual generation models is important and appreciate the idea that the authors proposed that using a set of fixed protocols to make the the spatial intelligence of visual generation models can be evaluated using existing benchmarks for VLMs.

Critical weaknesses
  1. One major concern is how the author distinguish the wrong answer from model's spatial intelligence and instruction following of protocols. Having fixed protocols makes it possible to evaluate visual generation models on benchmarks that is originally designed for VLMs. However, it also brings a more constrained evaluation on the spatial intelligence of visual generation models as only if they understand the protocol correctly and follow the protocol, they can reach to a stage that fully reveals their spatial intelligence.
  2. While the idea of using protocols to translate visual into text for evaluation if interesting, I'm wondering if this is a reasonable for evaluating visual generation model's spatial intelligence as they are mainly trained on how to generate more realistic figures given a text prompt instead of treating the text prompt as a question and give the answer to the question.
  3. Hallucination problem also matters. What if the visual generation model outputs correct answer (based on the protocol), but modifies key information such as object color, layouts, etc in the generated image?
Questions for authors

As illustrated in weaknesses.

Reviewer_kjnROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ProVisE, which converts protocol-constrained visual responses from image-generation models into task-specific predictions. It also presents SpatialGen-Bench and evaluates a broad set of image-generation models and VLMs. The paper identifies a meaningful limitation of evaluating spatial cognition for generative vision models, which is the mismatched answer interface. However, my main concern is construct validity. Stable scores do not establish that the parsers faithfully recover the answers expressed in generated images. The text v.s. visual comparison also relies mainly on different model families, which confounds output modality with model capability.

Critical strengths
  1. The paper identifies a meaningful interface mismatch in spatial evaluation.
  2. The framework is demonstrated at meaningful scale. The protocol pool covers points, masks, depth maps, relations, states, and trajectories. The agentic builder also offers a plausible route to extend benchmarks. The experiments cover 31 model-interface systems and various tests.
Critical weaknesses
  1. Parser stability does not establish parser validity.

    The paper shows that many parsers are deterministic and stable, but it does not independently verify that they recover the answers actually expressed in generated images. This is particularly important for protocols involving CLIP, OCR, auxiliary VLMs, or fallback parsing. Smoke-test success and repeated-parse agreement establish executability, not semantic correctness. Because all benchmark scores and the failure attribution analysis depend on these parsed predictions, the lack of human validation directly affects the credibility of the paper’s main results.

  2. The modality comparison is confounded by model selection.

    GPT-5.4 and GPT Image 2 are not a controlled interface pair. The only clearly matched pair, SenseNova, scores 46.68 in text mode and 32.22 in visual mode. The paper should evaluate several unified models through both interfaces under matched inputs and budgets. The current conclusions support differences between current text-answering and image-answering systems, but not differences caused by output modality.

Questions for authors
  1. How often does each parser agree with blinded human judgments of the answer expressed in the generated image?
  2. What proportion of the main results depends on CLIP, OCR, or auxiliary/fallback VLMs?
  3. Can the authors provide matched text-versus-visual results for several unified models?
  4. Could the authors provide a complete table mapping each of the 14 SpatialGen-Bench tasks to its generation prompt, visual response format, parser, output schema, scoring metric, invalid-output rules, and external dependencies for clairty?
  5. How does Fixed-Slot Coding preserve task-grounded spatial evidence rather than merely encode a symbolic choice through a colored slot?
  6. Beyond smoke-test and parse success, how to evaluate semantic fidelity of agentically constructed protocols evaluated?
Reviewer_GozDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address answer-interface mismatch between existing spatial benchmarks and image-generation models’ native visual answers, this paper propose ProVisE, a benchmark-agnostic framework that elicits protocol-constrained visual answers from image-generation models, and further introduce SpatialGen-Bench, comprising 470 samples across 14 spatial subtasks, four capability levels, and diverse answer forms. With substantial experiments comparing generative models and VLMs, this paper found that image-generation models are competitive when spatial answers can be externalized in pixel space, while text-output VLMs retain a clear advantage in compositional spatial reasoning.

Critical strengths

1.This paper introduces ProVisE, a protocol-based framework for evaluating spatial capabilities through visual answers, together with SpatialGen-Bench, which enables comparisons across diverse spatial tasks and answer formats.

2.SpatialGen-Bench covers 14 subtasks spanning four levels of spatial capability, the evaluation also includes a broad selection of text-answering and visual-answering systems, which is relatively comprehensive.

3.This paper systematically compare image-generation models and text-output VLMs across spatial capabilities, and found that visual answering is competitive when conclusions can be externalized directly in pixels, whereas text-output VLMs remain stronger in compositional spatial reasoning.

Critical weaknesses
  1. The text-answering and visual-answering systems differ in architecture, training data, scale, and inference procedures, although they are evaluated on shared tasks and scoring rules, the observed performance differences cannot be attributed specifically to the answer interface.

  2. Insufficient validation of protocol and parser accuracy. Passing smoke tests and successfully parsing an output do not establish that the recovered prediction faithfully represents the model’s visual answer.

  3. Limited sample size. SpatialGen-Bench contains 470 examples across 14 subtasks, leaving relatively few examples per task and limiting confidence in fine-grained rankings.

  4. The Agentic Protocol Construction section in Figure 4 is insufficiently clear. Besides, the aesthetic quality of the figures generally needs improvement, especially some labels and texts are too small to read comfortably.

Questions for authors

Please refer to the weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.