- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper mainly solves the problem of how to evaluate spatial abilities in image-generation models when existing benchmarks expect textual answers, coordinates, or option labels. It introduces ProVisE, which pairs constrained visual-answer prompts with parsers that convert generated markings, masks, depth maps, and trajectories into benchmark-compatible predictions. It also propose a benchmark SpatialGen-Bench, containing 470 samples across 14 task types. Through extensive experiments covering 20 text-answering and 11 visual-answering systems, several interesting findings such as the complementarity between visual and text are revealed.
Critical strengths
I think a reasonable way of evaluating the spatial intelligence of visual generation models is important and appreciate the idea that the authors proposed that using a set of fixed protocols to make the the spatial intelligence of visual generation models can be evaluated using existing benchmarks for VLMs.
Critical weaknesses
- One major concern is how the author distinguish the wrong answer from model's spatial intelligence and instruction following of protocols. Having fixed protocols makes it possible to evaluate visual generation models on benchmarks that is originally designed for VLMs. However, it also brings a more constrained evaluation on the spatial intelligence of visual generation models as only if they understand the protocol correctly and follow the protocol, they can reach to a stage that fully reveals their spatial intelligence.
- While the idea of using protocols to translate visual into text for evaluation if interesting, I'm wondering if this is a reasonable for evaluating visual generation model's spatial intelligence as they are mainly trained on how to generate more realistic figures given a text prompt instead of treating the text prompt as a question and give the answer to the question.
- Hallucination problem also matters. What if the visual generation model outputs correct answer (based on the protocol), but modifies key information such as object color, layouts, etc in the generated image?
Questions for authors
As illustrated in weaknesses.