- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents a new benchmark SUBTEXT which is designed to study how sensitive models are to implicit information in the captions. Specifically, the paper examines image generation models and compares their performance with contrastive models, one proprietary model, and human judgements.
The benchmark consists of 140 pairs of synthetic images, each with human-made implicit and explicit captions which, either implicitly or explicitly, carry a difference between the other image in its pair. The task is then to match each caption with the image it describes correctly.
The paper finds that image generators perform at about chance rate, while contrastive models perform slightly better, but much below humans and Claude Fable 5.1, and performance on explicit captions is higher for all models.
Critical strengths
The paper provides an interesting study for subtle context cues and implied information in image captions, and how different and models interpret implicit information.
The benchmark is thoughtfully and carefully constructed and contains interesting examples.
Interesting and careful analysis of model behavior towards implicit text information revealed with the benchmark dataset.
Critical weaknesses
While the paper specifically studies the performance of image generators on the proposed benchmark, it does not connect the link between benchmark performance and image generation quality. Instead, the paper leaves analyzing images generated from the implicit and explicit captions to future work. However if the focus of the paper is on investigating image generators as classifiers, and how it affects image generation quality, then I feel analyzing the generated images seems rather important.
What are some benefits of using generative classifiers over other kinds of models (e.g. contrastive models, VLMs?) for this kind of task?
Questions for authors
Since one of the findings is that the contrastive models outperform the image generators, were image-text alignment models (e.g. VQAScore, ImageScore, etc) used in addition to the contrastive models? I ask this mainly because contrastive models have been known to not consider details in captions and prefer more generic text. If so, how do they perform?
The paper leaves generating images from the implicit and explicit captions up to future work, but I am still curious if this was tried, and if the authors tried analyzing such generated images. Why does Claude do better than humans in some cases? For this task, is there a big gap in model performance between image generators and vision-language models?
Dear reviewer, thank you for your thoughtful questions and we hope to adequately address them.
Based on your feedback, we have tested four image-text alignment models on our benchmark. We were not sure which model "ImageScore" refers to, so we took it to mean ImageReward. If you had another scorer in mind, we would be happy to add it. Here are the results we have so far alongside the existing contrastive model scores for comparison:
Note: Chance is 50% for image choice, 25% for the image score, and 16.7% for the group score.
VQAScore beat every contrastive model in implicit image choice by 21.8 to 25.4 points (Holm p ≤ 5.6 × 10⁻¹¹), but explicit captions still raised its image choice score by 8.9 points. ImageReward reads the caption token by token, and PickScore and HPSv2.1 are fine-tuned from OpenCLIP. Yet none of these three models differ detectably from the contrastive models (Holm p ≥ 0.76), so it appears that the large gain only shows up when a large language model reads the caption. One possible explanation is that the language model supplies the missing inference, although we cannot confirm this from a single model's results.
We have also run the generation experiment that you and the other reviewers asked about. In our procedure, each generator drew two images from each of the 560 captions, using the steps and guidance their respective developers recommend. We used a vision-language judge (DeepSeek-V4.1-Flash) to judge which outcome each image showed, for example whether the chocolate bar is melted or solid. The judge model was never shown the caption. After testing on the benchmark images, DeepSeek-V4.1-Flash's accuracy averaged 95.0% for physical and 87.8% for socio-emotional images.
Here are the results from our generation experiment:
Note: The judge decided which outcome each image showed. Chance is 50% for single images and 25% for pairs.
Explicit captions beat implicit captions in generation for all seven generators (Holm p ≤ 4.4 × 10⁻⁵), and we find the same pattern in the physical pairs alone, where the judge is more reliable (Holm p ≤ 0.0033).
We use generative classifiers because they allow us to test the image generator itself for contextual understanding. The score is how much a caption reduces the frozen generator’s denoising error, so it reflects what that generator has learned about linking captions to images, with no additional training. Prior work also found that generative classifiers can match or beat CLIP on compositional tasks, since the generator reads the whole caption through cross-attention instead of a single embedding. Contrastive models and VLMs are separate and cannot tell us what a generator understands.
Regarding why Claude does better than people, we believe that the advantage partly reflects how the two are scored. Fable 5.1 must answer both display orders correctly, while each human answer is a majority vote of only two to four participants. People also match or beat Fable 5.1 on the implicit text score (93.2% against 85.7%) and group score (85.3% against 85.0%), so it appears that the advantage is only on some measures.
There is a sizable gap between generators and VLMs. In implicit image choice, the generators reach 48.6% to 55.7% against 99.3% for Fable 5.1. The contrastive models reach 56.4% to 60.0%, only slightly above the generators, and VQAScore falls in between at 81.8%.
If we have the opportunity to revise our paper, we will add in the results of our generation experiment and the results of the image-text alignment models.