acceptodds
Under review as a conference paper at ICLR 2027

SUBTEXT: Can Image Generators Infer What Captions Leave Implicit?

Abstract

Can image generators infer an outcome that a caption implies but does not state? Captions routinely omit direct descriptions of outcomes and rely on the reader to infer them from context. For example, a caption describing a chocolate bar left on a dashboard on a hot afternoon implies that the bar has melted. We introduce SUBTEXT, a benchmark of 140 minimal image pairs that differ in one physical or socio-emotional outcome within an otherwise matched scene. Each image has an implicit caption that specifies only the antecedent context and an explicit caption that is identical except for an inserted outcome term such as "melted". Following work on diffusion classifiers, we use seven frozen image generators as zero-shot classifiers that match captions to images and images to captions. We compare them with three contrastive models, Claude Fable 5.1, and human participants. With implicit captions, the generators select the correct image at near-chance rates and fall below chance when both captions of a pair must be matched. Human participants and Claude Fable 5.1 select the correct image in at least 93% of cases. Explicit captions significantly improve accuracy for six of the seven generators, which indicates that inference adds difficulty beyond visual discrimination. Unlike prior results on compositional benchmarks, the generators show no advantage over contrastive models, even when the two share a text encoder. These results suggest that learning to generate realistic images does not by itself teach a model to reliably link context to visible outcomes. SUBTEXT therefore provides a controlled test of whether future generators close the gap between what a caption states and what it implies.

Primary area:
datasets and benchmarks
Keywords:
text-to-image models, diffusion classifiers, image-text matching, implicit inference, vision-language benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7mVEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a new benchmark SUBTEXT which is designed to study how sensitive models are to implicit information in the captions. Specifically, the paper examines image generation models and compares their performance with contrastive models, one proprietary model, and human judgements.

The benchmark consists of 140 pairs of synthetic images, each with human-made implicit and explicit captions which, either implicitly or explicitly, carry a difference between the other image in its pair. The task is then to match each caption with the image it describes correctly.

The paper finds that image generators perform at about chance rate, while contrastive models perform slightly better, but much below humans and Claude Fable 5.1, and performance on explicit captions is higher for all models.

Critical strengths

The paper provides an interesting study for subtle context cues and implied information in image captions, and how different and models interpret implicit information.

The benchmark is thoughtfully and carefully constructed and contains interesting examples.

Interesting and careful analysis of model behavior towards implicit text information revealed with the benchmark dataset.

Critical weaknesses

While the paper specifically studies the performance of image generators on the proposed benchmark, it does not connect the link between benchmark performance and image generation quality. Instead, the paper leaves analyzing images generated from the implicit and explicit captions to future work. However if the focus of the paper is on investigating image generators as classifiers, and how it affects image generation quality, then I feel analyzing the generated images seems rather important.

What are some benefits of using generative classifiers over other kinds of models (e.g. contrastive models, VLMs?) for this kind of task?

Questions for authors

Since one of the findings is that the contrastive models outperform the image generators, were image-text alignment models (e.g. VQAScore, ImageScore, etc) used in addition to the contrastive models? I ask this mainly because contrastive models have been known to not consider details in captions and prefer more generic text. If so, how do they perform?

The paper leaves generating images from the implicit and explicit captions up to future work, but I am still curious if this was tried, and if the authors tried analyzing such generated images. 
Why does Claude do better than humans in some cases? For this task, is there a big gap in model performance between image generators and vision-language models?

1 comment

Dear reviewer, thank you for your thoughtful questions and we hope to adequately address them.

Based on your feedback, we have tested four image-text alignment models on our benchmark. We were not sure which model "ImageScore" refers to, so we took it to mean ImageReward. If you had another scorer in mind, we would be happy to add it. Here are the results we have so far alongside the existing contrastive model scores for comparison:

ModelImage choice, implicitImage choice, explicitImage score, implicitGroup score, implicit
CLIP ViT-L/1460.066.824.310.7
OpenCLIP ViT-H/1456.465.716.45.7
SigLIP 256.870.017.94.3
VQAScore (CLIP-FlanT5-XXL)81.890.766.436.4
ImageReward55.764.316.47.9
PickScore55.766.113.65.0
HPSv2.158.968.620.76.4

Note: Chance is 50% for image choice, 25% for the image score, and 16.7% for the group score.

VQAScore beat every contrastive model in implicit image choice by 21.8 to 25.4 points (Holm p ≤ 5.6 × 10⁻¹¹), but explicit captions still raised its image choice score by 8.9 points. ImageReward reads the caption token by token, and PickScore and HPSv2.1 are fine-tuned from OpenCLIP. Yet none of these three models differ detectably from the contrastive models (Holm p ≥ 0.76), so it appears that the large gain only shows up when a large language model reads the caption. One possible explanation is that the language model supplies the missing inference, although we cannot confirm this from a single model's results.

We have also run the generation experiment that you and the other reviewers asked about. In our procedure, each generator drew two images from each of the 560 captions, using the steps and guidance their respective developers recommend. We used a vision-language judge (DeepSeek-V4.1-Flash) to judge which outcome each image showed, for example whether the chocolate bar is melted or solid. The judge model was never shown the caption. After testing on the benchmark images, DeepSeek-V4.1-Flash's accuracy averaged 95.0% for physical and 87.8% for socio-emotional images.

Here are the results from our generation experiment:

Model% of images showing the stated outcome (generated from explicit captions)% of images showing the implied outcome (generated from implicit captions)% of pairs with both images correct (generated from explicit captions)% of pairs with both images correct (generated from implicit captions)
SD 1.573.956.141.314.0
SD 2.1 Base73.060.640.915.4
SDXL74.361.844.820.0
SD3 Medium72.154.936.07.2
SD3.5 Large78.459.550.013.1
FLUX.2 dev89.069.175.133.7
Qwen-Image 2.181.859.859.516.0

Note: The judge decided which outcome each image showed. Chance is 50% for single images and 25% for pairs.

Explicit captions beat implicit captions in generation for all seven generators (Holm p ≤ 4.4 × 10⁻⁵), and we find the same pattern in the physical pairs alone, where the judge is more reliable (Holm p ≤ 0.0033).

We use generative classifiers because they allow us to test the image generator itself for contextual understanding. The score is how much a caption reduces the frozen generator’s denoising error, so it reflects what that generator has learned about linking captions to images, with no additional training. Prior work also found that generative classifiers can match or beat CLIP on compositional tasks, since the generator reads the whole caption through cross-attention instead of a single embedding. Contrastive models and VLMs are separate and cannot tell us what a generator understands.

Regarding why Claude does better than people, we believe that the advantage partly reflects how the two are scored. Fable 5.1 must answer both display orders correctly, while each human answer is a majority vote of only two to four participants. People also match or beat Fable 5.1 on the implicit text score (93.2% against 85.7%) and group score (85.3% against 85.0%), so it appears that the advantage is only on some measures.

There is a sizable gap between generators and VLMs. In implicit image choice, the generators reach 48.6% to 55.7% against 99.3% for Fable 5.1. The contrastive models reach 56.4% to 60.0%, only slightly above the generators, and VQAScore falls in between at 81.8%.

If we have the opportunity to revise our paper, we will add in the results of our generation experiment and the results of the image-text alignment models.

Reviewer_JPViOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces SUBTEXT, a benchmark of 140 minimal image pairs that differ in a single outcome. Each image has an implicit caption that gives only the context and an explicit caption that adds the outcome term. Seven image generators are evaluated as zero-shot classifiers using caption-conditioned denoising error. With implicit captions they perform near chance, far below humans and Claude Fable 5.1 and no better than contrastive models, while explicit captions improve accuracy for six of the seven. The authors conclude that realistic image generation does not imply the ability to infer visual outcomes from context.

Critical strengths
  • New benchmark design. SUBTEXT is built from minimal pairs that differ only in a single outcome within the same scene, which isolates the one outcome that must be inferred from context. Because it evaluates fixed images without a judge model, the evaluation is reproducible and interpretable. The high accuracy of human participants (93.0%) and Claude Fable 5.1 (99.3%) shows that the task is solvable and that the correct answers are unambiguous.
  • Paired implicit/explicit captions. The explicit caption adds only the outcome term and serves as a control condition. This allows captions that state an outcome to be compared with captions that only imply it, on identical images. Existing compositional benchmarks do not have this setup, and it gives future work an evaluation framework to reuse.
Critical weaknesses
  1. Benchmark scale and reliability of data construction
  • 140 pairs are not enough to measure this capability reliably. The per-category confidence intervals are wide, which makes subgroup analyses such as the finding that five of the seven generators reverse their lean on the 12 event-level scene pairs difficult to trust. Moreover, each image has only one caption of each type, so sensitivity to caption wording has not been tested.
  • All 280 images were generated and edited with gpt-image-2 and iteratively refined according to the authors’ judgment. The style or editing artifacts of a single generator may act as a distribution shift. Refinement based on the authors’ judgment may also have favored pairs that are easy for humans and frontier VLMs to recognize. Therefore, it cannot be ruled out that part of the performance gap stems from the data source. At least a subset of the pairs should be replicated with real photographs or with a different generator.
  1. Lack of qualitative evidence
  • Figure 1 is the only example shown. Supporting a gap of roughly 40–50 percentage points requires per-model success and failure cases as well as per-pair score distributions. Even with explicit captions, image choice reaches only 55.7–65.4% for the generators and 65.7–70.0% for the contrastive models. Without examples, it is impossible to judge whether this reflects a genuine limitation of the models or a problem with the scorer or the image quality.
  1. No evidence that the metric reflects generative ability
  • The experiments only measure matching, with generators used as classifiers. The human evaluation is likewise a task of choosing which benchmark image matches a caption, so it only shows that the task is solvable. There is therefore no evidence that these scores reflect the generators’ actual inferential ability during generation. The authors acknowledge in Section 6.5 that validation under actual generation is a limitation, yet the title and conclusion generalize to imagegenerators as a whole. The authors should either narrow the scope of their claims or add a supplementary experiment, even at a small scale. Such an experiment would generate images from implicit captions, check whether the implied outcome appears, and compare the result with classifier accuracy.
Questions for authors
  1. Section 3.3 states that many pairs were constructed by editing one image into the other. In each pair, which image is the original and which is the edited version? Could you report implicit image choice and the direction of the fixed preference (toward image A or B) separately by editing direction and resampling history?

  2. Were the images generated and edited with gpt-image-2 using the implicit captions, or using prompts that state the outcome? If implicit captions alone produced the intended outcomes, this suggests that at least some generators can infer implied outcomes. How do you reconcile this with the suggestion that learning to generate realistic images does not by itself teach models to link context to visible outcomes?

3 comments

Dear reviewer, thank you for your detailed review, and we hope to address your concerns.

We agree that 140 pairs presents a limitation for subgroup analyses, so we have drafted 60 new caption pairs with implicit captions that require two or three steps of inference. To test sensitivity to wording, we have drafted two new phrasings of each of the 140 caption pairs. In our revision, we will present findings such as the lean reversal on the 12 event-level pairs as exploratory given the limitation. Our main comparison is better supported, with each model's implicit image choice having a 95% interval of about ±3 points, and gains of 4 to 7 points from explicit captions are detectable with 80% power. As for the image source, we are currently creating 50 new pairs with a second generator, Google's Nano Banana, which will also expand our benchmark size.

For qualitative evidence, here are three pairs along with their generation accuracy across the seven generators. People chose the correct image for all three. We will add visual examples of these pairs and their captions in our revision.

  • C1_29, a copper roof installed in 1920 or this spring. Accuracy is 100% when the explicit captions add "green-patinated" and "shiny orange", against 56.5% with the implicit captions.
  • C2_26, tulips and a cactus left unattended for a week or watered that morning. Accuracy is 74.0% with "wilted" and "fresh" stated explicitly against 46.4% when implied.
  • C4_29, a boy who wanted a game console but got socks, or got the socks he had asked for. Accuracy is 97.4% with "disappointed" and "excited" stated explicitly, against 46.2% when implied.

We have completed the generation experiment you and the other reviewers mentioned. Each generator drew two images from each of the 560 captions, and a vision-language judge (DeepSeek-V4.1-Flash) that never saw the caption judged which outcome each image showed. We tested the judge's accuracy on the benchmark images, where it averaged 95.0% for physical and 87.8% for socio-emotional images. The implied outcome was generated more often than chance, but 12 to 22 points less often than with explicit captions (Holm p ≤ 4.4 × 10⁻⁵). Our revision will narrow our conclusions to the generators we tested.

Model% of images showing the stated outcome (generated from explicit captions)% of images showing the implied outcome (generated from implicit captions)% of pairs with both images correct (generated from explicit captions)% of pairs with both images correct (generated from implicit captions)
SD 1.573.956.141.314.0
SD 2.1 Base73.060.640.915.4
SDXL74.361.844.820.0
SD3 Medium72.154.936.07.2
SD3.5 Large78.459.550.013.1
FLUX.2 dev89.069.175.133.7
Qwen-Image 2.181.859.859.516.0

Note: The judge decided which outcome each image showed. Chance is 50% for single images and 25% for pairs.

Our images were generated using a detailed starting prompt that explicitly laid out the entire scene, which we then iterated on over multiple rounds. None of the benchmark images were generated from our implicit captions. Image A was edited from image B in 77 pairs and B from A in 32, while 31 pairs were generated during earlier work before we logged each generation and edit step. The source image refers to the one the other image was edited from. We show editing direction for the socio-emotional pairs, where both directions are common (37 and 28 pairs). In physical pairs only 4 pairs had image B edited from image A, and image B was always the unchanged physical state (e.g. intact chocolate instead of melted chocolate), so a preference for the source image could instead be a preference for the unchanged state.

ModelImplicit image choice, A edited from B% of these pairs where the model picks source image B for both implicit captionsImplicit image choice, B edited from A% of these pairs where the model picks source image A for both implicit captions
SD 1.558.151.462.560.7
SD 2.1 Base55.443.250.053.6
SDXL54.145.960.753.6
SD3 Medium52.748.650.064.3
SD3.5 Large55.437.858.946.4
FLUX.2 dev56.837.853.639.3
Qwen-Image 2.144.627.051.835.7

Pairs by resampling history:

ModelImplicit image choice, first tryImplicit image choice, resampled
SD 1.550.055.5
SD 2.1 Base46.755.5
SDXL53.355.0
SD3 Medium46.750.0
SD3.5 Large56.754.1
FLUX.2 dev43.353.2
Qwen-Image 2.146.749.5

The first-try and resampled pairs do not differ detectably (p ≥ 0.15). There is a slight lean toward the source image for SD 1.5, but this lean cannot explain our main results, since implicit and explicit captions are scored on the same images.

Thank you for the detailed response and for adding the generation experiment. However, my third concern remains not fully resolved. I will maintain my original scores.

Dear reviewer, thank you for the follow-up. Comparing the two readouts directly, we find that the classifier score and generation accuracy agree in direction. In generation, explicit captions raise the share of images showing the intended outcome by 12 to 22 points for all seven generators. In classification, explicit captions raise image choice by 4.3 to 12.5 points, and the paper’s tests detect this gain for six of the seven generators. Additionally, the generative models often chose or generated the same outcome for both implicit captions of a pair, even though the captions imply different outcomes. This occurred in 76.4% to 90.7% of pairs as classifiers and 57% to 82% of pairs in generation.

However, the classification and generative experiments do not agree pair by pair, with Spearman correlations of only 0.01 to 0.20. FLUX.2 dev is the strongest generator in generation while scoring only slightly above chance as a classifier (51.8% implicit image choice). We agree that the classifier score alone is not direct evidence of generative ability.

We will therefore revise the paper in three ways. First, we will present the classifier score as a measure of how the generator matches captions to images, not as a representation of generative ability. Second, generation accuracy will become our direct measure of what a generator draws, which we will report alongside the classifier results. Third, we will change the title, abstract, and conclusion to refer to the seven generators we tested instead of generalizing to image generators as a whole.

Thank you again for raising this concern, and we hope these changes address it.

Reviewer_79LDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SUBTEXT, a benchmark of 140 minimal image pairs designed to evaluate whether image generation models can infer outcomes implicitly described by captions. The authors evaluate seven image generators as zero-shot classifiers using denoising-based image–text matching, alongside contrastive models, a frontier MLLM, and human participants. Results reveal that generative classifiers perform near chance on implicit reasoning tasks, while Claude Fable 5.1 and humans achieve substantially higher accuracy.

Critical strengths
  1. Interesting benchmark formulation. The paper investigates whether image generators can infer implicit physical and socio-emotional outcomes from contextual descriptions. The use of minimal image pairs and implicit/explicit caption comparisons provides a controlled setting for studying this capability.

  2. Relative comprehensive model evaluation. The authors evaluate diverse image generation architectures, including recent diffusion and flow-based models, alongside contrastive models, human participants, and a frontier MLLM. The experimental analysis provides useful insights into the limitations of generative classifiers.

Critical weaknesses
  1. Limited benchmark difficulty and potential early saturation. Fable 5.1 already achieves 99.3% implicit image-choice accuracy and 94.6% implicit caption-choice accuracy, suggesting that the benchmark is nearly saturated by an existing frontier MLLM on these measures. While the group-level metrics remain more challenging, the dataset contains only 140 image pairs, which may limit its ability to distinguish increasingly capable models.

More challenging examples involving multi-step causal reasoning, interacting physical conditions, or complex social inference would strengthen its broader value, and are actually needed.

  1. Limited evidence connecting classification results to image generation capabilities. The paper evaluates image generators exclusively as zero-shot classifiers using denoising-based matching scores, rather than assessing their actual generated images. Although this evaluation reveals weaknesses in generative classification, it remains unclear whether these results accurately reflect the models' ability to generate the implied outcomes. Direct relevant generation experiments with implicit captions are needed.
Questions for authors

Given that Claude Fable 5.1 already achieves nearly perfect implicit image-choice accuracy, how do the authors envision SUBTEXT remaining useful for evaluating future frontier models? Could more challenging reasoning cases be incorporated?

1 comment
AuthorsOpenReview ↗

New generation experiment and reply

Dear reviewer, thank you for your careful reading and feedback.

Regarding frontier models and benchmark saturation, our intent was for Fable 5.1 and people to serve as references to show that the benchmark can be solved from the captions and images, so we expected them to score nearly perfectly. Our benchmark is meant to be diagnostic for generative and contrastive models, which perform far below the ceiling. We designed each pair to keep the scene fixed and alter only one object/subject's outcome so the benchmark isolates the ability of generative and contrastive models to infer a visible outcome from context. The seven generators choose the correct image for 48.6% to 55.7% of implicit captions, and all seven fall below chance on the pair scores.

We fully accept your suggestion to extend the benchmark to involve more complex scenarios that require multi-step reasoning, and we appreciate your supportive read of our current benchmark formulation. So far, we have drafted 60 new caption pairs whose implicit captions require 2-3 steps of inference. To illustrate the idea with one of our examples:

  • Original implicit caption: "The chocolate bar had been left on the dashboard during a hot afternoon."
  • Multi-step caption: "The chocolate bar had been left on the dashboard on a July afternoon in Phoenix."

Additionally, we have drafted two new phrasings for each of the 140 original caption pairs. All of these captions use the existing image pairs. We are currently in the process of creating 50 additional pairs (new images) using Google's Nano Banana, which will include scenarios with more complex social reasoning and physical interactions.

We agree that classification does not measure the models' generation capability, so we have added a generation experiment. Our procedure is as follows:

Each generator drew two images from each of the 560 captions in our original benchmark. Afterward, we used a vision-language judge (DeepSeek-V4.1-Flash) to judge which outcome each image showed, for example whether the chocolate bar is melted or solid. The DeepSeek judge was never shown the caption, and we made sure to first check the judge's performance on the benchmark images. On those, its accuracy averaged 95.0% for physical and 87.8% for socio-emotional images.

Here are our generation results:

Model% of images showing the stated outcome (generated from explicit captions)% of images showing the implied outcome (generated from implicit captions)% of pairs with both images correct (generated from explicit captions)% of pairs with both images correct (generated from implicit captions)
SD 1.573.956.141.314.0
SD 2.1 Base73.060.640.915.4
SDXL74.361.844.820.0
SD3 Medium72.154.936.07.2
SD3.5 Large78.459.550.013.1
FLUX.2 dev89.069.175.133.7
Qwen-Image 2.181.859.859.516.0

Note: The judge decided which outcome each image showed. Chance is 50% for single images and 25% for pairs.

Explicit captions beat implicit captions in generation for all seven generators (Holm p ≤ 4.4 × 10⁻⁵), which aligns with the paper’s classification tests. In both the generation and classification tests, the generative models often give the same answer for both implicit captions, regardless of what the context implies. As classifiers, the generators choose the same image for both implicit captions in 76.4% to 90.7% of pairs, although each caption fits a different image. In generation, the two images generated from a pair’s implicit captions display the same outcome in 57% to 82% of generated pairs, even though the captions imply opposite outcomes. With explicit captions, this happens in 19% to 55% of pairs. In physical pairs, where the DeepSeek judge is more reliable, explicit captions beat implicit captions for all seven generators (Holm p ≤ 0.0033). In the generation experiment, FLUX.2 dev performed the best among the models tested, yet it is near chance as a classifier, so the two tests do not rank the models the same way.

If given the opportunity, we plan on making three main changes based on your feedback. First is an expanded benchmark of 190 image pairs (the 140 original pairs and 50 new pairs made with Google's Nano Banana), with 60 multi-step caption pairs and two new wordings of each of the 140 original caption pairs. After the benchmark expansion is complete, we plan on building off of our current generation experiment by adding the new captions to the run. We also plan on adding the generation code, the judge questions, and the outcome labels to the supplement, and we will discuss statistical power among the limitations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.