acceptodds
Under review as a conference paper at ICLR 2027

Seen but Unnamed: How Caption Coverage Affects Composition in Diffusion Models

22 views

Abstract

Compositional generation asks a model to satisfy a prompt that names several objects at once, each with its attributes and relation to the scene, and diffusion models occasionally and unpredictably fail at such a task. Two kinds of accounts attempt to explain why, in the hopes of arriving at a broader claim of where compositionality originates. Architectural claims attribute composition to a property of the denoiser such as locality, under which each output pixel depends only on nearby pixels. Data-centric claims attribute it to the training data, most often to image coverage, the attribute combinations in the training images. However, a diffusion model also learns from its captions, and human captions leave most of the objects in an image unnamed. We introduce and vary caption coverage, the fraction of multi-object training images whose caption names at least two of their objects, while holding other experimental settings constant. We find that diffusion models follow a prompt that names several objects only when enough of their captions name several objects together, while image coverage mainly governs transfer to unseen color and shape pairs. We restore composition by summing a failing model's single-object predictions, and conclude that the value of captions lies in teaching a model to read several objects from one prompt. Finally, across architectural ablations, we find that cross-attention and sampler design set the caption coverage a model needs, and locality supports prompts with more objects without predicting which models trained on the same data succeed. In contrast, anyone can count caption coverage before training and raise it by rewriting captions.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
compositional generalization, diffusion models, text-to-image generation, conditional generation, caption, coverage, conditioning, catastrophic neglect, attribute binding, score locality, systematicity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jcQFOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies caption coverage—the proportion of multi-object training images whose captions name multiple objects—as an actionable factor in diffusion models’ compositional ability. Its central contribution is to distinguish learning individual visual concepts from learning to interpret several object descriptions jointly in one prompt. By varying captions while keeping images fixed, it clarifies why exposure to complex scenes alone may not teach reliable multi-object generation. The work connects this gap to how models route object-specific conditioning and explains how architecture and sampling influence the supervision needed. It gives the community a measurable target for dataset auditing and recaptioning, alongside a framework for distinguishing failures of visual concept knowledge from failures of joint prompt interpretation.

Critical strengths
  1. This paper takes a look back the the fundamentals of the diffusion models. It delves into the topic of caption coverage, focuses on the object level accuracy and coverage, providing a complimentary perspective of the image-caption alignment.
  2. Training is constructed on the controlled dataset, providing an in-depth analysis of the diffusion models from scratch instead of simply peeking the pretrained models.
Critical weaknesses
  1. The author is recommend to clearly identify the range of discussion. It is better to clearly confine your discussion under traditional transformer-based diffusion models without external harness or agentic systems, as it will be a completely differenty story if an agentic system providing layout is available by the side.
  2. Is the conclusion really generalizable to prevailing diffusion models, if we conduct the experiments on the synthetic datasets like CLEVR, ShapeWorld and MS-COCO, as the distribution of the former two is completely different from the from the data source training modern diffusion models, such as LAION. The author is recommended to extend his conclusion to the prevailing pretrained diffusion models w.r.t. the dataset statistics and their actual performances.
Questions for authors
  1. Actually concepts have correlations. How can the authors seaprate the contribution to the explicit mentionings of the object, and the objects contained in the descriptions of the backgrounds? (e.g. Even in a description such as A dinning room with a round table, it is highly likely that the generated image would contain chairs inside. )
  2. Can the authors extend their discovery or their proposed method to other concepts in generation, such as style, color and texture?
  3. What does the distinction (note that this is not equivalent to the difference) lies between conditioning (Farid et.al, 2025) and captions, as the majority of the conditioning lies in text, and what is the difference between this paper's contribution and confining Farid et. al into caption only? Does the analysis of this paper really offer a complimentary perspective?

Note that I will adjust my evaluation based on the author's rebuttal.

Reviewer_e6A5OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether diffusion models' ability to follow multi-object prompts depends on caption coverage, defined as the proportion of multi-object training images whose captions mention at least two objects. By varying captions while keeping the training images fixed, the authors distinguish caption coverage from image coverage. Experiments on ShapeWorld, CLEVR, and MS-COCO, using U-Net/DiT backbones with either object-token or text conditioning, show that higher caption coverage improves multi-object prompt accuracy. The paper further analyzes object omission, continued training with rewritten captions, attention routing, and the effects of denoiser locality.

Critical strengths

1.The paper distinguishes between which objects are present in the training images and which of these objects are jointly mentioned in their captions. Caption coverage is easy to measure and manipulate, providing a valuable perspective for studying compositional generation.

2.Hypotheses H1-H3 and the control experiment clearly connect the research claims to observable outcomes. Keeping the training images fixed while varying only the caption construction helps reduce important confounding factors. Evaluations across multiple coverage levels, random seeds, datasets, and conditioning formats further strengthen the empirical evidence.

3.Decomposing prompt accuracy into occupancy and attribute match reveals that low caption coverage primarily leads to object omission. The experiments on summing single-object scores, measuring own share, and ablating gated attention provide complementary evidence about how multi-object conditions are processed.

Critical weaknesses

1.A substantial performance gap remains between natural-language captions and structured object tokens. Although the authors examine the effect of freezing the text encoder, the underlying causes of this gap remain insufficiently explored.

2.The experiments rely primarily on synthetic datasets, while real-image evaluation is limited to object presence using Stable Diffusion v1.5. Validation on more recent architectures and finer-grained compositional tasks, such as attribute binding and spatial relations, remains limited.

Questions for authors

1.Could the authors further control the total number of object mentions, input token length, and object co-occurrence patterns across different caption coverage levels? This would help clarify whether the improvements stem from joint-object supervision or simply from increased conditional information.

2.Could the authors compare the predictive power of caption coverage with finer-grained metrics, such as object mention rate and pairwise co-occurrence coverage, for multi-object generation accuracy?

3.Could the increased uncertainty in the conditional image distribution caused by partial captions partially explain the observed failures? Is it possible to independently manipulate conditional ambiguity and the joint occurrence of multiple objects in captions?

Reviewer_yLH4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how training captions affect multi-object generation in text-to-image diffusion models. It defines caption coverage as the fraction of multi-object training images whose caption names at least two objects. The authors keep the training images fixed and vary only the captions, on ShapeWorld, CLEVR and MS-COCO. Models trained with high coverage follow multi-object prompts much better. Low-coverage models mostly leave prompted objects out. The set of color and shape pairs in the training images mainly affects generation of unseen pairs.

Critical strengths
  1. Well-motivated hypothesis and experimental design. The paper is motivated by the gap between what an image contains and what its caption describes. From this it builds a testable hypothesis: multi-object generation depends on how many objects the training captions name. ShapeWorld is a well-designed testbed for this question. The analysis is extended to real dataset MS-COCO as well.
  2. Extensive analysis. The paper examines how low-coverage models fail and tests several alternative explanations, including the training images, attribute frequency, dataset size and the attention mechanism.
Critical weaknesses
  1. Evaluation fairness. Multi-object prompts are out of distribution in format for low-coverage models but not for high-coverage models, which puts low-coverage models at a disadvantage when evaluated on multi-object prompts. Table 2a supports this concern. When single-object predictions are summed (softmax, summed), low-coverage model receiving a one-object prompt as in training, the gain drops significantly across caption coverage. This suggests that low-coverage models already know each object and fail mainly on the unfamiliar multi-object prompt format.

  2. Limited evaluation.

    • Tab. 2 only reports three-object prompts. For fair comparison, evaluate models a single objects, or more than three object prompt for fairness. It would strengthen the paper to show high coverage model dominates in varying number of object in general.
    • The paper only reports Prompt Accuracy for all analysis. Standard image-quality and alignment metrics such as FID and CLIP score, especially on MS-COCO, would show whether caption coverage affects generation quality beyond object placement.
Questions for authors
  1. Figure 3. Figure 3b-e are schematic illustrations of what each hypothesis expects. Because they are drawn like data plots, readers may mistake them for actual results, and the caption of panel a likewise describes the planned caption-coverage schedule as if it were measured. I suggest the paper report only measured results in its figures and state the predictions in the text.
  2. Coverage versus number of tokens. Caption coverage is entangled with the number of tokens in each training condition, as most low-coverage conditions contain only one object token. Could the authors clarify whether the improvement comes from naming multiple objects together or from exposure to multi-token conditions? For example, would a low-coverage model still fail on multi-object prompts if trained with partial captions padded to match the token count of full captions using dummy or unspecified-object tokens?
  3. Choice of caption form per dataset. Could the authors explain the reason to use object token only for CLEVR and text captions for MS-COCO?

Minor comments

  1. Placement of key results. Several results central to the claims are reported only in the appendix, including results on CLEVR and MS-COCO. I encourage authors to move these result to main text.
  2. Related work. The authors might find it useful to discuss [1] in related works. Its image coverage rate, the proportion of objects in an image that the caption describes, is closely related to caption coverage. Its analysis of how this rate affects text-to-image training is also closely related to the paper’s experiments.
  3. Presentation. The paper is difficult to follow as results for one claim are spread across the sections and the appendix. For example, Sec. 5.1, on H1, cites Fig. 6e and Fig. 7a, which are placed with the H3 and locality results, and Sec. 5.3 relies on Fig. 14 in the appendix. Several results, such as Fig. 7b-c, or Fig. 8 would be clearer as tables with exact numbers as they contain a few values or uses separate conditions rather than variable as axis label.

[1] "Harnessing Caption Detailness for Data-Efficient Text-to-Image Generation", Wang et al., ArXiv 2025

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.