acceptodds
Withdrawn from ICLR 2027

Compositional Collapse in Diffusion Models: Scaling Laws of Local Fidelity

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

That diffusion models struggle with crowded scenes is known; how they fail, including the functional form of the failure, its onset, and the extent to which it is a measurement artifact, has never been systematically measured. We introduce a controlled framework that varies the requested entity count from 1 to 64 while keeping scene, lighting, and activity fixed, measuring local fidelity (entity recall, facial validity, identity duplication) alongside global quality (CLIPScore and per-density-bin KID). Three results emerge. (1) Collapse is exponential, not gradual: across three open models (SDXL, SD1.5, and PixArt-α), detector-corrected recall decays exponentially with entity count and is preferred over power-law fits by AIC in all models, with decay constants ranging from 0.019 to 0.034. This yields a characteristic capacity scale, corresponding to a recall half-life of 20 to 36 entities, rather than scale-free degradation. The same collapse pattern reproduces for cars instead of people, with rate constants unchanged in two of the three models, suggesting a largely class-independent bottleneck. (2) Naive measurement overstates collapse by approximately 40%: even a strong face detector misses more than one-third of ground-truth faces in dense real scenes, with recall dropping from 0.94 to 0.59 on WIDER FACE. We account for this confound through calibration on real-image controls. (3) Identity duplication has a measurable onset: it is rare below roughly 8 entities (0 to 11% of images) but appears in 75 to 84% of images at 64 entities using an ArcFace cosine similarity threshold greater than 0.4. Segmented regression estimates the onset between approximately 10 and 22 entities. Meanwhile, global metrics are either insensitive to or negatively correlated with collapse: CLIPScore remains flat or increases, while KID against real crowds improves. These findings reveal a decoupling between global realism and local structural correctness that remains invisible to standard evaluation metrics.

Primary area:
generative models
Keywords:
Diffusion Models, Compositional Collapse, Local Fidelity, Scaling Laws, Text-to-Image Generation, Compositionality, Multi-Object Generation, Attribute Binding, Visual Reasoning, Generative AI, Image Synthesis, Model Evaluation, Robustness, Foundation Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ogx6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how text-to-image generation degrades as the number of entities requested increases from 1 to 64. Across different models, it evaluates entity recall, face validity, identity duplication, CLIPScore, and density-matched KID, while additionally calibrating face-detector degradation on real images. The main findings are an approximately exponential decay in detector-corrected recall, increasing identity duplication at high density, and a mismatch between local structural fidelity and global image-quality metrics.

Critical strengths
  1. The paper identifies that in crowded scenes, face detectors themselves degrade and they explicitly measure this effect on real images. This is an important evaluation confound that is often overlooked.

  2. Studying entity counts up to 64 reveals failure modes that are not captured by standard compositional benchmarks or global metrics. The people/car comparison and local-vs-global evaluation are particularly useful.

Critical weaknesses
  1. The main prompt requires the model to satisfy “exactly N people ... no other people in the scene.” However, even at N=1, the models generate 1.4–3.2 times the requested number of people. This is important because a model simply can have poor numerical understanding and tends to generate something like “a few people,” “a group,” or “a crowd,” rather than faithfully representing 4, 16, 32, or 64. In that case, N^/N would decrease strongly with N even without demonstrating a separate visual-capacity collapse. Prior work such as T2ICountBench [1] and LLM-Grounded Diffusion [2] already shows substantial difficulty with numerical constraints and negation.

  2. The detector correction is calibrated on WIDER FACE using the actual number of faces present, whereas generated samples are grouped by the requested count N. These quantities may differ substantially. For example, an N=64 prompt that actually renders 20 people may not be as difficult for the detector as a real image containing 64 faces. Applying the 64-face correction could therefore over-correct recall at high N. An equivalent density-dependent calibration is also not provided for the YOLO person detector.

  3. The identity measure asks whether an image contains at least one highly similar face pair. However, the number of possible pairs grows rapidly with the number of faces: 4 faces give 6 pairs, while 20 faces give 190. Thus, even if the similarity distribution of individual pairs stayed unchanged, the probability of finding at least one high-similarity pair would naturally increase with density. A related effect is acknowledged in the paper's birthday-collision analysis, but a fixed-comparison analysis is needed to show that identities themselves become less diverse. Recent work such as DisCo [3] also directly studies duplicate identities in multi-human T2I generation.

  4. The empirical finding that an exponential fits the observed recall better than a power law is interesting. However, the paper further interprets this as evidence for an intrinsic finite entity budget and a constant per-entity survival hazard. The curve fit alone does not uniquely support these mechanisms: count saturation, worsening prompt adherence, spatial-resolution limitations, or other capacity effects could produce similar behavior. I think the paper should distinguish the empirical scaling result from the proposed mechanistic explanation more clearly.

[1] Cao et al., “Text-to-Image Diffusion Models Cannot Count, and Prompt Refinement Cannot Help,” 2025.

[2] Lian et al., “LLM-Grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models,” TMLR 2024.

[3] Borse et al., “Resolving the Identity Crisis in Text-to-Image Generation,” CVPR 2026.

Questions for authors

Q1: Does the same scaling behavior remain if the prompt is simplified, e.g. comparing “N people,” “exactly N people,” and “exactly N people, no other people”? This would help determine whether the observed decay reflects density itself or difficulty following numerical/negative instructions. It is also well known that these models behave badly with negation words like "no", "not". How would you handle this?

Q2: How is the correction factor chosen when the requested and actually rendered numbers of faces differ?

Q3: Does the identity effect remain if the same number of faces or face pairs is sampled from every image?

Q4: Could the authors justify why the exponential specifically indicates a finite entity budget, or compare against other plausible saturation/capacity models?

Reviewer_rSbBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how text-to-image diffusion models degrade when the number of requested people in a scene increases. A prompt template varies only the count N (1 to 64), with scene, lighting and activity as controlled factors, and three models are evaluated (SD1.5, SDXL, PixArt-alpha). Local fidelity is measured with person/face recall, face validity and identity duplication, and compared to global metrics (CLIPScore, per-bin KID). Detector-based metrics are corrected using real crowd images from WIDER FACE. Main claims are that recall decays exponentially with N (half-life of 20-36 entities), that duplicated identities appear after a model-dependent onset, and that global metrics stay flat or even improve while local fidelity collapses. A similar experiment with cars is added to show the effect is not specific to faces.

Critical strengths
  • Problem studied is interesting and methodology is adapted to study this problem.
  • Calibration with real data makes sense and is interesting
Critical weaknesses

W1: I find the abstract and introduction difficult to read in the current state. The intro is stating a lot of things without any citation to support it. Most sentences are very long —with nested sentences — which make it particularly difficult to read.

W2: Figure 1 is not convincing. From a first view, the face structure seem much better preserved in N=64 and it is in N=4 example shown, which does not support the main claim of the paper

W3: The models used for the analysis are “old”, and the diversity of models evaluated is low. All of them were introduced in 2022 (SD1.5) or 2023 (SDXL and PixArt-alpha), which make the current analysis outdated.

W4: I believe limitation does not state the most explicit bias: size of generation (resolution) and the only metric taking that into account is face validity. Indeed, as dentsity augment, face area tend to reduce (quadratically from intuition; this is completely ignored and could heavily explain the reduction of heterogeneity)

W5: There should be a dedicated experiment adressing this bias explicitly, for N=1, is the area of the face is generated with reduced size, i would expect the same results as reported using “density”.

W6: It seems a major criteria that define the trends is generation resolution. This is also highlighted by the results that SDXL shows quite different trends than the 2 others.

W7: Limited related work. Why is related work stopping in 2024 ? There are many more recent diffusion models, and studying, or at least mentioning related work would be strictly necessary. Notably FLUX.1/FLUX.2-klein, SD3.5, HunyuanImage-2.1, Qwen-Image, etc...

W8: Figure 2: Fitted scaling laws do not seem to follow the data, why ?

W9: “Figure 5: Global metrics stay comparatively stable while local metrics collapse”. It does not seem to be the case for Face Validity metrics, however the paper conveniently ignores to comment on that, why?

W10: Minor: references and citations are not linkable in the pdf.

Questions for authors

Why were all images with wrong number of persons not filtered out?

Reviewer_xbGSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how compositional collapse occurs in diffusion models as the number of objects to generate increases. The authors conduct controlled experiments using SDXL, SD1.5, and PixArt-α on human and car scenes. They show that entity recall decreases exponentially rather than following a power law as the number of objects increases. They also analyze face validity and identity duplication, showing that local fidelity degrades even when global image quality metrics remain stable.

Critical strengths
  1. It is interesting to see a controlled study on how local generation quality changes as the number of objects increases.
  2. It is interesting that entity recall follows an exponential decay rather than a power law, and that identity duplication becomes more common beyond a certain number of objects.
Critical weaknesses
  1. The significance of the findings seems somewhat limited. The experiments are conducted on relatively outdated models, including SD1.5, SDXL, and PixArt-α. It is unclear whether it generalizes to more recent diffusion models, which showed superior performance on counting.
  2. The evaluation is limited to human and car scenes, which makes it difficult to understand how general the findings are. In particular, the paper mainly focuses on entity recall, face validity, and identity duplication. However, compositionality involves other aspects, such as attribute binding, spatial relationships, and heterogeneous objects. It is unclear whether the observed exponential collapse is specific to counting and dense scenes or reflects a more general limitation of compositional generation.
Questions for authors
  1. It provides limited insights into how to address the problem. Although Section 5.3 provides some evidence that the observed collapse is architecture-invariant, the proposed explanations seem largely observational. For example, it is unclear why uniform cross-attention over a collective noun should indicate a failure of instance allocation or support the latent capacity hypothesis. What are the authors' thoughts on this?
  2. Compared with previous work [1, 2], the paper offers limited additional insight or controlled study into why compositional collapse occurs. For example, [1] investigates the local mechanisms required for compositional generalization for generating new scenes with more objects, and [2] studies how counting performance degrades as the number of objects increases. What kind of things can the paper provide that these works do not?

[1] Local Mechanisms of Compositional Generalization in Conditional Diffusion (ICML 2026) [2] When Do Diffusion Models Learn to Generate Multiple Objects? (ICML 2026)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.