acceptodds
Under review as a conference paper at ICLR 2027

Parachuting parrot: A dataset for evaluating visual compositional generalization

Abstract

Proprietary text-to-image models have become remarkably powerful at rendering complex compositional scenes. Did they overcome the longstanding challenge of compositional generalization, and if so, how did they do it? Current datasets that allow to systematically evaluate compositional generalization lack the complexity, diversity and scale required to train image generation models of remotely similar quality to answer this question. Here, we introduce Parachuting Parrot, a vision dataset and benchmark for evaluating compositional generalization consisting of over 2.5 million images with corresponding structured scene descriptions. Parachuting Parrot consists of complex scenes defined by a rich scene description language, rendered using large-scale text-to-image models and validated through a scalable vision-language model pipeline. Alongside it, we define systematic splits to evaluate compositional generalization over scenes with varying compositional complexity. We train a text-to-image latent diffusion model from scratch on Parachuting Parrot, achieving generalization to complex scene compositions with multiple entities, entity-attribute binding and relational binding in a fully controlled experimental setting.

Primary area:
datasets and benchmarks
Keywords:
visual compositional generalization, text-to-image generation, latent diffusion, dataset, benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QoZ7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a large-scale synthetic dataset for evaluating visual compositional generalization. It contains about 2.5 million samples with structured scene descriptions, generated using a text-to-image model and validated by a VLM. The authors design systematic training and evaluation splits to test generalization to unseen concept combinations. By training a latent diffusion model from scratch, they demonstrate that diffusion models can generalize to novel compositions, while showing that this ability depends on sufficient compositional exposure and training coverage.

Critical strengths
  1. This paper propose addresses a central question: compositional generalization in a high-complexity, large-scale setting that existing synthetic datasets do not reach.
  2. This paper perform the split based on the bindings between entities and their attributes, as well as those between entities and their relational roles. This allows for a formal definition of whether a specific combination has appeared during training.
  3. Experiments for this method fully account for the impact of different VAEs on the results.
Critical weaknesses
  1. The paper uses Gemini for both dataset filtering and final evaluation, without independent validation using other VLMs. The authors' manual inspection found 13 incorrectly accepted images out of 430 samples, involving key issues such as entity relations and attribute binding. These errors directly affect the evaluation of compositional generalization.
  2. All images in the dataset are generated using FLUX.2-dev, which does not guarantee sufficient visual diversity. The dataset may inherit FLUX-specific biases in object appearance, composition, and the representation of relationships between entities.
  3. a small issue is that the paper further analyzes the impact of compositional exposure and scene complexity on generalization ability. Since the authors only conducted experiments on a single-size DIT, it is currently unclear to what extent these observed generalization limitations are affected by model capacity.
Questions for authors
  1. How did the authors assess the potential visual distribution bias that might be introduced by relying solely on FLUX.2-dev? Is it possible to conduct control experiments with alternative generators while maintaining comparability in scene grammar to verify whether the main conclusions about combinatorial generalization remain robust across different data sources?
  2. Could the authors use VLMs from different model families to independently review the student model generation results covering different complexities and seen/unseen combination?
2 comments

We thank the reviewer for the constructive feedback and the suggested experiments. We address each point below.

Weakness: The paper uses Gemini for both dataset filtering and final evaluation, without independent validation using other VLMs. The authors' manual inspection found 13 incorrectly accepted images out of 430 samples, involving key issues such as entity relations and attribute binding. These errors directly affect the evaluation of compositional generalization.

Question: Could the authors use VLMs from different model families to independently review the student model generation results covering different complexities and seen/unseen combination?

Thank you for this suggestion, we additionally evaluated the student model with two further judges, a stronger Gemini model (Gemini 4 Argon) and a model from a different provider (Claude Opus 5.5). The results stay qualitatively the same, with both overall accepting more images due to their ability to identify less visually sharp concepts and the gap between training and held-out scenes remains across judges.

This implies that despite some training images containing partial attribute bindings or incorrectly labelled relations, the student learns to correctly compose scenes from the vast majority of correct training images and it also confirms that our evaluation numbers are not artificially inflated as a result of Gemini 3.7 Flash consistently accepting such faulty compositions.

Note that the Gemini 3.7 Flash numbers differ slightly from the paper, as we added rare concepts seen in only a single training scene, a new experiment described in our general response.

BlockFlash 3.7 trainFlash 3.7 held-outArgon trainArgon held-outOpus 5.5 trainOpus 5.5 held-out
A0.920.920.920.920.940.94
B0.870.810.900.850.940.91
C0.810.780.810.780.910.89
D0.630.600.650.660.660.65
E0.780.750.810.780.890.83
F-2-few0.660.520.630.520.780.72
F-2-all0.680.660.710.640.770.73
F-3-simple0.390.410.470.470.530.54
F-3-complex0.170.160.260.240.320.31
All blocks0.650.610.670.640.740.72

Weakness: How did the authors assess the potential visual distribution bias that might be introduced by relying solely on FLUX.2-dev? Is it possible to conduct control experiments with alternative generators while maintaining comparability in scene grammar to verify whether the main conclusions about combinatorial generalization remain robust across different data sources?

Question: All images in the dataset are generated using FLUX.2-dev, which does not guarantee sufficient visual diversity. The dataset may inherit FLUX-specific biases in object appearance, composition, and the representation of relationships between entities.

While it is true that our dataset inherits aesthetic biases from FLUX.2-dev, we would like to emphasize that the scene structure and its constituents (concepts, bindings) are determined by our scene description language and image generation pipeline which validates that scene structure is generated correctly using an independent VLM judge.

Thus, while FLUX.2-dev ultimately determines the low-level visual details of each image, high-level concept compositions and relations do not depend on FLUX.2-dev.

As a result, our training images are visually diverse (e.g. see Fig. 1 and the sample images provided in the supplementary material for qualitative evidence) covering the diverse concepts and scene structures defined by our scene description language.

In general, our dataset generation pipeline can be used with different generators but regenerating the millions of images contained in our dataset would incur significant cost (on the order of thousands of B200 hours and thousands of dollars in VLM API costs).


A small issue is that the paper further analyzes the impact of compositional exposure and scene complexity on generalization ability. Since the authors only conducted experiments on a single-size DIT, it is currently unclear to what extent these observed generalization limitations are affected by model capacity.

We agree with the reviewer that the effect of the DiT's capacity on generalization is an interesting question our dataset allows to investigate. We would first like to note that our primary contribution is the dataset and benchmark. Thus, the aim of our experiments was not to provide an exhaustive empirical evaluation of the benchmark, but to show, as a proof of concept, that a model can be trained and its generalization analyzed.

To give some early results in this direction, however, we did run experiments with two smaller DiTs, scaling the depth and width of the diffusion model and text encoder together (u = 8 is the size used in the main text). With increasing model size performance increases on both training and held-out scenes, while the gap between them stays similar.

ModelParametersIn-distributionHeld-outGap
u = 484M0.5370.5070.030
u = 6281M0.6130.5760.037
u = 8663M0.6480.6120.036
Reviewer_d2JFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work presents an isolated study of compositional generation in text-to-image models. The authors first systematically construct a compositional dataset in a controlled setting, where concepts are well-separated between the training and evaluation sets to ensure proper evaluation of generalization. They then train T2I models from scratch on the proposed dataset to have models with controlled knowledge for fair evaluation. They study the generalization behavior of these models on unseen concepts.

Critical strengths
  • Interesting work that specifically focuses on the issue of compositionality in isolation by designing datasets and training models from scratch to investigate their generalization ability. This is much more justified compared to methods that fine-tune pretrained models, where the knowledge already learned by the models is not controlled.

  • Modeling compositional text generation using a CFG (Context-Free Grammar) is a nice and more controlled approach compared to simply using an LLM with instructions to expand prompts. This also makes the VLM-based evaluation more structured and reliable.

  • Having a structured way to ensure the separation of training and test compositional sets (Concept Binding Graph) is well-defined and properly implemented.

Critical weaknesses
  • This is a well-studied area of text-to-image generation, and the overall research direction is not particularly novel. Moreover, numerous compositional datasets have already been proposed and extensively studied in prior work. That said, having such an isolated study is still valuable and well worth investigating.

  • The paper is missing related works that also incorporate automated compositional dataset generation, such as ConPair [1] or AgentComp [2]. In general, I think having a short related work section covering different approaches to addressing compositionality would be worthwhile (training-based [3], planning-based [4], agentic-based [5], heuristic attention-based binding [6], etc.).

  • It would be nice to have a conclusion section.

  • I am a little skeptical about potential train-test leakage based on Fig. 6. I've provided more details in the questions section.


[1] Progressive compositionality in text-to-image generative models.

[2] AgentComp: From Agentic Reasoning to Compositional Mastery in Text-to-Image Models

[3] T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation

[4] Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs

[5] MCCD: Multi-Agent Collaboration-based Compositional Diffusion for Complex Text-to-Image Generation

[6] Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models

Questions for authors
  • Could you explain and justify the improvements in compositionality when replacing your trained VAE with the SD VAE?

  • In Fig. 6, the top-left image from the held-out set, I assume it corresponds to a prompt containing "rabbit surfing" (based on Table 3 in the Appendix). However, I also see the same concept in the training data image you provided (which also shows a rabbit surfing). Based on your Concept Binding Graph and the way you defined the training and test splits, this shouldn't be the case, right? Could you clarify this? It would also be very helpful to include the corresponding captions for both the test and training images in Fig. 6.

Minor suggestion:

  • The claim "We see similar performance between our models trained on Parachuting parrot and the teacher across the different evaluation blocks" is not quite accurate. There is a clear performance gap between the models, which makes this phrasing somewhat misleading. That said, reaching the teacher's performance is not necessarily expected given the controlled setting in which your models were trained, and this gap is completely understandable.
  • I would suggest framing the paper as a "controllable study of the generalizability of compositionality in T2Is" as opposed to simply "providing a compositional dataset", as this has been done extensively previously.
3 comments

We thank the reviewer for the positive assessment and the helpful suggestions on framing and related work. We address each point below.

Weakness: This is a well-studied area of text-to-image generation, and the overall research direction is not particularly novel. Moreover, numerous compositional datasets have already been proposed and extensively studied in prior work. That said, having such an isolated study is still valuable and well worth investigating.

We respectfully disagree with the reviewer regarding the novelty of our work. While it is true that there are prior compositional datasets designed to measure whether a trained model can correctly generate specific compositions (see L447–454 for a discussion of such works), we are specifically interested in compositional generalization, which requires training models from scratch and measuring whether they generalize to OOD combinations of ID concepts by using explicit ID/OOD splits. The novelty of our work is to provide a large-scale dataset that offers explicit controllability to construct such ID/OOD splits while being sufficiently complex to support training large-scale models in which complex compositional abilities emerge. Existing benchmarks that offer such controlled splits, such as dSprites or PUG [7][8], are built from a small set of rendered assets (2D shapes, 3D meshes) and vary only in a few factors such as position, size or background. To our knowledge, no prior compositional generalization benchmark offers the complexity and scale of our dataset while remaining fully controllable.

  • [7] dSprites - Disentanglement testing Sprites dataset
  • [8] PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learning

Weakness: The paper is missing related works that also incorporate automated compositional dataset generation, such as ConPair [1] or AgentComp [2]. In general, I think having a short related work section covering different approaches to addressing compositionality would be worthwhile (training-based [3], planning-based [4], agentic-based [5], heuristic attention-based binding [6], etc.).

  • [1] Progressive compositionality in text-to-image generative models.
  • [2] AgentComp: From Agentic Reasoning to Compositional Mastery in Text-to-Image Models
  • [3] T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation
  • [4] Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs
  • [5] MCCD: Multi-Agent Collaboration-based Compositional Diffusion for Complex Text-to-Image Generation
  • [6] Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models

Minor suggestion: I would suggest framing the paper as a "controllable study of the generalizability of compositionality in T2Is" as opposed to simply "providing a compositional dataset", as this has been done extensively previously.

We appreciate the references on automated compositional dataset generation and will discuss them in the updated related work section. We would like to emphasize that all these prior works adapt a pretrained text-to-image model: ConPair [1] and AgentComp [2] generate data to fine-tune it, T2I-CompBench [3] benchmarks it, and [4–6] improve its compositions at inference time through planning, agents or attention guidance. In contrast, our dataset allows us to train a model from scratch with controlled compositional splits. In this regard, we fully agree with your suggested framing and will revise the manuscript to make this point clearer.


Weakness: It would be nice to have a conclusion section.

We thank the reviewer for the suggestion. We will add a conclusion section to the camera-ready version of the manuscript.


Weakness: I am a little skeptical about potential train-test leakage based on Fig. 6. I've provided more details in the questions section.

Question: In Fig. 6, the top-left image from the held-out set, I assume it corresponds to a prompt containing "rabbit surfing" (based on Table 3 in the Appendix). However, I also see the same concept in the training data image you provided (which also shows a rabbit surfing). Based on your Concept Binding Graph and the way you defined the training and test splits, this shouldn't be the case, right? Could you clarify this? It would also be very helpful to include the corresponding captions for both the test and training images in Fig. 6.

Thank you for making us aware of this possible confusion due to the missing scene description. We have added the prompts to Fig. 6. In this example, the held-out binding is rabbit-cardboard, not rabbit-surfing. So, the model has seen both a cat made of cardboard and a surfing rabbit, but never encountered a rabbit made of cardboard. It now generates a surfing rabbit made of cardboard, where the cardboard-rabbit binding is novel.

Question: Could you explain and justify the improvements in compositionality when replacing your trained VAE with the SD VAE?

The SD-VAE mainly improves image quality rather than compositional generalization: the model improves by a similar amount on training scenes (0.67 to 0.74) and on held-out scenes (0.63 to 0.71), so the gain on held-out scenes follows the better performance on training scenes. This can be explained by the SD-VAE producing sharper images, which helps on finely detailed concepts. We will clarify this point in the manuscript.

We further investigate this point in an additional experiment in our response to Reviewer DYLS where we train linear probes to verify whether either VAE provides a more or less structured latent space finding no clear difference between the two.


Minor suggestion: The claim "We see similar performance between our models trained on Parachuting parrot and the teacher across the different evaluation blocks" is not quite accurate. There is a clear performance gap between the models, which makes this phrasing somewhat misleading. That said, reaching the teacher's performance is not necessarily expected given the controlled setting in which your models were trained, and this gap is completely understandable.

We thank the reviewer for pointing this out; we agree that the phrasing is not accurate and we will revise it in the paper. As you correctly point out, the gap to the teacher is expected. The statement was meant to point out that the student models follow a similar trend to the teacher as scene complexity grows.

I thank the authors for providing further explanations and clarifications. Overall, I found the paper interesting and liked the analysis of training T2Is from scratch using controlled datasets and evaluation to evaluate the compositional generalizability of these models. Therefore, I maintain my score of weak acceptance.

Reviewer_DYLSOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors present a new dataset, Parachuting Parrot, comprising 2.55 million synthetically generated images. The main motivation of the dataset is for evaluation of generative models for visual compositional generation mainly for complex scenes. The dataset generation pieline consists of first creating structed and neutral language scene descriptions for a set and construct a scene grammer or tree. This grammer is used to convert the scene tree into captions which are then passed to FLUX.2-dev model for generating the complex images. Lastly a VLM based verifier is employed which aims to rescontruct the original grammer scene/tree just given the image. The authors train a diffusion model from scratch on a partial dataset and show generalization to the held-out combinations.

Critical strengths
  • The dataset presents an interesting motivation for evaluating the novel combinations of concepts that did not occur during training.
  • The dataset scenes include multiple entities, attributes, styles, backgrounds, and relational chains, and the way the dataset is constructed is exhaustive. This provides a useful range of compositional tasks within one dataset.
  • Training all the components of a diffusion model - VAE, text encoder, and the transformer layers makes the experiments in a controlled environment.
Critical weaknesses
  • The motivation and the premise of the paper are not clear and conflict with the paper in its current form. The title and abstract motivate the dataset as an evaluation dataset for compositional generalization of proprietary text-to-image models, whereas no proprietary or open-source models are shown as benchmarks for the dataset, which is the main weakness of the paper.

  • L133 mentions that the grammar excludes hard-to-conceive combinations like “parachuting pirate apples”. However, is it not exact aim to evaluate and train and see to see if the models are able to learn these complex visual compositions? When does a composition become hard to conceive, and when does it not? The modern models can generate "parachuting pirate apples".

  • The Parachuting Parrot example at L126 has a grammar scene where the image style is graffiti. In the generated image the parrot appears painted on the wall with a parachute, but the bicycle in a white marble texture appears to be an object and not a graffiti painted on the wall which is assigned to the overall scene. This is important, as even this example should have been flagged, or are partially applied attributes also accepted images?

  • The VLM-based verification and filtering stage retains images only when the judge reconstructs the original grammar scene correctly. This could potentially favor particular visual realizations, such as pose or certain scene layouts which the VLM is good at creating the grammar and discarding scenes where the VLM wasn't that good. The reported acceptance rates should also be described for different blocks and also show that the filtering does not substantially change the diversity for any block.

  • Table 1 headers are hard to read. Multiple by itself does not specify what is multiple. Is it multiple concepts, multiple objects, etc.?

  • In L275-278, how can we ensure that excluding a binding from a scene description guarantees that it is absent from the training images? For example, if a caption is "a checkered lizard and a parachuting parrot" and if the teacher (FLUX2) model accidentally made both the lizard and parrot checkered and the judge overlooked it. This becomes a leak image and is not a true held-out concept of "a checkerd parrot".

  • L299 mentions compositional exposure. How can we isolate the held-out bindings fully? Many would have close visual or semantically similar counterparts in training. For example, the model is asked to generate a held-out concept of "a parachuting parrot" but the model may have seen "parachuting eagles" or other birds which are similar. Are these still generalization to unseen bindings? The difficulty of generalization may be strongly dependent on semantically similar concepts like birds in this case are also held out.

  • The held-out evaluation should also include certain concepts that were never introduced in training, for example "made in" like an object made in a certain style. This would help in seeing the generalization capabilities of the models.

  • Evaluation of trained student models in section 3.1 also utilizes the same VLM pipeline used for verification. Human evaluation here as well would help make the paper stronger.

  • Figure 5 right shows the trained student model shows better results with a new VAE from Stable Diffusion. This does not help isolate if the gains are coming just from a VAE that is more heavily pre-trained of web-scale data or the fact that the VAE also has the ability to capture the complex cues in its latents.

  • L377 and Figure 7 mention that "generalization to evaluation scenes noticeably degrades for concepts with a lower compositional exposure". Figure 7 shows a dip in the pass rate for the evaluation scene of "cotton candy," which has a compositional exposure of 3. But two more scenes, "ice" and "honeycomb," with compositional exposures of 1 and 2, respectively, have significantly higher pass rates. How can we conclude that low exposure to scenes is the reason or is it that some scenes are inherently harder to generate?

  • Figure 7 right shows the pass rate wrt to the th number of entities being generated. While the model is trained on 3 entities, the performance dips significantly if more than 3 entities are present during inference. This is a fair result, but is the dip happening because the 4th entity is generated incorrectly, or the model fails to generate the first 3 entities too? It would be interesting to see the entity-level pass rate here.

  • In Table 5, the model performs similarly on the in-distribution (ID) and held-out scenes, and both scores are low for the hardest compositions. For example, F-3-rel complex has pass rates of 12% and 10% respectively, highlighting limited success on those scenes.

Questions for authors

Please see the weakness section.

6 comments

We thank the reviewer for the detailed review. We address each point below and hope this clarifies the concerns raised.

The motivation and the premise of the paper are not clear and conflict with the paper in its current form. The title and abstract motivate the dataset as an evaluation dataset for compositional generalization of proprietary text-to-image models, whereas no proprietary or open-source models are shown as benchmarks for the dataset, which is the main weakness of the paper.

Thank you for raising this possible point of confusion: The creation of our dataset and benchmark is motivated by the impressive compositional abilities of proprietary text-to-image generation models but by definition we do not have access to these models or their training data and cannot study them. We therefore create a large-scale dataset with controlled compositional splits that allows us to train text-to-image-generation models from scratch in which the ability for flexible scene generation emerges. We will revise the introduction to make this point clearer.


L133 mentions that the grammar excludes hard-to-conceive combinations like “parachuting pirate apples”. However, is it not exact aim to evaluate and train and see to see if the models are able to learn these complex visual compositions? When does a composition become hard to conceive, and when does it not? The modern models can generate "parachuting pirate apples".

We apologize for this misunderstanding and will clarify this in the manuscript. We do not exclude such combinations because they are particularly complex visual compositions, in fact our dataset contains much more complex scenes. In line with prior work [1,2,3], we exclude such combinations to avoid anthropomorphisations, whose outputs are often ambiguous or not well defined.


The Parachuting Parrot example at L126 has a grammar scene where the image style is graffiti. In the generated image the parrot appears painted on the wall with a parachute, but the bicycle in a white marble texture appears to be an object and not a graffiti painted on the wall which is assigned to the overall scene. This is important, as even this example should have been flagged, or are partially applied attributes also accepted images?

In rare cases, Flux.2-dev only partially applied the style to an image and Gemini 3.7 Flash accepted them but this does not broadly affect the dataset. We report this in Section 2.2 (L189-194) where human annotation found that some images had partially applied styles. Importantly, these images do not leak held-out compositions and therefore do not affect the evaluation of compositional generalization. Since this only affects a small portion of the dataset, our model trained from scratch on the dataset is able to apply the graffiti style to the whole image as can be seen for example in Figure 6 (“One frog among torii gates, in graffiti style”).


The VLM-based verification and filtering stage retains images only when the judge reconstructs the original grammar scene correctly. This could potentially favor particular visual realizations, such as pose or certain scene layouts which the VLM is good at creating the grammar and discarding scenes where the VLM wasn't that good. The reported acceptance rates should also be described for different blocks and also show that the filtering does not substantially change the diversity for any block.

We report average acceptance rates for the first attempt per scene for the different blocks in Table 2. When an image is rejected, we resample a new image for the same scene. With at least 45 attempts per scene, fewer than 0.1% of the images in the enumerated blocks are missing. In the sub-sampled blocks, complete enumeration is not possible, but every concept is still seen hundreds of times. This results in scene layouts that are significantly more varied than comparably controlled datasets that measure compositional generalization [4,5].

  • [4] dSprites - Disentanglement testing Sprites dataset
  • [5] PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learning

Table 1 headers are hard to read. Multiple by itself does not specify what is multiple. Is it multiple concepts, multiple objects, etc.?

Thank you for pointing this out. “Multiple” refers to multiple entities; we will clarify this in the header of Table 1.

In L275-278, how can we ensure that excluding a binding from a scene description guarantees that it is absent from the training images? For example, if a caption is "a checkered lizard and a parachuting parrot" and if the teacher (FLUX2) model accidentally made both the lizard and parrot checkered and the judge overlooked it. This becomes a leak image and is not a true held-out concept of "a checkerd parrot".

We construct the image generation pipeline to make it very unlikely that such a leak image is added to the dataset: both the generator and the judge would have to make this mistake independently. The judge itself does not see the ground-truth scene description but has to describe the image itself using our grammar, and the image is only accepted if this description matches the requested scene. In Section 2.2 we report that human evaluation found only 13 of 430 scenes were accepted despite ambiguous concept bindings where for example attributes were only partially bound to an entity.


L299 mentions compositional exposure. How can we isolate the held-out bindings fully? Many would have close visual or semantically similar counterparts in training. For example, the model is asked to generate a held-out concept of "a parachuting parrot" but the model may have seen "parachuting eagles" or other birds which are similar. Are these still generalization to unseen bindings? The difficulty of generalization may be strongly dependent on semantically similar concepts like birds in this case are also held out.

Yes, these are generalizations to held-out bindings: the concept binding itself never appears in training. Whether semantically close concept bindings help is a good question. Figure 6 shows the closest training neighbours measured by their distance in SigLIP 2 embedding space.

To isolate the held-out bindings further, we added an evaluation split with rare concepts, each seen in exactly one training scene. A rare concept has exactly one binding in training, e.g. popcorn is only ever seen on a chair, and this form of compositional generalization is much more difficult. Please see the general response for more details.


The held-out evaluation should also include certain concepts that were never introduced in training, for example "made in" like an object made in a certain style. This would help in seeing the generalization capabilities of the models.

We understand the suggestion as evaluating concepts with no training exposure. Prior work shows that compositional support, i.e. every concept appearing in training, is a necessary condition for compositional generalization [6,7]: a model needs some minimum amount of experience with a concept before it can compose it. With the rare concepts (see general response), we test how little experience is enough: each is seen in exactly one training scene, and the model has to generalize it to new partners and new structures from there.

  • [6] Compositional Generalization from First Principles
  • [7] Discovering modular solutions that generalize compositionally

Evaluation of trained student models in section 3.1 also utilizes the same VLM pipeline used for verification. Human evaluation here as well would help make the paper stronger.

We additionally evaluated the student model with two further judges, a stronger Gemini model (Gemini 4 Argon) and a model from a different family (Claude Opus 5.5). The results stay qualitatively the same, with both correctly identifying less visually sharp concepts. The gap between training and held-out scenes remains for every judge. The scores are therefore not artificially high because Gemini 3.7 Flash lets a high number of wrong student images pass, or because the student mimics images Gemini is likely to accept.

Note that the Gemini 3.7 numbers differ slightly from the paper, as the evaluation now includes the rare concepts (see general response).

BlockFlash 3.7 trainFlash 3.7 held-outArgon trainArgon held-outOpus 5.5 trainOpus 5.5 held-out
A0.920.920.920.920.940.94
B0.870.810.900.850.940.91
C0.810.780.810.780.910.89
D0.630.600.650.660.660.65
E0.780.750.810.780.890.83
F-2-few0.660.520.630.520.780.72
F-2-all0.680.660.710.640.770.73
F-3-simple0.390.410.470.470.530.54
F-3-complex0.170.160.260.240.320.31
All blocks0.650.610.670.640.740.72

Figure 5 right shows the trained student model shows better results with a new VAE from Stable Diffusion. This does not help isolate if the gains are coming just from a VAE that is more heavily pre-trained of web-scale data or the fact that the VAE also has the ability to capture the complex cues in its latents.

We agree that the comparison does not separate these effects. In the paper, we note that the pretrained VAE alleviates a large part of the gap to the teacher, but that we do not control its pretraining distribution. To test whether the SD-VAE captures the concepts better in its latents, we train linear probes on the latents to predict the concepts of the image. The probe predicts, for each of up to three entities, the entity and its attributes, as well as the scene attributes and the relations between the entities. As VAEs produce localised representations, we additionally apply our probe on semantic SigLIP 2 embeddings. We find that the concepts are not linearly decodable from the latents of either VAE, so the SD-VAE does not provide a more structured latent space and the gain likely comes from reconstruction quality. We will clarify this in the main text.

BlockOwn VAE latentsSD-VAE latentsSigLIP 2 embedding
A0.04 / 0.020.03 / 0.020.75 / 0.65
B0.02 / 0.010.02 / 0.010.90 / 0.82
C0.05 / 0.030.04 / 0.020.97 / 0.96
D0.01 / 0.000.01 / 0.000.58 / 0.38
E0.01 / 0.010.01 / 0.010.93 / 0.82
F-2-few0.01 / 0.000.00 / 0.000.67 / 0.40
F-2-all0.00 / 0.000.00 / 0.000.53 / 0.52
F-3-simple0.00 / 0.000.00 / 0.000.15 / 0.13
F-3-complex0.00 / 0.000.00 / 0.000.01 / 0.01
All blocks0.02 / 0.010.01 / 0.010.59 / 0.50

L377 and Figure 7 mention that "generalization to evaluation scenes noticeably degrades for concepts with a lower compositional exposure". Figure 7 shows a dip in the pass rate for the evaluation scene of "cotton candy," which has a compositional exposure of 3. But two more scenes, "ice" and "honeycomb," with compositional exposures of 1 and 2, respectively, have significantly higher pass rates. How can we conclude that low exposure to scenes is the reason or is it that some scenes are inherently harder to generate?

This is a good point: some concepts are likely harder than others, and both effects are relevant. In general, however, there is a trend that concepts at lower exposure generalize worse and none of these textures are categorically hard to generate, as all of them pass at a high rate on training scenes (reported in Table 6 of the appendix). We will clarify this in the manuscript.


Figure 7 right shows the pass rate wrt to the th number of entities being generated. While the model is trained on 3 entities, the performance dips significantly if more than 3 entities are present during inference. This is a fair result, but is the dip happening because the 4th entity is generated incorrectly, or the model fails to generate the first 3 entities too? It would be interesting to see the entity-level pass rate here.

This is an interesting question. In general, the model generates three entities correctly and misses the fourth. We will add images of failure cases to the manuscript to illustrate this as well as a table with the recall rate for the objects.

ObjectsRecallPass rate
40.8280.306
50.7190.035
60.6170.000
70.5430.000

In Table 5, the model performs similarly on the in-distribution (ID) and held-out scenes, and both scores are low for the hardest compositions. For example, F-3-rel complex has pass rates of 12% and 10% respectively, highlighting limited success on those scenes.

Indeed, our benchmark introduces a challenging compositional evaluation that a standard text-to-image generation training pipeline does not yet solve.

Comments on the paper

2 comments

In response to Reviewer DYLS and Reviewer d2JF we added an additional evaluation split consisting of rare concepts. Each rare concept appears in exactly one simple training scene (with multiple images of the same scene). For example, the concept popcorn only appears in the scene “a chair made of popcorn” for which we generate 2560 distinct images. We then evaluate generalization of rare concepts to novel scene compositions of varying complexity (number of entities and number of concepts). This is a stricter version of our compositional exposure split, which limited the number of unique bindings within which a given concept was observed but still allowed these bindings to appear in scenes of varying complexity.

We find that while a model trained from scratch on our entire training dataset is able to produce the rare concept training scenes, it has substantial difficulty generalizing rare concepts to novel scene compositions of larger complexity despite being able to generally produce scenes of the same complexity.

Pass rate on rare concepts

CategoryTraining sceneOOD scene (same complexity)OOD scene (more complex)
Living object0.938N/A0.384
Non-living object0.812N/A0.216
Texture0.9840.3670.050
Action0.9920.4380.002
Archetype1.0000.4300.009
Background0.9840.6720.036
Style0.4770.1170.007
Relation0.9840.2720.017

The rare concepts per category

CategoryRare concepts
Living objectsrhino, anteater
Non-living objectsviolin, hairdryer
Texturespopcorn, tinfoil
Archetypesgraduate, viking
Actionshula-hooping, trampolining
Backgroundsgiant dice, moai statues
Stylesjigsaw-puzzle, pointillism
Relationsbalancing

Dear Reviewers, we would like to thank you again for your thoughtful and constructive feedback. With the end of the discussion period coming closer, we wanted to kindly ask whether our additional experiments and clarifications have helped resolve your concerns. Please let us know if you have any further questions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.