- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a large-scale synthetic dataset for evaluating visual compositional generalization. It contains about 2.5 million samples with structured scene descriptions, generated using a text-to-image model and validated by a VLM. The authors design systematic training and evaluation splits to test generalization to unseen concept combinations. By training a latent diffusion model from scratch, they demonstrate that diffusion models can generalize to novel compositions, while showing that this ability depends on sufficient compositional exposure and training coverage.
Critical strengths
- This paper propose addresses a central question: compositional generalization in a high-complexity, large-scale setting that existing synthetic datasets do not reach.
- This paper perform the split based on the bindings between entities and their attributes, as well as those between entities and their relational roles. This allows for a formal definition of whether a specific combination has appeared during training.
- Experiments for this method fully account for the impact of different VAEs on the results.
Critical weaknesses
- The paper uses Gemini for both dataset filtering and final evaluation, without independent validation using other VLMs. The authors' manual inspection found 13 incorrectly accepted images out of 430 samples, involving key issues such as entity relations and attribute binding. These errors directly affect the evaluation of compositional generalization.
- All images in the dataset are generated using FLUX.2-dev, which does not guarantee sufficient visual diversity. The dataset may inherit FLUX-specific biases in object appearance, composition, and the representation of relationships between entities.
- a small issue is that the paper further analyzes the impact of compositional exposure and scene complexity on generalization ability. Since the authors only conducted experiments on a single-size DIT, it is currently unclear to what extent these observed generalization limitations are affected by model capacity.
Questions for authors
- How did the authors assess the potential visual distribution bias that might be introduced by relying solely on FLUX.2-dev? Is it possible to conduct control experiments with alternative generators while maintaining comparability in scene grammar to verify whether the main conclusions about combinatorial generalization remain robust across different data sources?
- Could the authors use VLMs from different model families to independently review the student model generation results covering different complexities and seen/unseen combination?
2 comments
A small issue is that the paper further analyzes the impact of compositional exposure and scene complexity on generalization ability. Since the authors only conducted experiments on a single-size DIT, it is currently unclear to what extent these observed generalization limitations are affected by model capacity.
We agree with the reviewer that the effect of the DiT's capacity on generalization is an interesting question our dataset allows to investigate. We would first like to note that our primary contribution is the dataset and benchmark. Thus, the aim of our experiments was not to provide an exhaustive empirical evaluation of the benchmark, but to show, as a proof of concept, that a model can be trained and its generalization analyzed.
To give some early results in this direction, however, we did run experiments with two smaller DiTs, scaling the depth and width of the diffusion model and text encoder together (u = 8 is the size used in the main text). With increasing model size performance increases on both training and held-out scenes, while the gap between them stays similar.
| Model | Parameters | In-distribution | Held-out | Gap |
|---|---|---|---|---|
| u = 4 | 84M | 0.537 | 0.507 | 0.030 |
| u = 6 | 281M | 0.613 | 0.576 | 0.037 |
| u = 8 | 663M | 0.648 | 0.612 | 0.036 |
We thank the reviewer for the constructive feedback and the suggested experiments. We address each point below.
Thank you for this suggestion, we additionally evaluated the student model with two further judges, a stronger Gemini model (Gemini 4 Argon) and a model from a different provider (Claude Opus 5.5). The results stay qualitatively the same, with both overall accepting more images due to their ability to identify less visually sharp concepts and the gap between training and held-out scenes remains across judges.
This implies that despite some training images containing partial attribute bindings or incorrectly labelled relations, the student learns to correctly compose scenes from the vast majority of correct training images and it also confirms that our evaluation numbers are not artificially inflated as a result of Gemini 3.7 Flash consistently accepting such faulty compositions.
Note that the Gemini 3.7 Flash numbers differ slightly from the paper, as we added rare concepts seen in only a single training scene, a new experiment described in our general response.
While it is true that our dataset inherits aesthetic biases from FLUX.2-dev, we would like to emphasize that the scene structure and its constituents (concepts, bindings) are determined by our scene description language and image generation pipeline which validates that scene structure is generated correctly using an independent VLM judge.
Thus, while FLUX.2-dev ultimately determines the low-level visual details of each image, high-level concept compositions and relations do not depend on FLUX.2-dev.
As a result, our training images are visually diverse (e.g. see Fig. 1 and the sample images provided in the supplementary material for qualitative evidence) covering the diverse concepts and scene structures defined by our scene description language.
In general, our dataset generation pipeline can be used with different generators but regenerating the millions of images contained in our dataset would incur significant cost (on the order of thousands of B200 hours and thousands of dollars in VLM API costs).