- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Compositional generalization tasks may require models to reuse a learned concept library. Do pretrained diffusion models (DM) internally acquire such a concept library? This work tackles the problem with gradient-ascent-based prototype discovery. The prototypes are collected by estimating the local argmax of the implicit density function of the generative distribution, which is modeled by Gaussian mixture models per time step. The discovered prototypes are leveraged to compose score functions in the style of a weighted product of experts for better out-of-distribution generalization. Experiments on the compositional settings in Color-MNIST, CelebA, and CIFAR-10 reveal that the proposed method achieved higher OOD-sample generation quality than the baselines, including editing-based SDEdit.
Critical strengths
- The idea of prototype composition is natural and attracting to explain the extraordinaly OOD generalizability of modern diffusion models.
- The experiments are small but well designed to observe compositional generalization behavior of the models
Critical weaknesses
-
The limited experimental scale and range: Although the small controlled experiments are good proof of concept, I would like to see whether larger scale, product-ready diffusion models also show similar prototyping behavior.
-
Why the discovered mode peaks can be called "prototypes" is not deeply discussed. While it is possible that they are internal prototypes, it is also possible that they are simply cluster centers of natural images. The results that the number of prototypes differs is partially insightful, but the number of clusters is also externally controllable in existing clustering algorithms, and I do not think that this is conclusive.
Questions for authors
-
CIFAR and CelebA include almost the same size objects. Is the proposed method expected to work for multi-scale objects? It is questionable how the prototype represents difference scales of objects.
-
Minor comments:
- Eqn. 5: "arg local max" is understandable but a bit strange. I prefer argmax_{x \in local}
We thank the reviewer for finding the idea of prototype composition natural and the experiments well designed.
They do. Section 4 of the revised version runs its analysis on four pretrained models. Next to the two DDPMs it has DiT-XL/2 (Peebles & Xie, 2023), a 0.68B-parameter model trained on ImageNet at 256×256, and the text-to-image latent diffusion model of Rombach et al. (2022), a 1.45B-parameter model trained on LAION-400M and the architecture that Stable Diffusion is built on. For each of the two we take 256 of its own samples, from 16 classes or 16 prompts. We report the number of distinct prototypes that the images reach at each noise level t, starting from the number of images (Figure 2 and Table 2 in the revised version, more examples in Figure 5):
Both large models show the prototyping behavior of Section 4. Every image has its own prototype at low noise, and more and more images end at the same prototype as the noise grows, until a single one remains. Images with the same content share a prototype first. On DiT-XL/2, 44% of the pairs of images from the same ImageNet class share a prototype at t=440, against 21% of the pairs from different groups of classes (for example a dog and a car). On the text-to-image model, 8.7% of the pairs from the same prompt share a prototype at t=580, against 4.2% of the pairs from unrelated prompts (Appendix B.2 and Figure 6 in the revised version).
A cluster center is a property of the images, and a prototype is a property of the model. Our ColorMNIST model is trained on 12 of the 16 combinations of digit color and background color. We ran mode ascent from digits in a combination that was never in training and from the same digits recolored to a combination that was in training, with the same background, so the two images differ only in the digit color. At t=300 the prototype still has the colors of its image in 89.8% of the runs for the combination seen in training and in 9.7% for the unseen one (40 digits, 32 runs each, Table 3 and Figure 7 in the revised version). What a prototype keeps depends on what the model was trained on. A shared prototype is also not the center of the images that share it. On DiT-XL/2 it lies 1.2 to 1.8 times as far from their mean as the images themselves. Mode ascent has no number of clusters and no step size, so the number of prototypes at each noise level is measured, not set. The prototypes are also reusable. Eight of them, taken only from the model's own samples, compose a ColorMNIST class the model never saw with 92.5% attribute correctness, against 37.7% for Comp. 1 (Table 11 and Figure 12 in the revised version).
Yes. A prototype is a full image at the resolution of the model, so it carries an object at whatever scale the model generates it. The ImageNet classes and the prompts in the experiment above cover objects of different sizes in the frame, from close-ups of fruit to dogs, birds and cars, and the behavior of Section 4 holds on them.
We changed Eq. (5) to xt∗∈argmaxxt∈Upt(xt) for an open neighborhood U of xt∗.
Since the new results only strengthen our arguments, we will update them in the final version.
Peebles & Xie. Scalable Diffusion Models with Transformers. ICCV 2023. Rombach et al. High-Resolution Image Synthesis with Latent Diffusion Models. CVPR 2022.