- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies model collapse in latent diffusion models, where recursive training on synthetic images progressively reduces generation quality and diversity and can amplify existing biases. The authors frame image generation as a Lewis signaling game: the diffusion model acts as a sender, and a frozen CLIP image encoder serves as a receiver. They introduce a communication loss that encourages alignment between generated images and their text prompts. The authors conduct recursive fine-tuning experiments on CelebA-HQ and COCO to evaluate the effects of regularization strength, prompt specificity, and regularization schedules. However, FID still deteriorates across generations.
However, the proposed method does not fundamentally change the deterioration trend. In Figure 3, FID generally increases from the second generation onward under both the baseline and regularized settings. Although stronger regularization reduces the degree of degradation, the results do not show a consistent delay in its onset or establish whether the improvements persist over more generations. The qualitative examples in Figure 1 also show substantial degradation across generations, providing limited evidence that the method effectively preserves image quality and diversity.
Overall, I find the experimental evidence insufficient to support the claimed effectiveness in mitigating model collapse. Given these limitations, I recommend clear rejection(1/4).
Critical strengths
The authors offer a new perspective on model collapse through the lens of Lewis signaling, examining how communication pressure can help preserve semantic information during recursive training.
Critical weaknesses
-
Dependence on predefined caption attributes. For CelebA-HQ, the method uses structured captions to specify properties for CLIP-based image–text alignment. These captions provide constraints on predefined attributes, but do not explicitly protect information omitted from the descriptions. Although facial attributes are relatively straightforward to enumerate, extending this approach to more complex scenes with diverse objects, relationships, and fine-grained details requires further investigation. The COCO experiments provide some evidence of broader applicability, but do not systematically evaluate which information is preserved or lost.
-
Failure modes beyond the specified attributes. During recursive training, alignment with a caption does not ensure preservation of the full visual content. As the paper demonstrates, unspecified properties such as viewing angle and hand position can disappear, while specified attributes can become exaggerated or be interpreted unexpectedly. These failures raise concerns about whether improved caption alignment reliably translates into preservation of the underlying data distribution.
-
Limited evidence of sustained collapse mitigation. Although the method reduces FID deterioration in some settings, FID continues to increase across generations, and the reported curves do not establish a consistent delay in the onset of degradation.
-
Limited experimental coverage and evaluation independence. The experiments are restricted to a lightweight diffusion model, small training sets, and a fixed 90% synthetic replacement rate, without reported variability across repeated runs. Moreover, the method directly optimizes CLIP image–text similarity and also uses CLIP Score for evaluation. This overlap makes improvements in CLIP Score difficult to interpret as independent evidence of preserved quality or diversity: a model may improve the optimized similarity measure while losing unspecified information or exploiting biases in the evaluator.
-
Insufficient characterization of the regularization weight. The weight is central to the method because it controls the balance between diffusion learning and semantic alignment. Although the paper compares , , and , the sparse sweep does not sufficiently characterize sensitivity or establish a reliable selection strategy. Notably, the low-weight setting performs worse than the baseline in the reported CelebA-HQ FID curves, whereas larger weights may encourage excessive emphasis on caption attributes.
Questions for authors
-
Could the authors clarify how the training dataset is constructed in the second generation and thereafter? Does each generation still retain 10% real images, and are these the same images retained in the first generation? How is the remaining 90% synthetic data generated—does the previous generation’s model use the original real images or the previous generation’s synthetic images as image-to-image inputs?
-
Could the authors extend the ablation of with a finer sweep and repeated runs to assess sensitivity and clarify how this weight should be selected?
-
Could the authors include additional evaluation metrics beyond FID and CLIP Score, particularly independent measures of diversity and attribute retention, given that CLIP alignment is directly optimized during training?
-
Could the authors provide a more theoretical account of the Lewis signaling formulation from a distributional perspective? In particular, under what conditions can the communication loss preserve modes or tail information in the data distribution during recursive training, rather than only improve alignment with specified caption attributes?
3 comments
Thank you for your response. Your clarification resolves my question about how the training data are constructed at each generation.
Thanks for the authors’ interpretation of the evidence for collapse mitigation, but I still disagree. I would expect to see a visible delay in image quality degradation: for example, if baseline images begin to degrade at generation 2, images produced with regularization should reach a comparable level of degradation only in later generations. The reported FID curves show less deterioration in some settings, but the examples in Figure 1 do not clearly show such a delay or visibly better preservation of image quality across generations.
I would also appreciate a finer sweep of the regularization weight to assess how sensitive the results are to this choice.
Official Comment by Authors to Reviewer DoSt
We thank the reviewer for their response and affording us the opportunity to engage further.
Regarding when there should be a visible delay in collapse: on lines 292 to 295 of the paper we mention "We investigated applying this Lewis Signaling regularised diffusion loss throughout training as well as waiting for generation 3 once the collapse has visibly and measurably started". Thus, we only start applying our regulariser from Generation 3 - exactly where we see an immediate difference between Baseline Collapse and Lewis Signalling Regulariser in Figure 1. Similarly this is where we see a distinction in the metrics on Figure 3. The initial drop in CLIP score and increase in FID score in Figure 3 up to Generation 2 corresponds to normal collapse and demonstrates how clears these metrics are for showing the collapse. Once we begin to apply the regulariser in Figure 3 the metrics clearly diverge - CLIP score goes back up and the FID flattens out depending on our regularisation scale. With this in mind, we see a clear monotonic improvement from increasing the regularisation rate on these plots. We will aim to provide a finer sweep of this regularisation rate, however this also explains why the three initial regularisation weights were used and concluded to provide a clear result.
There is an exception to which generation we apply the regulariser from: the LLM generated labels and the augmented labels. For Section 4.2 we apply the regulariser from the first generation as the purpose of that section is to isolate how the prompt quality can guide the collapse. This explains the difference in Figure 1 where the third set of plots: "Lewis Regularisation with Augmented Labels" has an immediate effect. The top left plot of Figure 3 is then also the exception and begins regularising from the first generation. If we apply Lewis Regularisation from the first generation with minimal prompts then the regulariser can overpower the diffusion training leading to the semantic overfitting we discuss later on. This is also why we focus so heavily on 1. the prompt quality, 2. the semantic overfitting the regulariser can be prone to.
In summary, we do see an immediate impact from the regulariser and the paper tries applying the regulariser from Generation 3 in Section 4.1 and from Generation 1 in Section 4.2 and 4.3. We do acknowledge that this is certainly not clear enough in the text or figure captions and is implicit in the figures. Thus, we will clarify this in the revised draft and thank the reviewer for assisting us in fixing a clarity issue which does stand to undermine our results.
Response by Authors to Reviewer DoSt
We thank the reviewer for their thoughtful consideration of our work. We address each point below:
We thank the reviewer again for their constructive feedback.
[1] Shumailov, Ilia, et al. "AI models collapse when trained on recursively generated data." Nature 631.8022 (2024): 755-759.
[2] Dohmatob, Elvis, Yুনzhen Feng, and Julia Kempe. "Model collapse demystified: The case of regression." Advances in Neural Information Processing Systems 37 (2024): 46979-47013.