acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Model Collapse in Latent Diffusion Models using a Lewis Signaling Regulariser

Abstract

Model Collapse, a degenerative process in which models forget the underlying data distribution when trained on sequences of synthetic data, is a looming problem as generated content proliferates on the internet. This stands to not only undermine the general utility of the models but also introduces mechanisms that lead to systemic bias. In this work, we aim to mitigate the collapse by introducing a novel regularisation method that frames latent diffusion as a Lewis Signaling Game. In this framing, the diffusion model is a “sender” that transforms a text embedding, the “state”, into an image which acts as the “signal”. This signal must then be processed by a “receiver”, a pretrained CLIP vision encoder whose goal is to retrieve the original text embedding. This provides a communication pressure which forces the diffusion model to remain sufficiently expressive. Our results show a clear improvement and mitigation of model collapse over multiple generations. Further, we find that the specificity of text prompts significantly impacts our method’s utility. Characteristics of the image described in the text tend toward a minimal set of stereotypical motifs. As such, precise, detailed prompts help encourage generations in the tail of the distribution, helping to prevent collapse along those axes. Finally, we show how this method takes a step towards avoiding bias in latent diffusion, such as the lightening of skin tones. Overall, this method supports the notion that model collapse results from ungrounded content generation and demonstrates that explicitly modeling a receiver or verification step in the training pipeline helps prevent collapse.

Primary area:
generative models
Keywords:
model collapse, clip, lewis signalling, latent diffusion

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_DoStOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies model collapse in latent diffusion models, where recursive training on synthetic images progressively reduces generation quality and diversity and can amplify existing biases. The authors frame image generation as a Lewis signaling game: the diffusion model acts as a sender, and a frozen CLIP image encoder serves as a receiver. They introduce a communication loss that encourages alignment between generated images and their text prompts. The authors conduct recursive fine-tuning experiments on CelebA-HQ and COCO to evaluate the effects of regularization strength, prompt specificity, and regularization schedules. However, FID still deteriorates across generations.

However, the proposed method does not fundamentally change the deterioration trend. In Figure 3, FID generally increases from the second generation onward under both the baseline and regularized settings. Although stronger regularization reduces the degree of degradation, the results do not show a consistent delay in its onset or establish whether the improvements persist over more generations. The qualitative examples in Figure 1 also show substantial degradation across generations, providing limited evidence that the method effectively preserves image quality and diversity.

Overall, I find the experimental evidence insufficient to support the claimed effectiveness in mitigating model collapse. Given these limitations, I recommend clear rejection(1/4).

Critical strengths

The authors offer a new perspective on model collapse through the lens of Lewis signaling, examining how communication pressure can help preserve semantic information during recursive training.

Critical weaknesses
  1. Dependence on predefined caption attributes. For CelebA-HQ, the method uses structured captions to specify properties for CLIP-based image–text alignment. These captions provide constraints on predefined attributes, but do not explicitly protect information omitted from the descriptions. Although facial attributes are relatively straightforward to enumerate, extending this approach to more complex scenes with diverse objects, relationships, and fine-grained details requires further investigation. The COCO experiments provide some evidence of broader applicability, but do not systematically evaluate which information is preserved or lost.

  2. Failure modes beyond the specified attributes. During recursive training, alignment with a caption does not ensure preservation of the full visual content. As the paper demonstrates, unspecified properties such as viewing angle and hand position can disappear, while specified attributes can become exaggerated or be interpreted unexpectedly. These failures raise concerns about whether improved caption alignment reliably translates into preservation of the underlying data distribution.

  3. Limited evidence of sustained collapse mitigation. Although the method reduces FID deterioration in some settings, FID continues to increase across generations, and the reported curves do not establish a consistent delay in the onset of degradation.

  4. Limited experimental coverage and evaluation independence. The experiments are restricted to a lightweight diffusion model, small training sets, and a fixed 90% synthetic replacement rate, without reported variability across repeated runs. Moreover, the method directly optimizes CLIP image–text similarity and also uses CLIP Score for evaluation. This overlap makes improvements in CLIP Score difficult to interpret as independent evidence of preserved quality or diversity: a model may improve the optimized similarity measure while losing unspecified information or exploiting biases in the evaluator.

  5. Insufficient characterization of the regularization weight. The weight ww is central to the method because it controls the balance between diffusion learning and semantic alignment. Although the paper compares w=0.10w=0.10, 0.500.50, and 0.850.85, the sparse sweep does not sufficiently characterize sensitivity or establish a reliable selection strategy. Notably, the low-weight setting performs worse than the baseline in the reported CelebA-HQ FID curves, whereas larger weights may encourage excessive emphasis on caption attributes.

Questions for authors
  1. Could the authors clarify how the training dataset is constructed in the second generation and thereafter? Does each generation still retain 10% real images, and are these the same images retained in the first generation? How is the remaining 90% synthetic data generated—does the previous generation’s model use the original real images or the previous generation’s synthetic images as image-to-image inputs?

  2. Could the authors extend the ablation of ww with a finer sweep and repeated runs to assess sensitivity and clarify how this weight should be selected?

  3. Could the authors include additional evaluation metrics beyond FID and CLIP Score, particularly independent measures of diversity and attribute retention, given that CLIP alignment is directly optimized during training?

  4. Could the authors provide a more theoretical account of the Lewis signaling formulation from a distributional perspective? In particular, under what conditions can the communication loss preserve modes or tail information in the data distribution during recursive training, rather than only improve alignment with specified caption attributes?

3 comments
AuthorsOpenReview ↗

Response by Authors to Reviewer DoSt

We thank the reviewer for their thoughtful consideration of our work. We address each point below:

  1. On predefined captions and preservation of the underlying data distribution: As discussed in our general response, our objective is not to preserve every feature of the original distribution, but to mitigate the loss of information expressed through textual conditioning. Structured captions and CelebA-HQ allow us to study this systematically in a controlled setting. While uncaptioned features may be lost, our contribution is demonstrating how semantic preservation can be made interpretable and steerable, including identifying failure modes such as over-stylisation and ambiguous prompts.
  2. On the limited evidence of sustained collapse: We explicitly aim to mitigate, rather than eliminate, model collapse. As discussed in our general response, some distributional change is inevitable, and our focus is on controlling which semantic information is preserved. Our experiments demonstrate both improvements in FID and interpretable failure modes arising from overfitting to a single receiver. We consider understanding these limitations and their fairness implications an important contribution, with multiple receivers representing a promising extension.
  3. On limited experimental coverage and experimental design: As discussed in our general response, our 90% replacement regime and smaller-scale experiments follow established model collapse literature [1], allowing collapse to be studied systematically and efficiently. While these conditions may accelerate collapse relative to practice, they serve our experimental objectives. Moreover, existing research demonstrates that model collapse can undermine neural scaling laws [2] and is not necessarily resolved by increasing model scale. Thus, controlled, smaller-scale experiments remain appropriate for studying the mechanisms of model collapse, even if the quantitative effects may vary with scale. Similarly, we report CLIP score for completeness, but base our conclusions primarily on FID and controlled qualitative comparisons, recognising the limitations of existing collapse metrics. We would welcome any suggestions for general metrics of collapse in diffusion models.
  4. On the number of images maintained across generations: At each generation (k), a random 10% of images is set aside from the real images, while the remaining 90% of real images are used to seed image generation using the model from the previous generation (k-1), these are combined to create the training set for generation k. Thus, image generation to create the training set is always conditioned on real image samples. This follows established recursive training protocols, including Shumailov et al. (2024) [1].
  5. On extending the regularisation sweep: We agree that regularisation strength requires careful tuning, and our experiments explicitly demonstrate the negative effects of poorly calibrated values. We are happy to extend the sweep during the discussion period and welcome the reviewer's guidance on the desired coverage. For practical applications, we recommend selecting the regularisation strength through standard hyperparameter tuning procedures. We will mention this in the revised manuscript.
  6. On additional metrics for model collapse: As discussed in our general response, no established metric specifically captures model collapse in diffusion models. Attribute-specific classifiers or measures such as intrinsic dimensionality may be informative, but would require additional validation before being used as reliable measures of collapse. We therefore rely primarily on FID, which has precedent in the literature, and welcome suggestions for alternative established metrics.
  7. On providing a theoretical analysis: We appreciate the suggestion, but a theoretical treatment is beyond the scope of this empirical study. Existing model collapse theory largely focuses on simplified settings, such as linear regression, whereas our approach involves complex semantic representations learned by CLIP. Formalising these dynamics is an interesting direction for future work. Importantly, our method operates over this semantic space rather than simply preserving individual caption attributes. This distinction between semantic alignment compared to literal text alignment at a word-level is crucial for understanding our results and for interpreting the failure modes we highlight. It is also what lends the Lewis Game framing and makes our method particularly interpretable and steerable.

We thank the reviewer again for their constructive feedback.

[1] Shumailov, Ilia, et al. "AI models collapse when trained on recursively generated data." Nature 631.8022 (2024): 755-759.
[2] Dohmatob, Elvis, Yুনzhen Feng, and Julia Kempe. "Model collapse demystified: The case of regression." Advances in Neural Information Processing Systems 37 (2024): 46979-47013.

Thank you for your response. Your clarification resolves my question about how the training data are constructed at each generation.

Thanks for the authors’ interpretation of the evidence for collapse mitigation, but I still disagree. I would expect to see a visible delay in image quality degradation: for example, if baseline images begin to degrade at generation 2, images produced with regularization should reach a comparable level of degradation only in later generations. The reported FID curves show less deterioration in some settings, but the examples in Figure 1 do not clearly show such a delay or visibly better preservation of image quality across generations.

I would also appreciate a finer sweep of the regularization weight ww to assess how sensitive the results are to this choice.

AuthorsOpenReview ↗

Official Comment by Authors to Reviewer DoSt

We thank the reviewer for their response and affording us the opportunity to engage further.

Regarding when there should be a visible delay in collapse: on lines 292 to 295 of the paper we mention "We investigated applying this Lewis Signaling regularised diffusion loss throughout training as well as waiting for generation 3 once the collapse has visibly and measurably started". Thus, we only start applying our regulariser from Generation 3 - exactly where we see an immediate difference between Baseline Collapse and Lewis Signalling Regulariser in Figure 1. Similarly this is where we see a distinction in the metrics on Figure 3. The initial drop in CLIP score and increase in FID score in Figure 3 up to Generation 2 corresponds to normal collapse and demonstrates how clears these metrics are for showing the collapse. Once we begin to apply the regulariser in Figure 3 the metrics clearly diverge - CLIP score goes back up and the FID flattens out depending on our regularisation scale. With this in mind, we see a clear monotonic improvement from increasing the regularisation rate on these plots. We will aim to provide a finer sweep of this regularisation rate, however this also explains why the three initial regularisation weights were used and concluded to provide a clear result.

There is an exception to which generation we apply the regulariser from: the LLM generated labels and the augmented labels. For Section 4.2 we apply the regulariser from the first generation as the purpose of that section is to isolate how the prompt quality can guide the collapse. This explains the difference in Figure 1 where the third set of plots: "Lewis Regularisation with Augmented Labels" has an immediate effect. The top left plot of Figure 3 is then also the exception and begins regularising from the first generation. If we apply Lewis Regularisation from the first generation with minimal prompts then the regulariser can overpower the diffusion training leading to the semantic overfitting we discuss later on. This is also why we focus so heavily on 1. the prompt quality, 2. the semantic overfitting the regulariser can be prone to.

In summary, we do see an immediate impact from the regulariser and the paper tries applying the regulariser from Generation 3 in Section 4.1 and from Generation 1 in Section 4.2 and 4.3. We do acknowledge that this is certainly not clear enough in the text or figure captions and is implicit in the figures. Thus, we will clarify this in the revised draft and thank the reviewer for assisting us in fixing a clarity issue which does stand to undermine our results.

Reviewer_PhqBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates model collapse in latent diffusion models under recursive fine-tuning on predominantly synthetic data. It interprets text-conditioned generation as a Lewis signaling game, with the diffusion model acting as a sender and a frozen CLIP image encoder acting as a receiver. Operationally, the method adds an image–text cosine-alignment loss, evaluated on decoded clean-latent predictions, to the standard diffusion objective. The experiments use segmind/tiny-sd on subsets of CelebA-HQ and COCO over five generations, replacing 90% of the dataset with synthetic images at each generation. The authors report improved FID and CLIP scores for selected regularization settings and investigate the effects of regularization weight, timestep range, and prompt specificity. The authors further argue that explicitly describing attributes such as race, age, and expression helps retain those attributes and mitigate bias during recursive training.

Critical strengths
  1. A simple intervention with preliminary evidence of effectiveness across two domains. The method uses a frozen pretrained semantic model to provide an additional training signal without modifying the diffusion architecture. The FID improvements reported for suitable regularization settings on both CelebA-HQ and COCO provide evidence beyond improvement in the directly optimized CLIP metric. This supports the feasibility of using an external semantic reference to reduce some forms of degradation during recursive fine-tuning.

  2. The prompt-specificity analysis provides useful insight into the intervention’s limitations. The paper examines both attribute-enriched captions and linguistic variations, and reports failures such as exaggerated features, loss of unspecified attributes, and literal interpretations of ambiguous descriptions. These observations help characterize which information the regularizer preserves and where its behavior becomes unreliable.

Critical weaknesses
  1. Insufficient novelty to meet the acceptance bar for ICLR. The proposed method essentially adds a conventional CLIP-based image–text alignment loss to the standard diffusion training objective, which is already a well-established regularization strategy. The Lewis signaling framework primarily serves as a conceptual reinterpretation rather than introducing a fundamentally new algorithm, optimization mechanism, or theoretical insight. Furthermore, the observation that more detailed prompts improve semantic consistency is largely expected. Overall, the technical contribution is highly incremental and, in my assessment, falls below the novelty standard expected for acceptance at ICLR.

  2. Insufficient experimental validation to support the main claims. The experiments are limited to a lightweight diffusion model (tiny-sd) and relatively small datasets, raising concerns about generalizability. More importantly, the evaluation relies primarily on FID and CLIP Score, with the latter directly optimized by the proposed objective. There is no quantitative evaluation of distributional diversity, mode coverage, or demographic bias mitigation, despite these being central claims of the paper. The lack of stronger baselines and statistical uncertainty further weakens the experimental evidence.

Questions for authors
  1. Clarification of the actual algorithm. Section 3.2 defines the communication term as a training loss, whereas Section 4.3 and Appendix B.2 describe applying it during inference and claim that restricting its application reduces inference cost. Is CLIP used during training, sampling, or both? Please specify the corresponding update procedure and explain how the range (t \in [0,350]) corresponds to the “first 350 inference steps,” given that low forward-diffusion timesteps normally correspond to later reverse-denoising stages. This clarification is necessary to interpret both the timestep ablations and the efficiency claims.

  2. Clarification of the recursive protocol and causal attribution. At each generation, are evaluation images produced from the same held-out original images or from the preceding synthetic images? What degradation occurs if the identical recursive image-to-image procedure is performed with a frozen diffusion model and no fine-tuning? Please also specify the per-generation training budget and image-to-image sampling settings. These details are necessary to distinguish degradation caused by recursive learning from cumulative image-transformation artifacts.

  3. Evidence for distributional preservation and bias mitigation. What quantitative evidence distinguishes improved caption compliance from preservation of within-prompt diversity, rare attributes, and demographic representation? In particular, are there matched comparisons between augmented prompts with and without the regularizer, extending the qualitative control in Figure 6, that measure group-wise attribute or skin-tone retention using independent evaluation signals? Such evidence, together with variability across independent runs, would materially affect my assessment of the central claims.

4 comments
AuthorsOpenReview ↗

Response by Authors to Reviewer PhqB

We thank the reviewer for their thoughtful consideration of our work. We address each point below:

  1. On the novelty of our method: As discussed in our general response, our method differs from CLIP-guided diffusion [1] by applying regularisation exclusively during training, and from REPA and softREPA [2,3] by aligning generated images with text embeddings rather than external visual representations. This semantic alignment makes model collapse more interpretable and steerable. We agree that REPA and softREPA are appropriate baselines and will endeavour to include these comparisons in the revised manuscript. We also welcome references to other closely related methods if we have not interpreted the prior works being suggested correctly.
  2. On the scale of our experiments: As discussed in our general response, our experimental design follows established model collapse literature, including Shumailov et al. (2024) [4]. Our objective is to induce collapse in a controlled and computationally efficient manner to study its mechanisms and mitigation. Existing research demonstrates that increasing model scale does not necessarily prevent model collapse, although its severity may depend on scale [5]. Thus, smaller-scale experiments in controlled settings remain appropriate for investigating the mechanisms of model collapse and evaluating mitigation strategies, while recognising that their quantitative effects may differ at larger scales. Hence, we study a “worst-case” scale in this work to induct model collapse efficiently.
  3. On using CLIP to both optimise and evaluate the model: We agree that CLIP score cannot independently establish the effectiveness of a method that directly optimises it. As discussed in our general response, we report CLIP score for completeness, while our conclusions rely primarily on FID and qualitative comparisons. We acknowledge the limitations of FID as a proxy for model collapse and welcome suggestions for additional independent evaluation metrics.
  4. On the first question (training versus inference): The regulariser is applied exclusively during training and is not used during inference, which then follows the standard Stable Diffusion generation process. We will clarify this distinction in the revised manuscript.
  5. On the clarification of the recursive protocol: At each generation (k), a random 10% of images is set aside from the real images, while the remaining 90% of real images are used to seed image generation using the model from the previous generation (k-1), these are combined to create the training set for generation k. This follows established recursive training protocols, including Shumailov et al. (2024) [4]. As discussed in our general response, our objective is to study accumulated model collapse under controlled conditions. Distinguishing degradation caused by repeated sampling from that caused by retraining is nontrivial, particularly without a reliable measure of model collapse. To our knowledge, now works in model collapse make a distinction between these two effects. Nevertheless, we will conduct the suggested frozen-model experiment to better distinguish these effects, while acknowledging that sampling errors may also reflect limitations of the pretrained model. Regarding the training and sampling settings: training time is 50 epochs per generation which is roughly 35600 update steps. Our diffusion budget during training is 1000 steps and during inference it is 250 steps since the first few generations produce realistic samples at this budget while leading to collapse. The guidance rate which pulls towards the sample image is 0.8 and this is the same for all runs.
  6. On the evidence for bias mitigation: As discussed in our general response, our objective is not to eliminate all distributional change, but to control which semantic information is preserved. The distinction between caption compliance and within-prompt diversity is important, but requires specifying which unmentioned attributes should be maintained. Our experiments, including the "hazelnut" failure case and bias analyses, aim to investigate precisely these limitations of semantic preservation. We use textual conditioning as a guide to relevant semantic information, with CLIP providing a general measure of text-image alignment. While CLIP is appropriate for this purpose, we acknowledge that optimising it limits its value as independent evidence of bias mitigation. Attribute-specific classifiers could provide complementary evidence, although they measure narrower aspects of the distribution. We will consider additional independent measures of semantic preservation and diversity, while emphasising that the CLIP alignment metric appears to be the appropriate evaluation metric for the question asked by the reviewer.

We thank the reviewer again for their constructive feedback and helpful suggestions.

AuthorsOpenReview ↗

Response by Authors to Reviewer PhqB References

[1] Kim, Gwanghyun, Taesung Kwon, and Jong Chul Ye. "Diffusionclip: Text-guided diffusion models for robust image manipulation." 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2022.
[2] Yu, Sihyun, et al. "Representation alignment for generation: Training diffusion transformers is easier than you think." arXiv preprint arXiv:2410.06940 (2024).
[3] Lee, Jaa-Yeon, et al. "Aligning text to image in diffusion models is easier than you think." Advances in Neural Information Processing Systems 38 (2026): 157106-157136.
[4] Shumailov, Ilia, et al. "AI models collapse when trained on recursively generated data." Nature 631.8022 (2024): 755-759.
[5] Dohmatob, Elvis, Yুনzhen Feng, and Julia Kempe. "Model collapse demystified: The case of regression." Advances in Neural Information Processing Systems 37 (2024): 46979-47013.

Reviewer_PhqBOpenReview ↗

Keep rating unchanged

I thank the authors for their rebuttal, however, the authors' rebuttal did not convince me and I will keep my rating toward this paper unchanged.

AuthorsOpenReview ↗

Response by Authors

We thank the reviewer for responding to our rebuttal. If the reviewer could let us know what remains unconvincing we would be grateful for the opportunity to use the discussion period productively. Additionally, we would still value the citations that the reviewer alluded to which undermines the novelty of our work. We have discussed and showed how the current literature that we thought the reviewer was alluding to is different from our approach in important ways, for example the difference between a visual or semantic embedding. Since our rebuttal is not convincing, should we assume that the reviewer has other works in mind?

Reviewer_jnXnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Lewis Signaling regularisation framework for mitigating model collapse in recursively trained latent diffusion models. A latent diffusion model (sender) generates images from noisy latents conditioned on captions, while a frozen pretrained CLIP image encoder (receiver) supplies a communication target by aligning generated-image embeddings with caption text embeddings during training. At inference only the diffusion model is used for generation. Experiments on CelebA-HQ and COCO 2017 across five recursive generations show gains in FID and CLIP Score under severe synthetic-data poisoning.

Critical strengths

a. It reframes recursive latent diffusion training as a Lewis Signaling Game, linking communication pressure and receiver-based grounding from language evolution to the mitigation of model collapse, benifiting generative model training.

b. The proposed regulariser is a plug-and-play module, which introduces only an auxiliary communication loss without changing the whole diffusion backbone.

c. With extensive experiments and ablations, this method showcase its effectiveness in mitigating model collapse under severe synthetic-data poisoning.

Critical weaknesses

a. The empirical evaluation is limited in scale and model choice. All recursive experiments use the lightweight segmind/tiny-sd model, only five generations, and a worst-case poisoning setting in which 90% of each generation’s data is synthetic. This makes it unclear whether the reported benefits generalize to larger diffusion models, longer recursive runs, or more realistic contamination rates. The extreme setup may overstate the practical advantage of the regulariser.

b. The evaluation is not fully independent of the training objective. The communication loss explicitly optimizes CLIP image–text cosine similarity, and the same one is then used to report CLIP Score.

c. The paper lacks a systematic sensitivity analysis of the CLIP receiver. The communication loss uses a single frozen CLIP image encoder as the sole receiver, with no comparison across CLIP variants, held-out receivers, or multi-receiver settings. Since the regulariser directly optimizes alignment to this receiver, it is unclear whether performance gains reflect true semantic preservation or receiver-specific overfitting.

Questions for authors

a. The paper is ambiguous about whether the communication loss is applied during training, during sampling, or both: the text refers to “apply the regulariser early during sampling” and “first 350 inference steps”, while Eq. (5) defines it as a training loss over denoising timesteps. The authors should clarify whether the loss is computed only during training, or also imposed as an inference-time constraint, and how these two settings relate.

b. Here FID is computed against a reference set sampled only once, and no multi-seed variance or confidence intervals are reported. The authors should clarify whether the reported FID improvements are statistically significant across random seeds, and whether the generation at which collapse starts is consistent across seeds.

2 comments
AuthorsOpenReview ↗

Response by Authors to Reviewer jnXn

We thank the reviewer for their thoughtful consideration of our work. We address each point below:

  1. On the scale of our experimentation: As discussed in our general response, our experimental design follows established model collapse literature, including Shumailov et al. (2024) [1], and deliberately induces collapse under controlled, computationally efficient conditions. While this may exaggerate the prevalence or rate of collapse in practice, it does not undermine our objective of studying and mitigating its effects. The real-world prevalence of model collapse remains an active debate [2,3].
  2. On the CLIP metric: As discussed in our general response, we agree that CLIP score cannot independently establish the effectiveness of a method that directly optimises it. We report it for completeness and make this point in the paper as well. Thus our conclusions rely primarily on FID and controlled qualitative comparisons. We welcome suggestions for additional metrics.
  3. On the evaluation of and overfitting to the CLIP receiver: Our quantitative improvements are measured using FID, which is independent of the CLIP receiver. As discussed in our general response, the implications of overfitting to the receiver is a central aspect of our analysis. The "hazelnut" hair example demonstrates interpretable and steerable failure modes arising from overemphasised prompt semantics, which can be mitigated through clearer prompting. We deliberately focus on a single receiver to investigate these behaviours and their fairness implications, while recognising multiple receivers as a promising extension that we mention in the paper.
  4. On the first question (training versus inference): The regulariser is applied exclusively during training. Inference follows the standard Stable Diffusion generation process, distinguishing our approach from inference-time methods such as CLIP-guided diffusion. We will clarify this in the manuscript.
  5. On the second question (statistical significance): We will discuss the statistical significance of our FID results. We note that collapse begins from the first generation and is a gradual process, although quantifying or defining its extent remains an open research question, as discussed in our general response.

We thank the reviewer again for their constructive feedback and helpful suggestions.

[1] Shumailov, Ilia, et al. "AI models collapse when trained on recursively generated data." Nature 631.8022 (2024): 755-759.
[2] Schaeffer, Rylan, et al. "Position: Multiple Definitions & Unrealistic Assumptions of Model Collapse Distract from Real World Threats." Forty-third International Conference on Machine Learning Position Paper Track. (2026).
[3] Jarvis, Devon, et al. "Position: the Stochastic Parrot in the Coal Mine. Model Collapse is a Threat to Low-Resource Communities." Forty-third International Conference on Machine Learning Position Paper Track. (2026).
[4] Kim, Gwanghyun, Taesung Kwon, and Jong Chul Ye. "Diffusionclip: Text-guided diffusion models for robust image manipulation." 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2022.\

I thank the authors for their detailed response. Some of my concerns are addressed. Moreover, I have read authors' discussion with other reviewers and finally decide to keep my score unchanged.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response by Authors

We thank the reviewers for their consideration of our work. Some common themes emerge across the reviews. We address these points collectively below before responding to each reviewer individually.

1. On the scale and experimental design of studying model collapse

The primary objective of this work is to study and mitigate model collapse, particularly its potential to reinforce existing inequalities through the loss of underrepresented information. Our experimental design, including the use of a relatively small diffusion model and a 90% synthetic replacement rate, is deliberately intended to induce model collapse in a controlled, systematic, and computationally efficient manner.

This approach follows established practices in the model collapse literature. Notably, Shumailov et al. (2024) [1], published in Nature, employ similarly controlled experimental settings, including relatively small models and high rates of synthetic data replacement, to study the mechanisms of collapse. Furthermore, existing theoretical and empirical work has demonstrated that model collapse can undermine neural scaling laws and is not necessarily resolved by increasing model size [2]. We therefore maintain that controlled, smaller-scale experiments are both appropriate and informative for studying the mechanisms of model collapse and evaluating potential mitigation strategies.

We acknowledge that our experimental conditions may accelerate the onset of collapse relative to some real-world settings. However, our aim is not to establish the prevalence or rate of model collapse in deployed systems, which remains an active area of debate [3,4]. Rather, we seek to understand how collapse manifests when it is induced and whether its effects can be mitigated once it occurs. We believe this distinction is important when assessing the applicability and significance of our results.

2. On the novelty and contribution of our proposed method

We believe there are important distinctions between recent inference [5] and training [6,7] time methods for guiding diffusion and our proposed approach that were insufficiently emphasised in the original manuscript. Existing approaches, such as REPA and softREPA [6,7], primarily align representations within diffusion models with those of external visual encoders, while methods such as CLIP-guided diffusion [5] employ semantic guidance during inference. In contrast, our method applies regularisation exclusively during training by re-embedding the generated image into CLIP's shared vision-language representation space and comparing it directly with the corresponding text embedding.

This distinction is important for several reasons. First, our approach leverages CLIP's existing vision-language representation rather than requiring a separately trained visual representation model. Second, by contrasting the generated image with its textual conditioning, our method explicitly prioritises semantic information expressed in the prompt, rather than alignment with visual features that need not correspond to the intended semantics. Finally, because the regulariser is applied only during training, the standard diffusion inference procedure remains unchanged.

Importantly, this text-grounded feedback mechanism is what motivates our Lewis-game formulation. The generated image serves as a signal whose intended meaning is evaluated by a semantic receiver, establishing a form of semantic cycle consistency. We therefore argue that the Lewis-game formulation is not merely a reinterpretation of existing representation-alignment methods, but describes a meaningful distinction in how information is communicated, evaluated, and preserved. Indeed, semantic controllability and interpretability are central contributions of our work. Our experiments demonstrate that the proposed method can influence which features are preserved across generations, including producing interpretable failure modes when the receiver overemphasises particular semantic attributes. We consider the ability to identify, explain, and potentially correct these behaviours through changes in textual conditioning to be an important property of the method. We nevertheless agree that stronger comparisons with existing approaches would help establish the empirical significance of these distinctions. We will expand our discussion of REPA, softREPA, and related methods, and endeavour to include additional experimental comparisons during the rebuttal period.

AuthorsOpenReview ↗

General Response by Authors 2

3. On the evaluation and interpretation of model collapse

Several reviewers raised concerns regarding our use of CLIP score to evaluate a method that directly optimises alignment within CLIP's representation space. We agree that improvements in CLIP score alone cannot provide independent evidence of successful model collapse mitigation. Indeed, this is a point we too explicitly make in the paper.

We report CLIP score to characterise the behaviour of the regularisation objective, much as one would report training loss when evaluating a learning method. However, our principal quantitative conclusions are based on FID, which is independent of the CLIP receiver used during training, alongside controlled qualitative comparisons of the generated images. More broadly, the measurement of model collapse, particularly in diffusion models, remains an open research problem. Unlike language models, where perplexity provides a useful, although debated [3], measure of distributional degradation, no similarly established metric exists for visual model collapse. While FID cannot capture every aspect of collapse, we consider it a useful and established proxy for changes in generated image distributions. We will clarify these limitations within the field and study of model collapse, and welcome suggestions for additional independent evaluation metrics.

4. On the scope and interpretation of our contribution

We also wish to clarify that our proposed method is intended to mitigate model collapse, rather than eliminate all distributional change across generations. Complete preservation of the original data distribution is neither necessarily achievable nor always desirable. Distributional change is a natural consequence of iterative learning, including in human cultural transmission [8]. Our central concern is therefore not whether all information can be preserved indefinitely, but rather which information is preserved, how that information is selected, and the extent to which its preservation can be controlled. In text-to-image diffusion models, textual conditioning provides a natural mechanism for communicating which semantic features are important. Our proposed method seeks to strengthen the preservation of those features across generations. We acknowledge that this introduces limitations, particularly when important properties of the original distribution are not captured by the text or when the model overfits to a particular semantic receiver. Take for example the clear semantic confusion which occurs from the phrase of “Hazelnut hair”. Rather than obscuring these limitations, we have deliberately examined them through controlled experiments. We consider this analysis important for understanding both the capabilities and the potential fairness implications of semantic regularisation. Our current study focuses on a single receiver to establish and systematically investigate these properties. Extensions involving multiple receivers and alternative semantic representations are promising directions for future work. We have rather chosen for a thorough evaluation of a single receiver first before moving on towards a community of receivers.

5. On the clarity and presentation of our work

Finally, we acknowledge that our original manuscript did not sufficiently communicate aspects of our methodology and motivation. In particular, we will clarify the rationale for our experimental design, the significance of the Lewis-game formulation, and the fact that our regulariser is applied only during training, leaving the standard diffusion inference procedure unchanged. We will also better distinguish semantic preservation from the complete prevention of distributional change, clarify the interpretation of our evaluation metrics, and emphasise semantic controllability and interpretability as central contributions. These revisions will make our work more accessible to the broader machine learning community, beyond researchers already familiar with model collapse and do not require significant rewriting of the paper.. We thank the reviewers for their constructive suggestions. We believe these revisions, together with the proposed additional experiments, will significantly strengthen both the technical evaluation and the presentation of our work.

AuthorsOpenReview ↗

Common Response by Authors References

[1] Shumailov, Ilia, et al. "AI models collapse when trained on recursively generated data." Nature 631.8022 (2024): 755-759.
[2] Dohmatob, Elvis, Yুনzhen Feng, and Julia Kempe. "Model collapse demystified: The case of regression." Advances in Neural Information Processing Systems 37 (2024): 46979-47013.
[3] Schaeffer, Rylan, et al. "Position: Multiple Definitions & Unrealistic Assumptions of Model Collapse Distract from Real World Threats." Forty-third International Conference on Machine Learning Position Paper Track. (2026)
[4] Jarvis, Devon, et al. "Position: the Stochastic Parrot in the Coal Mine. Model Collapse is a Threat to Low-Resource Communities." Forty-third International Conference on Machine Learning Position Paper Track. (2026).
[5] Kim, Gwanghyun, Taesung Kwon, and Jong Chul Ye. "Diffusionclip: Text-guided diffusion models for robust image manipulation." 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2022.
[6] Yu, Sihyun, et al. "Representation alignment for generation: Training diffusion transformers is easier than you think." arXiv preprint arXiv:2410.06940 (2024).
[7] Lee, Jaa-Yeon, et al. "Aligning text to image in diffusion models is easier than you think." Advances in Neural Information Processing Systems 38 (2026): 157106-157136.
[8] Kirby, Simon, Hannah Cornish, and Kenny Smith. "Cumulative cultural evolution in the laboratory: An experimental approach to the origins of structure in human language." Proceedings of the national academy of sciences 105.31 (2008): 10681-10686.

Dear Reviewers,

The authors have submitted detailed responses to your reviews.

Please review their feedback, engage with the authors if any points remain unclear, and update your evaluation and score if their response has addressed your concerns.

Thank you

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.