Mitigating Model Collapse in Latent Diffusion Models using a Lewis Signaling Regulariser

Model Collapse, a degenerative process in which models forget the underlying data distribution when trained on sequences of synthetic data, is a looming problem as generated content proliferates on the internet. This stands to not only undermine the general utility of the models but also introduces mechanisms that lead to systemic bias. In this work, we aim to mitigate the collapse by introducing a novel regularisation method that frames latent diffusion as a Lewis Signaling Game. In this framing, the diffusion model is a “sender” that transforms a text embedding, the “state”, into an image which acts as the “signal”. This signal must then be processed by a “receiver”, a pretrained CLIP vision encoder whose goal is to retrieve the original text embedding. This provides a communication pressure which forces the diffusion model to remain sufficiently expressive. Our results show a clear improvement and mitigation of model collapse over multiple generations. Further, we find that the specificity of text prompts significantly impacts our method’s utility. Characteristics of the image described in the text tend toward a minimal set of stereotypical motifs. As such, precise, detailed prompts help encourage generations in the tail of the distribution, helping to prevent collapse along those axes. Finally, we show how this method takes a step towards avoiding bias in latent diffusion, such as the lightening of skin tones. Overall, this method supports the notion that model collapse results from ungrounded content generation and demonstrates that explicitly modeling a receiver or verification step in the training pipeline helps prevent collapse.