acceptodds
Withdrawn from ICLR 2027

Self-Reinforcing Creativity in Diffusion Generative Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Creativity is commonly understood as the production of content that is both novel and meaningful. While existing diffusion models can synthesize images that appear to generalize beyond the training data, their creative capacity remains limited. In this paper, we present a method for fine-tuning a diffusion model to generate highly creative image samples that extend beyond its initial output distribution. This is achieved by rewarding the model for generating images that are perceptually distant from its original outputs, using a distance function derived from the model itself. At the same time, we keep its distribution close to the pre-trained one through a KL penalty. Despite using no external supervision or reward functions, our model preserves the pre-trained model's image quality while producing highly novel samples. We further show that existing reinforcement-learning methods for diffusion-model post-training can fine-tune such models directly, as they optimize KL-regularized reward objectives of the same form. These off-the-shelf methods can therefore use the model's own novelty signal to amplify its creative capacity, making the process self-reinforcing. We demonstrate our method across settings ranging from a simple two-dimensional distribution and unconditional MNIST generation to a large-scale text-to-image model. Across these domains, the fine-tuned models produce outputs that depart from the typical generations of their pretrained counterparts while preserving coherent structure. The resulting models generate such outputs through standard sampling, eliminating the need for per-sample optimization or specialized inference-time steering.

Primary area:
generative models
Keywords:
Generative Models, Creativity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_J5joOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a post-training framework that enables diffusion models to generate more creative images. The core idea is to encourage candidate images to deviate from reference images by defining a novelty reward based on their average perceptual distance from a reference set. This training strategy facilitates self-reinforcement without requiring a separately trained reward model to evaluate creativity. Experiments on MNIST and T2I generation demonstrate that encouraging deviations from conventional data distributions can lead to more novel and creative outputs.

Critical strengths
  1. The proposed method is intuitively well motivated. Creativity often involves departing from conventional data distributions while remaining within a meaningful and coherent range. However, a similar principle has also been adopted by ConceptLab. The main distinction is that this work applies the idea directly to the post-training of diffusion models.

  2. Although training a diffusion model may initially appear computationally expensive, the proposed post-training procedure is a one-time investment. Once completed, it avoids the repeated optimization cost required for each creative generation task, as in methods such as ConceptLab.

  3. I particularly appreciate the experiments on MNIST. Despite their simplicity, they provide an intuitive and compelling illustration of the core idea. This is a notable strength compared with prior methods that are evaluated exclusively on T2I generation.

Critical weaknesses
  1. During training, the proposed method uses CLIPScore and PickScore as reward signals, thus training procedure is not fully self-supervised. Since none of the baseline methods appear to use these additional supervision signals, I am uncertain whether the improvements in VQAScore reported in Table 2 reflect a fair comparison. I would also appreciate clarification on why VQAScore is considered an appropriate metric for evaluating image creativity. '0.002, N/A' in Table 2 appears to be a typo.

  2. Although Table 2 reports computational costs, the cost reported for the proposed method appears to include only online inference, while omitting the offline cost of post-training the diffusion model. I expect this offline training cost to account for a substantial proportion of the total computational expense. It would therefore be helpful to report the complete training cost and provide a break-even analysis: how many batches of images must be generated before the one-time training cost is offset by the reduction in per-generation overhead? For example, if a user wishes to generate creative images for only a single concept, conducting large-scale post-training may be less flexible or cost-effective than an optimization-based method such as ConceptLab.

  3. To the best of my understanding, the proposed method relies on relatively detailed prompt templates to guide creative generation, whereas many related methods use substantially more generic prompts, such as 'a creative {subject}'. If such detailed prompts are already required, it is unclear why users could not simply describe the desired creative concept directly in natural language. In my preliminary experiments with Stable Diffusion 3.5 and FLUX, natural-language descriptions alone were sufficient to produce images that departed substantially from conventional representations and exhibited forms of creativity similar to those shown in the paper. GPT-5.6 produced even stronger results. This raises questions about how much of the observed improvement should be attributed to the proposed post-training method rather than to prompt engineering or the capabilities of the underlying model. I am also curious about the method's sensitivity to the prompt template at test time. How does performance change when a different template is used?

  4. Regarding the qualitative evaluation, why are only 20 object categories considered, and what criteria were used to select them? In my view, this number is too small to support robust conclusions about the method's qualitative performance. Moreover, the paper does not present any failure cases, making it difficult to assess the method’s limitations and generalization ability. For example, after post-training, can the model generate creative images across arbitrary domains, such as 'a creative cat' or 'a creative car'? The range of domains demonstrated in the paper remains limited, making it difficult to determine whether the benefits of the proposed method justify its training cost.

Questions for authors

See Weaknesses

1 comment

We thank the reviewer for highlighting the intuitive motivation, the value of amortizing post-training across future generations, and the MNIST experiments as a clear illustration of the central idea.

Training signals and metrics - CLIPScore and PickScore are used only during the initial guidance-distillation stage. The resulting SD3.5 checkpoint serves as both the initialization for creativity training and a matched baseline, so this comparison isolates the effect of our model-derived creativity objective. VQAScore measures category validity rather than creativity; novelty is measured by KID. We also confirm that “0.002, N/A” in Table 1 is a typo and should read “0.002 | N/A.”

Efficiency - Table 1 reports online generation and per-concept adaptation costs but omits our one-time post-training cost. Our intended setting is a reusable model serving many prompts and samples, over which this cost is amortized. For a single concept, a training-free or per-concept method may be faster.

Prompting - Our method and the matched SD3.5 baseline share the same underlying pretrained model and prompt template, ensuring that the observed performance differences cannot be attributed to differences in the base model's capabilities or prompt formulation. Figure 9 also uses the same neutral prompt and seed, without creativity-inducing words, and our model still produces the less conventional interpretation. Detailed instructions specifying what should vary require the user to anticipate the novel outcome in advance. Our method allows the generation of surprising interpretations that the user would not think to request.

Evaluation - The 20 categories were selected randomly, and no evaluation prompt appeared in the fine-tuning set. This exceeds the four categories used by VLM-Guided and five used by C3 in their quantitative evaluations. Because we fine-tune a shared generator rather than optimize a concept-specific representation, the same model remains applicable to free-form prompts, including categories such as cats or cars.

Reviewer_Jf9TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a fine-tuning framework for creative visual generation that rewards perceptual distance from samples produced by a pretrained diffusion model while using a KL-divergence penalty to encourage preservation of image quality and coherence. The framework adopts the model-derived Information-Estimation Metric (IEM) as its novelty reward and trains a shared model across multiple prompts, enabling generation without per-sample optimization.

Critical strengths
  • The paper provides a clear formulation of two complementary goals in creative generation. It explicitly connects novelty to departure from the pretrained model’s typical outputs and seeks to preserve meaningfulness through distributional regularization. This provides an understandable formulation of the tension between generating unfamiliar content and maintaining coherence.
  • The overall methodology is straightforward and well presented. The progression from reference sampling to reward computation and KL-regularized fine-tuning is easy to follow. The ability to reuse the fine-tuned model across prompts without additional per-sample optimization is also practically appealing.
Critical weaknesses
  • The evaluation does not sufficiently disentangle conceptual novelty from aesthetic and stylistic changes. The additional examples and human study provide useful evidence of perceived creativity, but the study does not separately assess conceptual originality, visual appeal, and rendering style. Differences in prompt composition across methods further complicate this interpretation, motivating more controlled comparisons.
  • The rationale and coverage of the concept-level evaluation remain insufficiently justified. Appendix A.3 lists the 20 evaluation categories and Appendix C describes the training-prompt sources, but the category-selection criteria and training–evaluation concept overlap are not established. Moreover, the novelty evaluation is restricted to Building, Hat, and Lamp; broader category-level results and an explicitly concept-disjoint evaluation would strengthen the evidence for generalizable creative generation.
  • The justification for the IEM-based reward and its optimization choices remains insufficient. While deriving the novelty metric from the pretrained model is intuitively reasonable, it is unclear why maximizing IEM should necessarily encourage meaningful conceptual novelty rather than stylistic or other perceptual deviations. Furthermore, several important design choices, including the modified IEM weighting (λ1=2λ2\lambda_1=2\lambda_2), the novelty reward scaling, and the fine-tuning duration, are not sufficiently justified. Although the paper provides some related ablations, these choices are not systematically isolated, making it difficult to assess their individual contributions and the robustness of the proposed novelty–fidelity trade-off.
Questions for authors
  • Can the authors evaluate conceptual novelty separately from aesthetic quality and rendering style, for example through style-controlled comparisons and human judgments that distinguish these factors?
  • What motivated the selection of the 20 evaluation categories and the three categories used for KID, and how do these concepts overlap with the fine-tuning prompts?
  • Can the authors compare equal and unequal IEM-term weighting while keeping the optimizer, noise range, reward-estimation settings, and inference configuration fixed?
  • Beyond the aggregate scores in Table 1, can the authors quantify the frequency of the two baseline limitations discussed in Section 4.2—loss of category recognizability and insufficient departure from familiar forms—across more prompts and samples, with uncertainty estimates?
  • How were the fine-tuning duration (12 epochs for guidance distillation and 17 epochs for creativity fine-tuning) and the novelty-related weighting coefficients selected? In particular, how do these choices affect the trade-off between novelty and meaningfulness?
1 comment

We thank the reviewer for recognizing the clear formulation of novelty and meaningfulness, the straightforward presentation, and the practical advantage of reusing one fine-tuned model across prompts without per-sample optimization.

Evaluation scope - Our aim is to evaluate creative generation as the combination of novelty and meaningfulness, rather than isolate conceptual novelty, rendering style, and aesthetics as independent effects. Accordingly, VQAScore evaluates category recognizability and validity, while KID measures departure from familiar forms. In the user study, participants were instructed to base their judgments only on the main object, ignoring differences in background and composition.

Category selection - The 20 evaluation categories were selected randomly, and no evaluation prompt appeared in the fine-tuning set. Our evaluation also covers more categories than the principal protocols used by closely related creative-generation methods.

Interpretation of the quantitative evaluation - The two baseline failure modes discussed in Section 4.2 are reflected by complementary metrics: VQAScore measures whether the requested category remains recognizable, while KID measures departure from familiar forms. The human study provides a complementary perceptual evaluation.

Prompt protocol - Our model and the matched SD3.5 baseline use the same prompt template. For competing methods, we use the prompt formulation that works best for each, because applying our template substantially degraded their outputs. Thus, the SD3.5 comparison uses identical prompts and isolates the effect of our method, while the cross-method comparisons evaluate each method with its best-performing prompt.

Reviewer_M94vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies creative image generation while avoiding excessive deviation from the original sample domain. The core idea is to encourage novelty by increasing the distance from reference images, while using KL divergence to keep the generated distribution close to the pretrained distribution. A key consideration is how to measure the distance between generated and reference images. The paper adopts the recently proposed Information-Estimation Metric (IEM).

Critical strengths
  1. The mathematical formulation is clear and interpretable. In addition to natural-image concepts, the paper demonstrates its effects on settings such as MNIST that are easier to analyze, making the proposed approach more convincing. Although the formulation appears fairly standard, the reviewer found it surprisingly difficult to identify prior creativity-oriented fine-tuning methods that use precisely this objective.
  2. By directly fine-tuning the model, the approach achieves substantially faster inference than methods that search for or guide creative outputs at inference time.
Critical weaknesses
  1. The methodological novelty appears limited, as several components overlap substantially with existing approaches. Reward maximization with KL regularization is widely used in RL-based post-training. Distance-based signals have also been used to promote creative or diverse image generation, for example through inference-time guidance in ProCreate and diversity-aware training in DiverseGRPO. Furthermore, IEM is adopted from prior work rather than introduced in this paper.

  2. It is unclear whether distance in an image representation space sufficiently captures creativity. Being farther from reference images does not necessarily imply being more creative. For example, the MNIST results appear to populate sparse regions of the manifold, raising the question of whether the effect is similar to interpolation.

Questions for authors
  1. Related to the concern in weakness (2), the generated outputs appear quite similar to one another. For example, in Figure 11, the proposed method produces images with similar color palettes, and their shapes appear less diverse than those generated by VLM-Guided. These results give the impression of interpolation. How different are the outputs from those obtained through latent-space interpolation?

  2. Although the prompts include creativity-related words, they seem relatively simple. Could the authors compare against more explicit prompting strategies, such as asking the base model to generate objects with novel shapes and color combinations while preserving the defining characteristics of the original domain?

  3. Although the task differs, unlearning methods such as Direct Unlearning Optimization use examples to be avoided in an optimization framework related to preference learning. How does the proposed approach differ from treating the reference images as negative examples and fine-tuning the model to discourage their generation?

1 comment

We thank the reviewer for highlighting the clear formulation, informative MNIST analysis, and benefits of directly fine-tuning a reusable model with fast inference. We also appreciate the observation that prior creativity-oriented fine-tuning work using this precise objective is difficult to identify.

Methodological novelty - Our contribution lies not in the individual components, but in their novel integration: we combine IEM-based novelty estimation with KL-regularized post-training to directly optimize a generative model for creativity.

Distance and creativity - Our hypothesis is that a sample novel with respect to the pretrained model’s distribution should be far, under a perceptually meaningful metric, from samples drawn from that distribution. Expected distance therefore quantifies model-relative novelty, one component of creativity.

Latent interpolation - In our interpolation experiments, interpolating between latent endpoints produced blurred intermediate samples, compromising object recognizability and meaningfulness.

Diversity - Our Vendi score is 3.032 versus 2.821 for SD3.5, and Figure 11 shows greater variation in the generated objects. Higher scores from some competing methods partly reflect unconstrained backgrounds and compositions, as well as departures that compromise category recognizability and meaningfulness.

Explicit prompting - Explicitly requesting novel shapes, colors, or other attributes already supplies the creative direction and requires users to anticipate the outcome. Our method instead allows surprising interpretations that users would not think to request.

Unlearning - We agree that unlearning-style optimization is an interesting direction for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.