acceptodds
Under review as a conference paper at ICLR 2027

Uncovering the Generative Landscape of Diffusion Models: A Spectral-Guided Quality-Diversity Diffusion Framework

Abstract

Denoising Diffusion Probabilistic Models have emerged as powerful generative priors across domains ranging from molecular design to high-resolution image synthesis. However, standard diffusion sampling is inherently biased toward high-likelihood regions, often limiting the diversity of outputs. In many domains, a single, high-likelihood sample is insufficient. Instead, diverse portfolios of high-performing solutions are required, motivating methods that uncover a model's full expressive range. Here we show that integrating Quality-Diversity (QD) optimisation with diffusion models enables the discovery of diverse, high-performing outputs. We first propose Quality-Diversity Diffusion, a framework that adapts QD to diffusion models by directly optimising the model's latent noise and conditioning embeddings and leveraging embedding models to quantify diversity. Second, we introduce a novel h-space Spectral Mutation operator, which guides search along directions of maximal semantic sensitivity by analysing principal singular vectors of the model's input-to-activation Jacobian. We evaluate our combined approach, Spectral-Guided Quality-Diversity Diffusion, on text-to-image generation. Against baselines, our method consistently achieves the lowest FID score, demonstrating an excellent balance of image quality and diversity.

Primary area:
generative models
Keywords:
Diffusion, Quality-Diversity, Generative Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5PHvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper applies quality-diversity search to a pretrained diffusion model: the genotype is the pair (initial noise, conditioning embedding), fitness is CLIP score, the behaviour space is 768-d DreamSim, and the archive is Dominated Novelty Search with an added fitness threshold that rejects invalid but novel samples. Its second contribution is a spectral mutation operator which perturbs the latent at an intermediate timestep along the top singular vectors of the Jacobian from latent to an internal hidden layer, then inverts deterministically back to the noise, with the analysis extended from the noise to the conditioning vectors. Evaluation is text-to-image only, on SDXL-Turbo with 36 prompts and FLUX.1-dev with 12, against five QD/optimizer frameworks and seven variation operators, with the headline that the method attains the lowest FID.

Critical strengths
  • The framing is clean and reusable: a diffusion model as a QD landscape, with a genotype that avoids gradients through the reward and leaves the backbone untouched, so it can in principle transfer to non-image domains.
  • The comparison matrix is broad for a paper of this scope — five framework baselines, seven operators, two architectures, per-operator and per-hyperparameter ablations, and significance tests with multiple-comparison correction. The diagnosis of why grid-based QD fails in 768-d is the most convincing part.
Critical weaknesses
  • The FID protocol cannot support the headline. FID is computed on a selected archive pooled across 36 unrelated prompts and compared to COCO, so it is dominated by prompt coverage; FID also rewards coverage, so a method optimized for spread wins by construction; and methods with fewer than 200 samples are excluded, removing exactly the baselines that failed. "Lowest FID against 12 baselines" therefore compares against a filtered subset on a metric biased toward the proposal.
  • The operator's novelty is overstated and the motivation is untested. The text says the Jacobian analysis follows Park et al. (2023) and that the extension is to conditioning vectors, yet the operator is called novel. Meanwhile the entire introduction motivates molecular and protein design while every experiment is text-to-image, so "domain-agnostic" is asserted rather than shown.
Questions for authors
  1. How many images, from how many prompts and seeds, enter each FID? Please report per-prompt FID or per-prompt precision and recall, which separate quality from coverage.
  2. What are the FID values for CVT-ME and CMA-MAE at a matched sample size? Reporting "n/a" for the failing methods while using FID as the headline is the largest threat to the conclusion.
  3. One spectral mutation needs a partial forward pass, power iteration and an inversion, so "2000 evaluations" equalizes NFE but not FLOPs. What is the wall-clock per iteration, and does the advantage hold under equal compute?
1 comment

We thank the reviewer for the careful reading and specific questions.

W1: FID protocol

Pooling. FID needs thousands of images for a stable estimate, while each archive holds at most 64 images per prompt. Per-prompt FID, or per-prompt precision/recall, would therefore not be meaningful, and COCO provides no per-prompt reference set. We therefore pool archives across prompts and extrapolate to FID∞ (Chong and Forsyth, 2020).

Bias. FID is designed to balance fidelity and diversity; it does not reward spread for its own sake, and our method optimises neither FID nor Inception features. Spread from degraded images is penalised: on FLUX, Intermediate-layer Gaussian is slightly more diverse than ours (0.498 vs 0.489) but has a much worse FID (163.31 vs 137.58).

Excluded baselines. CVT-ME and CMA-MAE were not excluded by choice: they found too few solutions for FID (under 10 per prompt vs 58–63 for ours, Table 2), which is exactly the limitation of grid archives the analysis highlights. They are also clearly worse on the per-prompt metrics, e.g. Total Fitness 2.45 (CVT-ME) and 2.55 (CMA-MAE) vs 20.8 for ours on SDXL-Turbo (p<4×10−3p<4\times10^{-3}). We have reworded the headline to "the lowest FID among all methods with enough solutions to compute it".

Per-prompt evidence. The paper reports per-prompt Total Fitness and diversity with paired Wilcoxon tests (Figs. 5–6), which separate quality from spread; there we Pareto-dominate the variation operators on both models, except Intermediate-layer Gaussian on FLUX (level). We now add per-prompt held-out metrics (Appendix H): DINOv2 Vendi score (effective number of distinct images) and CLIP ViT-L/14 alignment. On SDXL-Turbo our archives have a significantly higher Vendi score than every baseline (7.19 vs 3.31 for Random Seeds, p<10−7p<10^{-7}), with the same top-16 alignment as Random Seeds (0.275 vs 0.278, n.s.).

Q1: Images, prompts and seeds per FID

Each FID pools the final archives of one run (seed) per prompt. On SDXL-Turbo: ours 2,256 images from 36 prompts; Random Seeds 2,094; Gaussian and Isoline-DD 2,112; Intermediate-layer Gaussian 2,160; Sep-CMA-ES 2,212; Adam 2,304; ImageBreeder 1,920; Diverse Diffusion 2,035 (35 prompts). On FLUX: [N images from P prompts per method]. New Appendix I lists the raw FID at every N = 200, 400, …, 2000 used for the extrapolation; e.g. ours falls from 191.2 (N = 200) to 88.3 (N = 2000), giving FID∞ = 81.9.

Q2: CVT-ME and CMA-MAE at matched sample size

CMA-MAE has 298 images on SDXL-Turbo, enough only for N = 200. At this matched size all methods lie between 190 and 247, as small-sample bias dominates; CMA-MAE (201.0) remains worse than ours (191.2), Random Seeds (198.7) and Diverse Diffusion (194.9). [CVT-ME: …]. On FLUX, CMA-MAE has 71 images, far too few for a reliable estimate in the 2048-dimensional Inception space. Since a single small N cannot be extrapolated, we keep FID∞ in the main tables and report matched-N values in Appendix I.

Q3: Wall-clock per iteration and equal compute

On SDXL-Turbo, one 16-image iteration takes about 20 s for ours vs 7.3 s for simple mutations; on FLUX, each 4-image iteration adds about 106 model calls to the 120 for generation (about 1.9×). At matched end-to-end time (Appendix G.3, tables in our response to Reviewer Mdbi), our method remains significantly more diverse than Random Seeds with the same archive (0.365 vs 0.326, p=.013), with no significant fitness difference. With one spectral direction (k=1k=1, 1.62× Random Seeds; 14 prompts), it is significantly more diverse than every baseline except CMA-MAE (e.g. 0.424 vs 0.308 for Gaussian, p=.001; all p≤.016) with no significant fitness difference against the mutation baselines (p≥.41), and stays ahead of Gaussian, Isoline-DD and Intermediate-layer Gaussian at half that budget (p≤.012).

W2: Novelty of the operator

We agree "novel" overstated the relationship to Park et al. (2023) and have revised it. Our contribution is turning their single-image editing analysis into a population mutation operator: (i) direction estimates are shared across each offspring batch; (ii) deterministic inversion keeps every genotype in xTx_T, so the archive stays consistent; (iii) the analysis extends to conditioning embeddings, which drives the gains: the mixed genotype reaches diversity 0.473 vs 0.351 for latent-only at similar fitness (Table 7).

W2: "Domain-agnostic"

We agree our experiments establish the framework for text-to-image generation only, and have revised the abstract, introduction, method and conclusion accordingly. We now say the framework is applicable in principle beyond text: it uses no text-specific operators, its genotype is the noise and conditioning embedding any conditional diffusion model has, and its feature space can be any pretrained embedding model.

Reviewer_MdbiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Spectral-Guided Quality-Diversity Diffusion, a framework for exploring diverse and high-quality outputs from pretrained diffusion models. The method integrates Quality-Diversity optimization with diffusion sampling by searching over latent noise and conditioning embeddings, using pretrained visual embeddings to measure diversity. Its main technical contribution is an h-space Spectral Mutation operator that identifies directions of high semantic sensitivity through Jacobian-based spectral analysis. Experiments on SDXL-Turbo and FLUX.1 compare the framework with several optimization and diversity-oriented baselines, reporting improvements in image quality and diversity.

Critical strengths
  • The paper addresses a meaningful problem in generative modeling by explicitly exploring the quality-diversity trade-off rather than focusing solely on generating high-likelihood samples. Integrating Quality-Diversity optimization with pretrained diffusion models provides an interesting direction for discovering diverse outputs.
  • The proposed h-space Spectral Mutation operator offers a principled way to explore semantically sensitive directions in diffusion models. The evaluation includes multiple optimization and mutation baselines, and the experiments on both SDXL-Turbo and FLUX suggest that the framework can be applied across different architectures.
Critical weaknesses
  • The experimental comparisons raise concerns about fairness and the practical significance of the proposed framework. In Fig. 10, several baselines contain missing or empty results due to computational constraints or limited archive coverage. Presenting these outputs in the same Top-16 comparison makes the visual advantage difficult to assess fairly, especially when some baselines use different archive mechanisms. Moreover, the paper does not compare against simpler alternatives, such as LLM-based prompt diversification followed by standard diffusion sampling, leaving unclear whether the proposed framework provides sufficient additional benefit over more practical approaches to generating diverse images.
  • The computational cost is extremely high relative to the demonstrated benefits. The Jacobian-based spectral mutation introduces substantial overhead, particularly for FLUX, where the reported runtime is orders of magnitude higher than that of simpler mutation and sampling methods. Although the framework improves diversity-related metrics, it is unclear whether these gains justify such a computationally expensive pipeline. The practical benefit of the approach in the evaluated text-to-image setting therefore remains unclear under a matched wall-clock budget.
Questions for authors
  • Could the authors explicitly distinguish baselines that were not evaluated from those with limited archive coverage in Fig. 10 and provide a more comparable qualitative evaluation? In particular, how does the proposed method perform when baselines are evaluated using a common postprocessing and presentation protocol for generated candidates, in addition to the existing archive-based evaluation?
  • Have the authors considered simpler diversity-generation alternatives, such as using an LLM to produce semantically varied prompts followed by standard diffusion sampling? A comparison under matched computational budgets would help establish the practical benefit of the proposed framework.
  • Given the substantial runtime overhead, particularly on FLUX, can the authors demonstrate whether the diversity gains remain meaningful when compared against simpler methods under the same wall-clock budget? What practical applications would justify this additional computational cost?
1 comment

We thank the reviewer for their careful assessment.

W1 / Q1: Figure 10

We agree that methods that were not evaluated should not appear, so we removed the Adam and QDHF panels from the FLUX figure. For every other method the visualisation is identical: we display the 16 highest-fitness solutions of its final archive. Black squares mean the archive holds under 16 solutions, as the caption now states. This is a result the figure is meant to show: grid-based archives struggle to retain solutions in high-dimensional feature spaces, keeping under 10 solutions per prompt vs 58–63 for ours (Table 2). The comparison already uses a common protocol wherever the archive is not under study: all non-QD and variation-operator baselines pass every candidate through our DNS archive with fitness thresholding (Appendix C.4), so their grids differ only in the candidates. Only the QD framework baselines keep their own archives, as the archive is what that comparison evaluates. On held-out metrics over each method's 16 best images (Appendix H), our SDXL-Turbo portfolios are significantly more diverse than every baseline except CMA-MAE, at the alignment of Random Seeds and the noise operators.

W1 / Q2: LLM prompt diversification

We considered this, but felt it was outside this paper's scope. Our framework searches over the initial noise and conditioning embeddings, so it does not rely on language priors and applies where conditioning is not text. LLM-based variation could be valuable here, and we discuss LLM prompt-evolution methods in our related work (Inch et al., 2025; Simon, 2025), but such priors would not transfer to domains without a language interface. We therefore focused on our two contributions, the framework and the mutation operator, against 12 baselines. The approaches are complementary: our search could run on each LLM-rewritten prompt.

W2 / Q3: Computational cost at matched wall-clock

Thank you for pressing on this: Table 12 overstated our cost, as its FLUX column gave per-run rather than per-image times (now corrected). End to end, our method costs 1.24 s vs 0.46 s per image for Random Seeds on SDXL-Turbo (2.70×); on FLUX each 4-image iteration adds about 106 model calls to the 120 for generation (about 1.9×). This is a constant factor, not orders of magnitude.

We added a matched end-to-end time comparison (Appendix G.3): our run stops once its cumulative wall-clock time equals each baseline's full run (ours / baseline, bold = higher):

SDXL Div.SDXL Fit.FLUX Div.FLUX Fit.
Random Seeds.365 / .32620.02 / 19.08.451 / .31818.63 / 18.72
Diverse Diffusion.358 / .32719.61 / 19.10.411 / .29818.40 / 18.77
Gaussian.366 / .32820.01 / 19.55.392 / .26318.40 / 17.69
Isoline-DD.365 / .34620.02 / 19.46.392 / .32818.39 / 17.59
Interm. Gaussian.365 / .34119.96 / 19.59.455 / .47418.81 / 19.73
Particle Guidance––.472 / .32619.53 / 16.72
Sep-CMA-ES.366 / .19920.06 / 20.32.406 / .09518.42 / 18.72
ImageBreeder.369 / .02419.93 / 17.42.408 / .03118.41 / 18.95
CMA-MAE.316 / .49419.42 / 2.55.430 / .57118.78 / 1.74

On SDXL-Turbo we are more diverse than every baseline except CMA-MAE, which keeps about 8 solutions per prompt and is far less fit (p<.001). Diversity gains are significant against Random Seeds (p=.013), Sep-CMA-ES, ImageBreeder and Adam (p<.001), and level against the other operators (p≥.061). Fitness differences are not significant except that Gaussian and Isoline-DD score slightly higher on most prompts (p=.009, .006) and the fitness-only optimisers are fitter at far lower diversity. On FLUX, diversity gains are significant against Random Seeds (p=.034), Diverse Diffusion (p=.021), Particle Guidance (p=.007), Sep-CMA-ES and ImageBreeder (p=.004); no fitness difference is significant except CMA-MAE (p=.009). Looking at the samples, much of the remaining baselines' diversity comes from corrupted or degraded images (Sections 5.1–5.2, App. E.2); on held-out DINOv2 features, Gaussian and Isoline-DD archives are far less varied (Vendi 4.23, 4.76 vs 7.19).

The cost is also adjustable: with one spectral direction (k=1, 1.62× Random Seeds), our method is significantly more diverse than every baseline except CMA-MAE at matched time, with no significant fitness difference (see response to Reviewer 5PHv).

Practical applications. The approach suits settings where evaluating a candidate costs far more than generating it, such as scientific design (an assay or simulation can take hours); there a more diverse portfolio outweighs the extra generation cost, and our default k=3 gives the best results per evaluation. For creative use, a full SDXL-Turbo search takes about 40 min per prompt on one GPU.

Reviewer_BMygOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Spectral-Guided Quality-Diversity Diffusion, an inference-time evolutionary search framework that constructs prompt-specific image portfolios by optimizing latent noise and, where enabled, conditioning embeddings of a pretrained diffusion model. It combines a fitness-thresholded Dominated Novelty Search archive with a Jacobian-based spectral mutation operator and evaluates the resulting quality–diversity trade-off on SDXL-Turbo and FLUX.1.

Critical strengths
  • The framework provides a coherent approach to searching for diverse, prompt-aligned image portfolios. Separating candidate mutation from archive selection makes the method understandable, while fitness thresholding addresses the practical problem of retaining highly novel but semantically inappropriate outputs.
  • The experiments include useful comparisons and component analyses. The paper separately evaluates archive mechanisms and mutation operators, includes random sampling with archive-based selection under matched candidate budgets, and provides ablations of several important design choices. The reported results support a promising quality–diversity trade-off within the evaluated settings.
Critical weaknesses
  • The practical advantage under matched computational resources remains unclear. Although the experiments match the number of candidate evaluations and include a random-sampling-and-selection baseline, spectral estimation and intermediate-state inversion introduce substantial additional computation. Appendix G acknowledges this overhead, but the paper does not establish whether the method remains advantageous when simpler alternatives receive the same end-to-end runtime budget. A comparison against a larger random candidate pool followed by the same archive selection, together with quality–diversity-versus-time curves, is important for assessing the practical value of the approach.
  • The specific semantic benefit of spectral mutation warrants stronger validation. The framework builds on established archive and latent-geometry techniques, making the contribution of the spectral operator particularly important. Maximizing local hidden-activation sensitivity does not by itself establish that the resulting changes are semantically meaningful or preserve the requested content. While the operator comparisons and supplementary ablations are useful, matched-perturbation analyses and extensions of the key component ablations to the multi-step FLUX setting would better establish when and why this operator is preferable to simpler mutations.
  • The quality–diversity conclusions rely heavily on the proxies used during search. CLIP-based fitness and DreamSim-based distances are central to both optimization and evaluation, while the additional FID analysis compares pooled generations with a COCO reference rather than directly assessing diversity within each prompt. This leaves uncertainty about whether the measured improvements consistently correspond to meaningful visual variation without losing fine-grained prompt constraints, particularly when conditioning embeddings are modified. Independent human or held-out evaluator assessments, preferably on fixed-size portfolios, would strengthen the conclusions.
Questions for authors
  • What exactly is included in the timing measurements in Table 12: mutation alone, candidate generation, or the complete generation-and-evaluation loop? Does the method retain its advantage over random sampling with the same archive selection when end-to-end GPU time is matched?
  • Could the authors extend the existing component ablations to FLUX and compare spectral and nonspectral mutations at matched perturbation magnitudes and computational budgets? Such an analysis would help distinguish the benefit of spectral directions from that of intermediate-state editing.
  • Can the quality and diversity improvements be validated independently of the CLIP and DreamSim objectives, while checking adherence to the original prompt and controlling the number of images in each evaluated portfolio?
2 comments

We thank the reviewer for a detailed and constructive review.

Q1: What is included in Table 12?

Thank you for the question, and for catching this. Table 12 measures the mutation step alone, excluding image generation, scoring and archive updates. Some entries were accidentally entered incorrectly: the FLUX column reported the total mutation time of a 400-image run rather than per image. The table now shows the median mutation time per image for every method on both models (e.g. FLUX: ours 37.4 s, Particle Guidance 46.9 s, Intermediate-layer Gaussian 14.5 s). This changes the FLUX entries but not the conclusions of Appendix G: our operator remains more expensive than additive-noise operators, as it computes Jacobian-based directions and, on multi-step models, rolls the parent forward and inverts it. The overhead is a constant factor per image rather than the orders-of-magnitude gap the original table suggested, and is shared by other methods that steer sampling, such as Particle Guidance.

W1 / Q1: Matched end-to-end time

We agree that a comparison at matched compute is a useful complement to the matched-evaluation results in the paper. We carried out a full matched end-to-end time analysis on SDXL-Turbo and FLUX (new Appendix G.3; tables in our response to Reviewer Mdbi). On "Does the method retain its advantage over random sampling with the same archive selection when end-to-end GPU time is matched?": this is precisely our Random Seeds baseline, which samples fresh seeds but passes every candidate through our DNS archive with fitness thresholding (Appendix C.4). At matched end-to-end time on SDXL-Turbo, our method remains significantly more diverse (0.365 vs 0.326, p=.013) with no significant difference in Total Fitness (p=.56), and the margin widens with a single spectral direction, k=1 (0.425 vs 0.330, p=.016).

W2 / Q2: Spectral directions vs intermediate-state editing

Our Intermediate-layer Gaussian baseline isolates exactly this: it applies the same intermediate-state edit and inversion, but with Gaussian noise instead of spectral directions. On the paper's metrics, spectral directions are clearly better on SDXL-Turbo: higher Total Fitness (20.78 vs 19.59, p<5×10−5p<5\times10^{-5}), higher diversity (0.473 vs 0.341, p<3×10−10p<3\times10^{-10}) and lower FID (81.90 vs 109.73). On FLUX the two are level on Total Fitness and diversity (p>0.7p>0.7), while spectral directions give a substantially lower FID (137.58 vs 163.31). Much of the FLUX diversity gain therefore comes from intermediate-state editing, which deterministic inversion makes possible, while the Intermediate-layer Gaussian's diversity comes largely from corrupted images (Appendix E.2). The operator is also robust to its step size: α\alpha was not tuned for FLUX, and Table 8 shows stability across a 4× range. On the held-out metrics below, spectral directions also give more diverse SDXL-Turbo archives (DINOv2 Vendi score 7.19 vs 4.24, higher on all 35 prompts) with no loss of prompt alignment among the 16 best images (CLIP ViT-L/14 0.275 vs 0.278, n.s.). We will add a FLUX ablation of spectral vs random directions at matched norm, with the kk and α\alpha ablations, to the final version.

W3 / Q3: Validation independent of CLIP and DreamSim

We now additionally evaluate every archive with models the search never saw (new Appendix H). For diversity, we compute the Vendi score (Friedman and Dieng, 2023) on DINOv2 features: the exponential of the entropy of the eigenvalues of the images' similarity matrix, i.e. the effective number of distinct images. For prompt alignment, we compute the CLIP score with CLIP ViT-L/14, a larger model than the ViT-B/32 used as fitness: the cosine similarity between each image and the prompt. Metrics are per prompt, with paired Wilcoxon tests (Holm-corrected), on whole archives and on each method's 16 best images.

  • SDXL-Turbo: archive Vendi score is significantly higher than every baseline (7.19 vs 3.31 for Random Seeds, all p<10−7p<10^{-7}), and our 16-image portfolios match Random Seeds, Diverse Diffusion, Gaussian, Isoline-DD and Intermediate-layer Gaussian on alignment (e.g. 0.275 vs 0.278 for Random Seeds, n.s.).
  • FLUX: Vendi score is significantly higher than Random Seeds, Diverse Diffusion, Particle Guidance, Gaussian, Sep-CMA-ES and ImageBreeder (6.07 vs 3.63 for Random Seeds, p=.015), again with matched portfolio alignment.
  • Adherence when conditioning embeddings are modified (11 of every 16 offspring): taking each method's 16 most on-prompt images, alignment matches Random Seeds on both models (0.291 vs 0.291; 0.277 vs 0.275), while Vendi score remains higher (4.06 vs 2.56, p<10−7p<10^{-7}; 3.65 vs 2.46, p=.024). This holds even within the top quartile of alignment (3.39 vs 2.26, p=.003; 3.45 vs 2.25, p=.029).

A human preference study would be a valuable complement, but is outside the scope of this paper.

I thank the authors for their detailed response and additional experiments. While the new results address W1 and W3, my main concern regarding the specific contribution of spectral mutation (W2) remains insufficiently resolved, particularly in the FLUX setting. Therefore, I will maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.