- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper applies quality-diversity search to a pretrained diffusion model: the genotype is the pair (initial noise, conditioning embedding), fitness is CLIP score, the behaviour space is 768-d DreamSim, and the archive is Dominated Novelty Search with an added fitness threshold that rejects invalid but novel samples. Its second contribution is a spectral mutation operator which perturbs the latent at an intermediate timestep along the top singular vectors of the Jacobian from latent to an internal hidden layer, then inverts deterministically back to the noise, with the analysis extended from the noise to the conditioning vectors. Evaluation is text-to-image only, on SDXL-Turbo with 36 prompts and FLUX.1-dev with 12, against five QD/optimizer frameworks and seven variation operators, with the headline that the method attains the lowest FID.
Critical strengths
- The framing is clean and reusable: a diffusion model as a QD landscape, with a genotype that avoids gradients through the reward and leaves the backbone untouched, so it can in principle transfer to non-image domains.
- The comparison matrix is broad for a paper of this scope — five framework baselines, seven operators, two architectures, per-operator and per-hyperparameter ablations, and significance tests with multiple-comparison correction. The diagnosis of why grid-based QD fails in 768-d is the most convincing part.
Critical weaknesses
- The FID protocol cannot support the headline. FID is computed on a selected archive pooled across 36 unrelated prompts and compared to COCO, so it is dominated by prompt coverage; FID also rewards coverage, so a method optimized for spread wins by construction; and methods with fewer than 200 samples are excluded, removing exactly the baselines that failed. "Lowest FID against 12 baselines" therefore compares against a filtered subset on a metric biased toward the proposal.
- The operator's novelty is overstated and the motivation is untested. The text says the Jacobian analysis follows Park et al. (2023) and that the extension is to conditioning vectors, yet the operator is called novel. Meanwhile the entire introduction motivates molecular and protein design while every experiment is text-to-image, so "domain-agnostic" is asserted rather than shown.
Questions for authors
- How many images, from how many prompts and seeds, enter each FID? Please report per-prompt FID or per-prompt precision and recall, which separate quality from coverage.
- What are the FID values for CVT-ME and CMA-MAE at a matched sample size? Reporting "n/a" for the failing methods while using FID as the headline is the largest threat to the conclusion.
- One spectral mutation needs a partial forward pass, power iteration and an inversion, so "2000 evaluations" equalizes NFE but not FLOPs. What is the wall-clock per iteration, and does the advantage hold under equal compute?
We thank the reviewer for the careful reading and specific questions.
W1: FID protocol
Pooling. FID needs thousands of images for a stable estimate, while each archive holds at most 64 images per prompt. Per-prompt FID, or per-prompt precision/recall, would therefore not be meaningful, and COCO provides no per-prompt reference set. We therefore pool archives across prompts and extrapolate to FID∞ (Chong and Forsyth, 2020).
Bias. FID is designed to balance fidelity and diversity; it does not reward spread for its own sake, and our method optimises neither FID nor Inception features. Spread from degraded images is penalised: on FLUX, Intermediate-layer Gaussian is slightly more diverse than ours (0.498 vs 0.489) but has a much worse FID (163.31 vs 137.58).
Excluded baselines. CVT-ME and CMA-MAE were not excluded by choice: they found too few solutions for FID (under 10 per prompt vs 58–63 for ours, Table 2), which is exactly the limitation of grid archives the analysis highlights. They are also clearly worse on the per-prompt metrics, e.g. Total Fitness 2.45 (CVT-ME) and 2.55 (CMA-MAE) vs 20.8 for ours on SDXL-Turbo (p<4×10−3). We have reworded the headline to "the lowest FID among all methods with enough solutions to compute it".
Per-prompt evidence. The paper reports per-prompt Total Fitness and diversity with paired Wilcoxon tests (Figs. 5–6), which separate quality from spread; there we Pareto-dominate the variation operators on both models, except Intermediate-layer Gaussian on FLUX (level). We now add per-prompt held-out metrics (Appendix H): DINOv2 Vendi score (effective number of distinct images) and CLIP ViT-L/14 alignment. On SDXL-Turbo our archives have a significantly higher Vendi score than every baseline (7.19 vs 3.31 for Random Seeds, p<10−7), with the same top-16 alignment as Random Seeds (0.275 vs 0.278, n.s.).
Q1: Images, prompts and seeds per FID
Each FID pools the final archives of one run (seed) per prompt. On SDXL-Turbo: ours 2,256 images from 36 prompts; Random Seeds 2,094; Gaussian and Isoline-DD 2,112; Intermediate-layer Gaussian 2,160; Sep-CMA-ES 2,212; Adam 2,304; ImageBreeder 1,920; Diverse Diffusion 2,035 (35 prompts). On FLUX: [N images from P prompts per method]. New Appendix I lists the raw FID at every N = 200, 400, …, 2000 used for the extrapolation; e.g. ours falls from 191.2 (N = 200) to 88.3 (N = 2000), giving FID∞ = 81.9.
Q2: CVT-ME and CMA-MAE at matched sample size
CMA-MAE has 298 images on SDXL-Turbo, enough only for N = 200. At this matched size all methods lie between 190 and 247, as small-sample bias dominates; CMA-MAE (201.0) remains worse than ours (191.2), Random Seeds (198.7) and Diverse Diffusion (194.9). [CVT-ME: …]. On FLUX, CMA-MAE has 71 images, far too few for a reliable estimate in the 2048-dimensional Inception space. Since a single small N cannot be extrapolated, we keep FID∞ in the main tables and report matched-N values in Appendix I.
Q3: Wall-clock per iteration and equal compute
On SDXL-Turbo, one 16-image iteration takes about 20 s for ours vs 7.3 s for simple mutations; on FLUX, each 4-image iteration adds about 106 model calls to the 120 for generation (about 1.9×). At matched end-to-end time (Appendix G.3, tables in our response to Reviewer Mdbi), our method remains significantly more diverse than Random Seeds with the same archive (0.365 vs 0.326, p=.013), with no significant fitness difference. With one spectral direction (k=1, 1.62× Random Seeds; 14 prompts), it is significantly more diverse than every baseline except CMA-MAE (e.g. 0.424 vs 0.308 for Gaussian, p=.001; all p≤.016) with no significant fitness difference against the mutation baselines (p≥.41), and stays ahead of Gaussian, Isoline-DD and Intermediate-layer Gaussian at half that budget (p≤.012).
W2: Novelty of the operator
We agree "novel" overstated the relationship to Park et al. (2023) and have revised it. Our contribution is turning their single-image editing analysis into a population mutation operator: (i) direction estimates are shared across each offspring batch; (ii) deterministic inversion keeps every genotype in xT, so the archive stays consistent; (iii) the analysis extends to conditioning embeddings, which drives the gains: the mixed genotype reaches diversity 0.473 vs 0.351 for latent-only at similar fitness (Table 7).
W2: "Domain-agnostic"
We agree our experiments establish the framework for text-to-image generation only, and have revised the abstract, introduction, method and conclusion accordingly. We now say the framework is applicable in principle beyond text: it uses no text-specific operators, its genotype is the noise and conditioning embedding any conditional diffusion model has, and its feature space can be any pretrained embedding model.