acceptodds
Under review as a conference paper at ICLR 2027

How Many Images Are Enough to Compare Diversity? Adaptive Sample Sizes for Text-to-Image Evaluation

Abstract

Diversity comparisons between text-to-image generators are often computed from a fixed number of images per prompt, yet there is no principled rule for choosing . We study this question for the Vendi score, whose value keeps changing as more images are generated per prompt. However, a comparison between two generators need not be equally unstable: because the two scores tend to change by similar factors, their ratio can stabilize while each score is still changing. The number of images required for this stabilization differs widely across guidance settings, diversity interventions and model architectures. To address these issues, we introduce DivShift, a prospective procedure that determines the required number of images separately for each Vendi-based comparison. Given a user-specified reference size and a tolerance , DivShift tracks the observed Vendi log-ratio, predicts how much the ratio may still change before , accounts for variation across prompts, and either stops or requests additional images. At and , the five-panel macro-average shows that DivShift saves of generations over evaluation at , while only of stopping decisions remain outside the target tolerance. DivShift therefore provides a practical protocol for determining when a Vendi-based generator comparison has stabilized sufficiently to stop sampling, avoiding both premature stopping and unnecessary generations.

Primary area:
generative models
Keywords:
text-to-image generation, diversity evaluation, diffusion models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6r7BOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper asks how many images per prompt are needed to compare the diversity of two text-to-image configurations with the Vendi score. It observes that the individual Vendi scores keep growing with the sample size kk, while their log-ratio stabilizes much earlier, and that the required kk varies across comparisons. The proposed method, DivShift, is a sequential stopping rule. It fits a power law to the log-ratio trajectory, extrapolates it to a reference size KK, and uses a prompt bootstrap to decide whether to stop or to generate more images. In a retrospective replay over five panels (guidance scales, diversity interventions, and model architectures), DivShift saves 79.3% of generations relative to K=512K = 512, with 0.10% false adequacy (stops whose ratio differs from the ratio at KK by more than ρ\rho) at ρ=10%\rho = 10\%.

Critical strengths
  1. Relevant problem and useful observation. Existing Vendi-based evaluations use 3–16 images per prompt without justification. The paper shows that most of the score growth cancels in the ratio (Table 3, Fig. 7), while the required kk still varies widely across comparisons (Fig. 3). This finding is useful for anyone reporting Vendi and motivates choosing kk per comparison.
  2. Rigorous evaluation that makes the reported numbers credible. Paired nested seeds, strictly prospective replay, and thorough ablations make the results internally consistent. Without hindsight, DivShift selects a kk close to the best fixed kk for each panel (Fig. 5).
Critical weaknesses
  1. Unclear practical significance of the controlled quantity. False adequacy measures closeness to the ratio at an arbitrary finite KK, and this ratio still drifts at K=512K = 512 (Fig. 7b, Table 10). The 10% tolerance is reasonable for the CFG comparisons (ratios of 1.2–1.8, Fig. 7b), but it exceeds typical method-versus-baseline gains, e.g., about 2% in Vendi for CNO and CADS in Kim et al. (2026). It also does not capture sign or ranking agreement. Moreover, the 79% saving is relative to K=512K = 512. On the CFG and architecture panels, DivShift stops at roughly 130–160 images per prompt, which exceeds the 3–16 used in current practice. The paper should therefore show that small fixed budgets lead to wrong conclusions that DivShift avoids.

  2. Advantage over simpler alternatives not established. DivShift is compared only with fixed kk, not with simple sequential rules or with directly reporting the extrapolated ratio (as in FID$_\infty$, Chong & Forsyth, 2020). DivShift outperforms the best fixed kk within a panel only on the architecture panel (Fig. 5), so a fixed kk calibrated once per comparison family may perform similarly. In addition, hyperparameters were selected on the evaluation panels, and the 0.10% false adequacy corresponds to only 3 of 600 instances on a single panel.

  3. Possible protocol mismatch for batch-coupled methods. The nested-prefix and permutation replay protocol assumes that the samples for each prompt are exchangeable. CNO, however, obtains its diversity gain by jointly optimizing a batch of noises, and generating in 8-image chunks and then permuting the pool may largely remove this effect. This may explain why CNO and the baseline are nearly identical in the paper (∣Δ∣≤8×10−4|\Delta| \le 8\times10^{-4}, Fig. 8). If so, the intervention panels, which yield the highest savings (91–92%), are less informative than they appear, and the method does not apply to such samplers.

Questions for authors
  1. How was CNO configured (batch size, chunking, and γ\gamma)? Does it show a non-trivial Vendi gain over the baseline in your setting?
  2. For k∈{4,8,16}k \in \{4, 8, 16\}, how often does the sign or ranking of Δk\Delta_k differ from that of ΔK\Delta_K?
  3. How does DivShift compare with simple sequential rules and with reporting the extrapolated ratio at k=32k = 32?
Reviewer_p3KvOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates how many images per prompt are needed to reliably compare the diversity of text-to-image generators using the vendi score. The authors show that relative diversity comparisons can stabilize before individual vendi scores, and that the required sample size varies across configurations. They propose DivShift, an adaptive method that uses power-law forecasting and bootstrap uncertainty estimation to determine when to stop generating images. Experiments across different models, guidance settings, and diversity interventions show 79.3% generation savings with only 0.10% false adequacy, compared to evaluating all configurations at 512 images per prompt.

Critical strengths
  1. This is a very relevant and underexplored studied evaluation problem, that could directly improve the efficiency and reliability of generative-model evaluation.
  2. The proposed methodology is very well explained.
  3. The results are promosing and suggest that the method could be useful for reducing the computational cost of diversity evaluations.
Critical weaknesses
  1. The authors show that DivShift performs better than using a fixed number of images, but it is less clear how much of this improvement comes from the proposed forecasting method rather than from adaptive sampling itself. For example, a simpler approach could progressively increase the number of images and stop when the vendi ratio changes very little between consecutive rounds. It would be useful to include such a baseline to better understand whether the power-law forecasting provides a meaningful advantage.

  2. The reported generation savings and very low false-adequacy rates are promising. However, the evaluation relies on 200 permutations of the same generated image pools, rather than independently generated samples. Although this is a reasonable way to study the effect of sample ordering, it is unclear wheher the stopping decisions would be equally reliable with entirely new generations. Could the authors investigate how much the stopping sample sizes and false-adequacy rates vary across independent runs?

  3. The method evaluates stability relative to a fixed reference size of (K=512), but it is unclear whether this provides a stable comparison across all experimental settings. The authors investigate this by extending the evaluation to (K=1024), but only for the CFG–COCO panel. Could they extend this analysis to the other comparison families, particularly the architecture comparisons?

Questions for authors

See "critical weaknesses" section for questions.

Reviewer_cHxCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper studies how many images are needed per prompt to compare text-to-image models using the average Vendi score. The authors show that the ratio between two models’ scores can stabilize before the individual scores and propose DivShift to adapt the sample count to each comparison, reducing image generation while keeping the ratio close to a larger-budget reference.

Critical strengths
  • The paper studies an important practical problem in text-to-image evaluation: how many images are needed to compare diversity reliably? The authors show that this number can differ substantially across model comparisons, which questions the common practice of using the same fixed sample count for every evaluation.

  • The observation that two models’ Vendi scores can change similarly as more samples are added, while their ratio changes much less, is interesting. The authors use this observation to develop an adaptive sampling procedure and demonstrate that it can reduce the number of generated images while keeping the comparison close to the larger-budget reference.

Critical weaknesses
  • The sample-count dependence motivating this paper has been studied in [1], which shows that standard order-one Vendi can continue growing substantially at practical sample sizes and proposes truncated Vendi with finite-sample convergence guarantees for unconditional generative models. Also [2] introduces FKEA-Vendi, which also admits convergence guarantees for Vendi score, while the RKE score proposed in [3], corresponding to Rényi order α=2\alpha=2, has dimension-free convergence guarantees.

  • Since these alternatives already address the estimation problem motivating this paper, a direct comparison of sample requirements across these variants is needed to establish whether the proposed adaptive procedure offers benefits beyond existing approaches with convergence guarantees. Specifically comparison with RKE score seems needed as the dimension-free convergence of this score can require substantially less samples.

  • For text-to-image models specifically, Conditional-Vendi and Conditional-RKE extend spectral diversity measures to conditional settings and provide finite-sample convergence guarantees for prompt-aware evaluation. A more in-depth comparison with this work is needed to clarify how the sample requirements of these conditional measures differ from those of the proposed averaging scheme, and whether the adaptive procedure offers similar benefits when applied to them.

  • The choice to average Vendi scores across exact prompts needs further justification. Consider a strong model operating at a very low temperature, whose outputs are almost deterministic for each exact prompt. Changing just one word could still produce substantially different outputs, even when the prompts remain very similar. Each per-prompt Vendi score would be close to one, so their average would not capture this variation. Should we interpret such a model as lacking diversity, or should a prompt-aware measure also account for variation across closely related prompts? The authors should clarify why averaging per-prompt scores is the appropriate target for their intended evaluation. Without a clearer justification for this choice, it remains unclear whether determining the number of samples needed to stabilize the average Vendi score is sufficient to support meaningful conclusions about the diversity of conditional models.


[1] Ospanov, A., and Farnia, F. . Do Vendi Scores Converge with Finite Samples? Truncated Vendi Score for Finite-Sample Convergence Guarantees. UAI 2025

[2] Ospanov et al. "Towards a Scalable Reference-Free Evaluation of Generative Models". NeurIPS 2024

[3] Jalali et al. "An Information-Theoretic Evaluation of Generative Models in Learning Multi-modal Distributions". NeurIPS 2023.

Questions for authors
  • How do the sample requirements and benefits of DivShift change when using truncated Vendi, FKEA-Vendi, or RKE, which already have convergence guarantees? This comparison would help clarify whether the observed behavior is specific to standard order-one Vendi.

  • How does averaging Vendi scores across prompts compare with Conditional-Vendi and Conditional-RKE? Can the proposed adaptive procedure also be applied to these prompt-aware measures?

  • Why is the average per-prompt Vendi score the appropriate target for the intended evaluation? A model can be almost deterministic at each exact prompt but produce substantially different outputs across closely related prompts. Since the average would not capture this variation, what conclusions about conditional-model diversity does stabilizing it support?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.