- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper asks how many images per prompt are needed to compare the diversity of two text-to-image configurations with the Vendi score. It observes that the individual Vendi scores keep growing with the sample size , while their log-ratio stabilizes much earlier, and that the required varies across comparisons. The proposed method, DivShift, is a sequential stopping rule. It fits a power law to the log-ratio trajectory, extrapolates it to a reference size , and uses a prompt bootstrap to decide whether to stop or to generate more images. In a retrospective replay over five panels (guidance scales, diversity interventions, and model architectures), DivShift saves 79.3% of generations relative to , with 0.10% false adequacy (stops whose ratio differs from the ratio at by more than ) at .
Critical strengths
- Relevant problem and useful observation. Existing Vendi-based evaluations use 3–16 images per prompt without justification. The paper shows that most of the score growth cancels in the ratio (Table 3, Fig. 7), while the required still varies widely across comparisons (Fig. 3). This finding is useful for anyone reporting Vendi and motivates choosing per comparison.
- Rigorous evaluation that makes the reported numbers credible. Paired nested seeds, strictly prospective replay, and thorough ablations make the results internally consistent. Without hindsight, DivShift selects a close to the best fixed for each panel (Fig. 5).
Critical weaknesses
-
Unclear practical significance of the controlled quantity. False adequacy measures closeness to the ratio at an arbitrary finite , and this ratio still drifts at (Fig. 7b, Table 10). The 10% tolerance is reasonable for the CFG comparisons (ratios of 1.2–1.8, Fig. 7b), but it exceeds typical method-versus-baseline gains, e.g., about 2% in Vendi for CNO and CADS in Kim et al. (2026). It also does not capture sign or ranking agreement. Moreover, the 79% saving is relative to . On the CFG and architecture panels, DivShift stops at roughly 130–160 images per prompt, which exceeds the 3–16 used in current practice. The paper should therefore show that small fixed budgets lead to wrong conclusions that DivShift avoids.
-
Advantage over simpler alternatives not established. DivShift is compared only with fixed , not with simple sequential rules or with directly reporting the extrapolated ratio (as in FID$_\infty$, Chong & Forsyth, 2020). DivShift outperforms the best fixed within a panel only on the architecture panel (Fig. 5), so a fixed calibrated once per comparison family may perform similarly. In addition, hyperparameters were selected on the evaluation panels, and the 0.10% false adequacy corresponds to only 3 of 600 instances on a single panel.
-
Possible protocol mismatch for batch-coupled methods. The nested-prefix and permutation replay protocol assumes that the samples for each prompt are exchangeable. CNO, however, obtains its diversity gain by jointly optimizing a batch of noises, and generating in 8-image chunks and then permuting the pool may largely remove this effect. This may explain why CNO and the baseline are nearly identical in the paper (, Fig. 8). If so, the intervention panels, which yield the highest savings (91–92%), are less informative than they appear, and the method does not apply to such samplers.
Questions for authors
- How was CNO configured (batch size, chunking, and )? Does it show a non-trivial Vendi gain over the baseline in your setting?
- For , how often does the sign or ranking of differ from that of ?
- How does DivShift compare with simple sequential rules and with reporting the extrapolated ratio at ?