- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes a finite-sample certification framework for evaluating semantic mode coverage in generative models. Given a predefined set of semantic modes, a mode classifier, and a minimum coverage-probability threshold, the proposed method draws a finite number of samples from a generative model and classifies each mode as present, absent, or inconclusive, based on its observed sample count. Building on these per-mode decisions, the method also provides a statistically valid lower confidence bound on the model’s overall mode recall or coverage. Unlike simply counting how many semantic categories appear among the generated samples, the proposed framework explicitly accounts for classifier misclassification, finite-sample uncertainty, and the multiple-testing problem arising from simultaneously evaluating many modes.
Critical strengths
1.The paper addresses a clearly defined and relevant evaluation problem. Rather than focusing on the perceptual quality of generated images, the paper investigates whether a generative model fails to cover important predefined semantic categories. This perspective provides a useful complement to existing evaluation metrics for generative models. 2.The present/absent/inconclusive decision framework is well designed. The three-way classification explicitly distinguishes between a mode that is certified as sufficiently covered, a mode that is certified as failing to meet the required coverage threshold, and a mode for which the available evidence is insufficient. This design avoids treating an unobserved mode as automatic evidence of mode dropping and offers good interpretability and practical value for auditing generative models.
Critical weaknesses
1.The practical motivation is not sufficiently developed. The paper argues that the proposed method provides a more precise assessment of how many semantic categories a generative model can cover. However, many modern generative models are conditioned on class labels, text prompts, or reference images, allowing users to specify the desired semantic category directly. The paper should explain more clearly what indispensable role this certification framework plays in the context of conditional generation and what failure modes it can detect that cannot be identified through direct conditional sampling or existing evaluation procedures. 2.The experimental datasets are not sufficiently representative. The datasets used in the main experiments are generally dated and relatively small. As a result, the current evaluation provides limited evidence that the proposed method can scale or generalize to modern large-scale generative models and open-vocabulary semantic settings. Experiments on more recent and challenging datasets would substantially strengthen the paper. 3.Figure 2 does not clearly or accurately illustrate the actual sample-splitting procedure. The figure appears to first compute per-mode counts (c_j) using all generated samples and then apply thresholds derived through sample splitting. However, Algorithm 1 requires two mutually independent sample batches, (A) and (B). Batch (A) is used to estimate label leakage and determine the per-mode certification thresholds, whereas batch (B) is used to compute the test counts (c_j). I recommend redrawing the figure to explicitly show the complete pipeline: sample splitting, separate counting on the two batches, threshold estimation using batch (A), and independent hypothesis testing using batch (B). 4.The related literature and baseline comparisons are outdated. The reference list contains only six papers published after 2021, and the experimental baselines do not adequately represent recent progress in generative-model evaluation. This weakens the paper’s positioning relative to the current state of the art. The authors should expand the discussion of recent literature and include stronger, more contemporary baselines where applicable.
Questions for authors
see weakness
Thanks for the response
Thank you for the detailed rebuttal and substantial revisions. I appreciate the authors' efforts to address my concerns, particularly the additional experiments on modern generative models and the improved presentation. However, I still share some of the concerns raised by other reviewers, especially regarding the reliability of the underlying assumptions and the tightness of the certification bounds. Given my limited expertise in the theoretical aspects of this work, I would like to hear the other reviewers' perspectives before reconsidering my assessment. For now, I am inclined to maintain my original rating.