acceptodds
Under review as a conference paper at ICLR 2027

Certifying What Your Generative Model Covers: Labeller-Conditional, Finite-Sample Mode-Recall Guarantees

Abstract

Recall- and coverage-style metrics are the standard tools for asking whether a generative model has silently dropped modes, yet they carry no guarantee and entangle fidelity with coverage—a collapsed GAN and a fully-covering flow model can both get nearest-neighbour recall , and in controlled settings recall and FID instead over-state coverage. We introduce certified mode-recall: given a mode labeller (a class or attribute classifier) and a tolerance , a finite-sample, two-sided per-mode procedure that certifies each mode present (), absent (, genuine collapse detected), or inconclusive—so we both lower-bound coverage and detect missing modes, rather than reporting a floor a low value cannot distinguish from under-sampling. It is labeller-conditional and otherwise distribution-free (assuming only a bound on the labeller's confusion on the generator), with per-mode thresholds made rigorous by sample splitting and a deconvolution to true (not predicted) mode frequencies; it certifies presence/absence of labeller-defined subgroups, not within-subgroup diversity. We prove two-sided validity, consistency for -presence (the threshold closes the otherwise uncertifiable band ), and a witnessing rate (in the leakage-dominated regime; under low leakage) that delimits when rare protected subgroups are certifiable at all. Empirically the certificate is valid (zero false present/absent over seeds) and detects collapse across GANs, flow-matching, and pretrained diffusion (CIFAR DDPM, CelebA-HQ ), exposing over- and under-statement that precision/recall, density/coverage, and FID all miss.

Primary area:
generative models
Keywords:
generative model evaluation, mode collapse, finite-sample guarantees, coverage certificate, fairness auditing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jFoJOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a finite-sample certification framework for evaluating semantic mode coverage in generative models. Given a predefined set of semantic modes, a mode classifier, and a minimum coverage-probability threshold, the proposed method draws a finite number of samples from a generative model and classifies each mode as ​present​, ​absent​, or ​inconclusive​, based on its observed sample count. Building on these per-mode decisions, the method also provides a statistically valid lower confidence bound on the model’s overall mode recall or coverage. Unlike simply counting how many semantic categories appear among the generated samples, the proposed framework explicitly accounts for classifier misclassification, finite-sample uncertainty, and the multiple-testing problem arising from simultaneously evaluating many modes.

Critical strengths

1.The paper addresses a clearly defined and relevant evaluation problem. Rather than focusing on the perceptual quality of generated images, the paper investigates whether a generative model fails to cover important predefined semantic categories. This perspective provides a useful complement to existing evaluation metrics for generative models. 2.The present/absent/inconclusive decision framework is well designed. The three-way classification explicitly distinguishes between a mode that is certified as sufficiently covered, a mode that is certified as failing to meet the required coverage threshold, and a mode for which the available evidence is insufficient. This design avoids treating an unobserved mode as automatic evidence of mode dropping and offers good interpretability and practical value for auditing generative models.

Critical weaknesses

1.The practical motivation is not sufficiently developed. The paper argues that the proposed method provides a more precise assessment of how many semantic categories a generative model can cover. However, many modern generative models are conditioned on class labels, text prompts, or reference images, allowing users to specify the desired semantic category directly. The paper should explain more clearly what indispensable role this certification framework plays in the context of conditional generation and what failure modes it can detect that cannot be identified through direct conditional sampling or existing evaluation procedures. 2.The experimental datasets are not sufficiently representative. The datasets used in the main experiments are generally dated and relatively small. As a result, the current evaluation provides limited evidence that the proposed method can scale or generalize to modern large-scale generative models and open-vocabulary semantic settings. Experiments on more recent and challenging datasets would substantially strengthen the paper. 3.Figure 2 does not clearly or accurately illustrate the actual sample-splitting procedure. The figure appears to first compute per-mode counts (c_j) using all generated samples and then apply thresholds derived through sample splitting. However, Algorithm 1 requires two mutually independent sample batches, (A) and (B). Batch (A) is used to estimate label leakage and determine the per-mode certification thresholds, whereas batch (B) is used to compute the test counts (c_j). I recommend redrawing the figure to explicitly show the complete pipeline: sample splitting, separate counting on the two batches, threshold estimation using batch (A), and independent hypothesis testing using batch (B). 4.The related literature and baseline comparisons are outdated. The reference list contains only six papers published after 2021, and the experimental baselines do not adequately represent recent progress in generative-model evaluation. This weakens the paper’s positioning relative to the current state of the art. The authors should expand the discussion of recent literature and include stronger, more contemporary baselines where applicable.

Questions for authors

see weakness

1 comment
Reviewer_jFoJOpenReview ↗

Thanks for the response

Thank you for the detailed rebuttal and substantial revisions. I appreciate the authors' efforts to address my concerns, particularly the additional experiments on modern generative models and the improved presentation. However, I still share some of the concerns raised by other reviewers, especially regarding the reliability of the underlying assumptions and the tightness of the certification bounds. Given my limited expertise in the theoretical aspects of this work, I would like to hear the other reviewers' perspectives before reconsidering my assessment. For now, I am inclined to maintain my original rating.

Reviewer_Ww17OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present a new method that takes a mode classifier to label generated samples and decides (based on a tolerance γ\gamma) whether or not a mode is present, absent or inconclusive for a given sample budget. The method also corrects for the classifier's confusion under the assumption that its conufsion on real data bounds its confusion on generated samples. The theory includes: a finite sample lower bound on mode recall, finite sample validity, consistency for the linear programming variant, and a sample complexity rate . The authors have controlled experiments with known ground truth mode dropping and show how existing NN methods and FID misstate coverage. Finally, they provide experimental analysis on StackedMNIST where there is 1000 modes for 3 generative models from 2 families, and they show that the certificate is valid. There is also experiments on other datasets such as CIFAR-10 and CelebA.

Critical strengths
  1. Mode dropping is an important problem and introducing a per-mode present, absent, or inconclusive decision with finite sample guarantees is a good fit of fairness auditing.
  2. The theoretical results seem correct. Validity follows from binomial tail bounds, sample splitting and a union bound given the confusion assumption.
Critical weaknesses
  1. The whole method is based off of an assumption that the confusion on real data bounds its confusion on generated samples. This can be a valid assumption, but if it fails default certificates are no longer valid and the method's reliability is in question. The paper's own evidence shows this potential shortcoming; on Human calibrated audit in the appendix table 12 the Male/Smiling attributes show classifier accuracy drops when the assumption fails (by 0.27 and 0.22).
  2. The experimental results do not show that the certificates are tight; although they are valid. Table 3 shows this where values of 0.29 vs 0.54, 0.53 vs 0.92, etc.
Questions for authors

The work is well formulated and seems theoretically sound. The paper proposes a solution to an important problem from an original perspective. There are several key issues that either need to be addressed or corrected. If the authors can address the questions I would be willing to adjust my score:

  1. The experiments suggest some evidence that the fundamental assumption that confusion on real data bounds its confusion on generated samples. LP is the only method that survives the shift why isnt it the default then?

  2. The bounds are loose and although experimentally the certificates are valid they are not tight; see table 3: 0.29 vs 0.54, 0.53 vs 0.92 etc. Where does this looseness come from?

Reviewer_b6V3OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Metrics commonly used to evaluate the coverage of generative models lack formal guarantees and tend to overestimate the coverage. To overcome this, the authors propose a classifier-based metric to assess whether a generative model sufficiently covers the different modes of the data distribution, which they assume are given by the different class labels.

Critical strengths

Evaluating generative models performance quantitatively remains a challenging and open problem. In this case, the authors propose a metric to improve the evaluation of the mode coverage of these models.

Critical weaknesses
  • The quality of the writing could be improved. The paper is in general very difficult to follow, as different elements are introduced without sufficient explanation or motivation. For example, variables and equations are used without prior definition (as the expressions used in the abstract and the introduction), theorems are used as part of the explanations before being presented, and acronyms are used without being defined. Additionally, the background required to understand the proposed method is not properly introduced. For example, different statistical corrections and tests introduced in the method, while might be widely used in other fields, may not be familiar to the broader ML community.
  • Evaluating class-label coverage does not necessarily provide a reliable measure of how well the generative model is capturing the modes and variance of the data distribution. For example, according to my understanding, a model that generates samples from every class but collapses by producing the same image for each class could still achieve a high score according to this metric, despite failing to capture the intra-class diversity.
  • The paper only cites some related works in the introduction and related work sections, but does not reference any work in the description of the method.
Questions for authors
  • The metric proposed assumes that the modes of the data distribution correspond to the semantical classes. However, in complex datasets, a single semantic class can have many different modes (for example, many different breeds within the class ‘dog’), as well as a large diversity that should be also captured by the model. Even relatively simple datasets, such as MNIST, have a considerably diversity in the styles used to represent each digit. This raises the question of whether generating images from all the semantic classes enough to claim that the generative model has a good coverage of the data distribution. For example, a model that generates samples from every class but collapses by producing the same image for each class could still achieve a high score according to this metric? Is this a good evaluation of the generative model performance?
  • The method relies on having access to an independent real calibration sample. Is this a realistic assumption in practice? How sensitive is the metric to the size of this sample? How large needs this set to be in order to obtain tight bounds? How sensitive it is to potential distribution shifts between the real calibration sample and the ones generated by the model, or the ones used for training the classifier?
  • How does the selection of the hyperparameters and thresholds affect the tightness and reliability of the metric?
  • The authors should carefully polish the text, as there are several typos, including wrong numbers in the discussions on the experimental results (as in line 463).
2 comments

Thank you for your clarifications and the additional experimental results. I see that inspecting whether the model can generate samples from specific semantic classes can be valuable for certain applications, such as fairness. However, I would suggest being more explicit in the naming of the method and make it more aligned with what the metric actually measures (maybe label coverage instead of mode coverage?). While I value the effort of the authors, I think there are some aspects that deserve further careful investigation and discussion in a broader revision of the manuscript. For example, although the authors claim that in the CIFAR-10 experiments the certification produces no false calls regardless the number of samples used for certification, the collapsed-model experiment suggests that this is not always the case. I also agree with the other reviewers that the tightness deserves further discussion. More generally, I think the paper could benefit from a proper discussion on practical considerations, limitations, failure cases, which would help readers better understand how to use the proposed metric in practice.

I have also gone through the revised version and I still think the text and presentation can be further polished (for example, the conclusions are limited to 3 lines and do not give any intuition/summary on the limitations, they directly refer the reader to the appendix).

I will raise my score to a weak reject given the new results and effort put on the rebuttal. However, I think the article could benefit from a major revision.

Second response to Reviewer b6V3

Thank you for reading the revision and for raising your score. We have revised the paper again along each of your points; new changes are in blue.

Naming. We agree. The subtitle is now "Labeller-Conditional, Finite-Sample Label-Coverage Guarantees". The abstract and the introduction say that what is certified is label coverage: the generator's mass on each cell of the auditor's labeller partition. We keep the word "mode" for these cells, following the mode-collapse literature. The text also says explicitly that the certificate makes no claim about diversity below the chosen partition.

False calls. You are right that our earlier statement was too broad. The CIFAR-10 sweep (210 runs, no false call) calibrates the labeller on clean images from its own distribution, so Assumption 1 holds there. Under that assumption, validity holds at every budget and every calibration size. The template generator ("one image per class") is a different case: it violates Assumption 1. Its few distinct images are labelled the same way every time, so the labeller's errors do not average out. In that case false calls occur at any budget (20 of 20 draws). Calibrating on 200 generated samples with trusted labels removes them (0 of 20). The abstract, Appendix D and the new Section 6 now state this scope wherever a zero-false-call result is reported.

Practical considerations, limitations and failure cases. The main text has a new Section 6, "Practical use, limitations and failure cases". It contains three paragraphs.

  • When the guarantee holds, and when it fails. It lists the four cases in which we observed or measured a failure of Assumption 1:

    1. few distinct outputs (the template generator);
    2. severe calibration shift (CIFAR-10-C with labeller accuracy ≤ 0.65: a class just below γ is called present in 5 of 7 settings);
    3. a labeller that is less accurate on generated images (CelebA-HQ human audit: Male and Smiling accuracy drop by 0.27 and 0.22);
    4. attribute-dependent pre-processing (a face detector that misses mostly one group).

    It also gives the remedy: an audit set of generated samples. Without one, a margin protects only against shifts within the margin.

  • Tightness and budget (see below).

  • Recommended use. There are four steps:

    1. choose the partition and γ for the question at hand;
    2. calibrate the labeller on held-out labelled data (in our CIFAR-10 sweep, 25 images per class gave valid detection and about 100 gave tight results);
    3. use an audit set and the default rule when possible, and otherwise the LP with a stated margin, reporting the result as conditional on that margin;
    4. report present, absent and inconclusive per mode.

Tightness. Section 6 now discusses this in the main text.

  • The certificate is tight when each mode receives many samples: on CIFAR-10 (M = 10) it matches the oracle.
  • It is conservative when each mode receives few: on StackedMNIST, with 8 samples per mode, it certifies 0.40 for a GAN that covers 0.97.
  • In an ablation, most of this gap remains even when the true leakage is known. The gap closes as the budget grows: 0.83 and 0.97 of the witnessed modes at 4× and 16× the samples.
  • The sample-complexity theorem gives the budget each mode needs. Rare and easily confused modes are the most expensive, and a subgroup with almost no calibration images is never certified.

Presentation. The conclusion now summarises what the certificate does and states its main limitation in the main text: the guarantee rests on the labeller's calibration transferring to generated images, and an audit set is needed when that is in doubt. It no longer only refers to the appendix. To keep the main text within 10 pages, we moved the consistency theorem statement to Appendix A.4 (the main text keeps its conclusion in one sentence) and merged a CIFAR-10 paragraph that repeated results already given in the two-sided detection paragraph.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.