- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies how to certify whether a set of trained synthetic data generators (SDGs) satisfies dataset-specific fidelity requirements. This is a practically relevant problem, especially in domains such as healthcare and finance, where data quality is critical. The authors formulate generator certification as a multiple hypothesis testing problem, combining MMD concentration bounds, independent holdout data, and a graph-based multiple testing procedure to guarantee that all certified generators meet a predefined fidelity threshold. Experiments on 7 synthetic data distributions and 6 GAN architectures illustrate the certification results under different numbers of candidate models.
My main concerns are: (i) the core p-value formula is inconsistent with the implementation, and the graph-based multiple testing code contains an error that changes the certification results (W1); and (ii) the contributions beyond existing statistical certification methods are not sufficiently clear, while the experiments lack direct evaluation of the false certification rate and statistical power (W2). I would consider increasing my rating if the authors address these issues, revalidate the affected results, and provide the necessary statistical evaluations. If the authors adequately address my concerns in W2, I would consider increasing my rating to 2. If both W1 and W2 are satisfactorily resolved, I would consider increasing my rating to 3.
Critical strengths
- A practically relevant problem. Certifying the fidelity of trained generators on a specific dataset can help practitioners determine which models meet explicit quality requirements, rather than relying solely on empirical metrics or benchmark performance.
- A clear and flexible certification framework. The proposed approach separates model training from certification, uses independent holdout data for statistical testing, and applies multiple testing to certify several generators simultaneously. The framework is architecture-agnostic and allows multiple qualified generators to be retained.
Critical weaknesses
W1. Errors in the methodology and code implementation require correction.
Errors in the statistical test and its implementation affect the certification guarantee and experimental results.
§5.2 and Appendix A define the p-value as:
This expression does not correctly account for the direction of the one-sided test. When , the estimated discrepancy already exceeds the acceptable threshold, yet the squared term can still produce an extremely small p-value, leading to an incorrect rejection of the null hypothesis.
For the Hoeffding-type concentration bound used in the paper, the correct expression should be:
As written, this formula does not support the claimed finite-sample guarantee. The original concentration bound in 📍R1, Theorem 10 does not justify the untruncated expression.
After examining the supplementary code, the supplementary implementation compute_mmd_pvals() already handles correctly. The authors should revise the formula and proof to ensure consistency with the implementation.
I also found that the graph weight update in the supplementary code is incorrect.
The function
bretz_algorithm()useswhereas 📍Algorithm 1 and R2 specify
These updates are not equivalent. Recomputing the results using the saved MMD values in the supplementary notebook shows that the corrected procedure changes the certified subset for Dataset 0 with Top-4 candidates from
{3, 1}to{3, 1, 2, 4}. This directly affects the discussion in §6.2 regarding how the number of candidates influences certification.
W2. Limited experimental validation of the certification procedure.
The experimental scope and statistical validation are relatively limited. The paper evaluates 7 simulated data distributions, 6 GAN architectures, and only two candidate set sizes (). In comparison, R3, §3–4 evaluates finite-sample risk control on real datasets such as MS COCO and CIFAR-10, using 100–1000 random splits in some experiments. R4, §6 conducts 100 repetitions per setting across Gaussian, RBM, and MNIST experiments, reporting empirical rejection rates and 95% confidence intervals. The main limitation is therefore not simply the number of datasets, but the lack of repeated statistical evaluation.
The experiments provide limited evidence of the stability of certification outcomes. In §6.2, the authors interpret the absence of incorrect certification markers in 📍Figure 2 and 📍Figure 3 as supporting the certification guarantee. However, the experiments do not report variability across repeated independent trials. Although such repetitions are not necessary to establish the theoretical guarantee, they would provide additional evidence of the stability and empirical reliability of the procedure.
References
- [R1] A Kernel Two-Sample Test. Gretton et al., JMLR, 2012.
- [R2] A Graphical Approach to Sequentially Rejective Multiple Test Procedures. Bretz et al., Statistics in Medicine, 2009.
- [R3] Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control. Angelopoulos et al., Annals of Applied Statistics, 2025.
- [R4] Kernel Tests of Equivalence. Liu and Gandy, arXiv, 2026.
Questions for authors
- Choice of fidelity threshold. The experiments use a fixed threshold of across all seven datasets, while the Gaussian kernel bandwidth is selected separately for each dataset. What motivates this threshold, and how should practitioners choose a meaningful fidelity tolerance for a new dataset? Would the certification results be sensitive to different kernel bandwidths or threshold choices?
- Candidate selection strategy. The experiments select the top generators based on training-set MMD estimates, and the results show that increasing can sometimes cause previously certified models to lose certification. How should practitioners choose the candidate set and its size in practice? Is there a principled strategy for balancing the number of candidates against the difficulty of certification?
- Practical value of certifying multiple generators. One motivation of the paper is to retain multiple certified generators for subsequent downstream decisions. Could the authors provide a concrete example where retaining a certified subset offers a meaningful advantage over selecting a single generator based on empirical fidelity? This would help clarify the practical benefits of simultaneous certification.