acceptodds
Under review as a conference paper at ICLR 2027

Simultaneous Finite-Sample Certification of Synthetic Data Generators under Fidelity Constraints

Abstract

Synthetic data generators (SDGs) are increasingly used in high-stakes settings, yet practitioners often lack a principled way to determine whether the generated data adequately represent the real data distribution for a specific dataset. While theoretical guarantees for SDGs do exist, they are often asymptotic and therefore do not directly provide finite-sample guarantees for the trained generator and dataset at hand. As a result, it can be difficult to determine which among several candidate generators satisfy a required standard of data fidelity. We address this problem by formulating the certification of SDGs as a multiple hypothesis testing problem over a collection of candidate generators. For each generator, we test whether a population-level fidelity criterion, defined through a user-specified discrepancy measure between the real and synthetic distributions, falls below a prescribed tolerance level. By constructing valid tests from finite-sample concentration bounds and applying a family-wise error rate controlling procedure, our framework guarantees, with probability at least , that all selected generators satisfy the specified fidelity criterion. This provides dataset-specific, finite-sample certification of synthetic data fidelity and allows practitioners to screen multiple candidate generators and retain a statistically supported subset that meets a desired fidelity standard before considering other downstream objectives. The framework is agnostic to the generator architecture and applies to a broad class of discrepancy measures whose estimators admit suitable concentration inequalities, making it useful for applications where reliable finite-sample assessment of synthetic data is important.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
synthetic data, multiple hypothesis testing, distributional closeness, fidelity

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_CyhgOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how to certify whether a set of trained synthetic data generators (SDGs) satisfies dataset-specific fidelity requirements. This is a practically relevant problem, especially in domains such as healthcare and finance, where data quality is critical. The authors formulate generator certification as a multiple hypothesis testing problem, combining MMD concentration bounds, independent holdout data, and a graph-based multiple testing procedure to guarantee that all certified generators meet a predefined fidelity threshold. Experiments on 7 synthetic data distributions and 6 GAN architectures illustrate the certification results under different numbers of candidate models.

My main concerns are: (i) the core p-value formula is inconsistent with the implementation, and the graph-based multiple testing code contains an error that changes the certification results (W1); and (ii) the contributions beyond existing statistical certification methods are not sufficiently clear, while the experiments lack direct evaluation of the false certification rate and statistical power (W2). I would consider increasing my rating if the authors address these issues, revalidate the affected results, and provide the necessary statistical evaluations. If the authors adequately address my concerns in W2, I would consider increasing my rating to 2. If both W1 and W2 are satisfactorily resolved, I would consider increasing my rating to 3.

Critical strengths
  • A practically relevant problem. Certifying the fidelity of trained generators on a specific dataset can help practitioners determine which models meet explicit quality requirements, rather than relying solely on empirical metrics or benchmark performance.
  • A clear and flexible certification framework. The proposed approach separates model training from certification, uses independent holdout data for statistical testing, and applies multiple testing to certify several generators simultaneously. The framework is architecture-agnostic and allows multiple qualified generators to be retained.
Critical weaknesses

W1. Errors in the methodology and code implementation require correction.

  • Errors in the statistical test and its implementation affect the certification guarantee and experimental results.

    • §5.2 and Appendix A define the p-value as: pipaper=exp⁡[−m28(ϵ−θ^i)2],m2=⌊c/2⌋.p_i^{\mathrm{paper}}=\exp\left[-\frac{m_2}{8}(\epsilon-\hat{\theta}_i)^2\right],\qquad m_2=\lfloor c/2\rfloor.

      This expression does not correctly account for the direction of the one-sided test. When θ^i>ϵ\hat{\theta}_i>\epsilon, the estimated discrepancy already exceeds the acceptable threshold, yet the squared term can still produce an extremely small p-value, leading to an incorrect rejection of the null hypothesis.

    • For the Hoeffding-type concentration bound used in the paper, the correct expression should be: pi=exp⁡[−m28(max⁡{ϵ−θ^i,0})2].p_i=\exp\left[-\frac{m_2}{8}\left(\max\{\epsilon-\hat{\theta}_i,0\}\right)^2\right].

    As written, this formula does not support the claimed finite-sample guarantee. The original concentration bound in 📍R1, Theorem 10 does not justify the untruncated expression.

    After examining the supplementary code, the supplementary implementation compute_mmd_pvals() already handles θ^i≥ϵ\hat{\theta}_i\geq\epsilon correctly. The authors should revise the formula and proof to ensure consistency with the implementation.

  • I also found that the graph weight update in the supplementary code is incorrect.

    • The function bretz_algorithm() uses gℓk←gℓk+gℓjgjk1−gℓjgjℓ,g_{\ell k}\leftarrow g_{\ell k}+\frac{g_{\ell j}g_{jk}}{1-g_{\ell j}g_{j\ell}},

    • whereas 📍Algorithm 1 and R2 specify gℓk←gℓk+gℓjgjk1−gℓjgjℓ.g_{\ell k}\leftarrow\frac{g_{\ell k}+g_{\ell j}g_{jk}}{1-g_{\ell j}g_{j\ell}}.

    These updates are not equivalent. Recomputing the results using the saved MMD values in the supplementary notebook shows that the corrected procedure changes the certified subset for Dataset 0 with Top-4 candidates from {3, 1} to {3, 1, 2, 4}. This directly affects the discussion in §6.2 regarding how the number of candidates influences certification.

W2. Limited experimental validation of the certification procedure.

  • The experimental scope and statistical validation are relatively limited. The paper evaluates 7 simulated data distributions, 6 GAN architectures, and only two candidate set sizes (N=3,4N=3,4). In comparison, R3, §3–4 evaluates finite-sample risk control on real datasets such as MS COCO and CIFAR-10, using 100–1000 random splits in some experiments. R4, §6 conducts 100 repetitions per setting across Gaussian, RBM, and MNIST experiments, reporting empirical rejection rates and 95% confidence intervals. The main limitation is therefore not simply the number of datasets, but the lack of repeated statistical evaluation.

  • The experiments provide limited evidence of the stability of certification outcomes. In §6.2, the authors interpret the absence of incorrect certification markers in 📍Figure 2 and 📍Figure 3 as supporting the certification guarantee. However, the experiments do not report variability across repeated independent trials. Although such repetitions are not necessary to establish the theoretical guarantee, they would provide additional evidence of the stability and empirical reliability of the procedure.

References

Questions for authors
  1. Choice of fidelity threshold. The experiments use a fixed threshold of ϵ=0.15\epsilon=0.15 across all seven datasets, while the Gaussian kernel bandwidth is selected separately for each dataset. What motivates this threshold, and how should practitioners choose a meaningful fidelity tolerance for a new dataset? Would the certification results be sensitive to different kernel bandwidths or threshold choices?
  2. Candidate selection strategy. The experiments select the top NN generators based on training-set MMD estimates, and the results show that increasing NN can sometimes cause previously certified models to lose certification. How should practitioners choose the candidate set and its size in practice? Is there a principled strategy for balancing the number of candidates against the difficulty of certification?
  3. Practical value of certifying multiple generators. One motivation of the paper is to retain multiple certified generators for subsequent downstream decisions. Could the authors provide a concrete example where retaining a certified subset offers a meaningful advantage over selecting a single generator based on empirical fidelity? This would help clarify the practical benefits of simultaneous certification.
Reviewer_vumWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a way to certify synthetic data generators under fidelity constraints. Given several trained generators and a real holdout set that was not used for training, the authors test, for each generator, whether the squared Maximum Mean Discrepancy (MMD²) between real and synthetic data is below a user-chosen threshold ε. The estimate is turned into a p-value through a known concentration bound, and the p-values go into a family-wise error rate (FWER) procedure, the graphical method of Bretz et al. The generators that pass are then all below the threshold with probability at least 1−α. Experiments use six small GANs on seven simulated datasets and compare the selections with reference MMD² values computed from extra samples. No generator with a reference value above the threshold was selected.

Critical strengths

The question of synthetic data quality is important. People who work with synthetic data, for example in health, often have to choose between several generators, and there is no common way to assess which sets are good enough with an actual error guarantee. Putting "not good enough" as the null hypothesis is the right choice, because the error being controlled is certifying a bad generator. The method also does not depend on the generator type/architecture or on one specific metric, so it is easy to add on top of existing pipelines.

Critical weaknesses
  1. No control of privacy. A generator could copy the original training data and do very well. The assumption here is that privacy requirements are already ensured. That is OK, but it limits the applicability. One would expect utility to be inversely correlated with privacy, so as a minimum, the authors should have discussed this.

  2. Limited novelty. The main result follows directly from standard pieces: turn a concentration bound into a p-value, then run an FWER procedure. This is very close to Learn-then-Test (Angelopoulos et al.), which the paper cites but does not discuss. There, a set of model configurations is tested for whether their risk is above a level, using Hoeffding-type p-values and FWER control. Here the configurations become generators and the risk becomes MMD². The paper should state this connection clearly and explain what is new.

  3. The experiments do not check the main claim. The guarantee is a statement about probability over holdout draws, but each dataset is run once. With about twenty decisions in total, seeing no wrong selection says little about whether the error rate is really below α. I would expect many repeated splits, reporting the empirical error rate and how often truly good generators are selected (power). There are also no baselines, such as simply keeping generators whose estimated MMD² is below ε, or testing each generator without multiple-testing correction, so it is not shown what the procedure protects against. All data is simulated, the data dimension is not reported, and α and the graph weights are not given. Given the healthcare and finance motivation, real tabular dataset is needed.

Other points that matter less for my decision. Figures 2 and 3 are hard to read (the same symbol means different things in the two columns, and model and dataset indices do not match Tables 1 and 2).

Questions for authors
  1. Which α\alpha and graph weights were used, and what is the dimension of each simulated dataset?
  2. If the holdout split is repeated many times, what are the empirical FWER and power?
  3. What does the method add beyond applying Learn-then-Test with MMD² as the risk?
  4. How should ϵ\epsilon be chosen in practice, and would a generator that copies its training data be certified (privacy concern)?
  5. Do you agree that the method would generally favour the least private synthetic sets?
Reviewer_5KbYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper addresses the pertinent question of assessing the fidelity of synthetic data generators. The key idea is use a certification approach, formulated as a multiple hypothesis testing problem. The approach is illustrated by a synthetic data set.

Critical strengths

The problem which the paper addresses is important, and phrasing fidelity as a multiple hypothesis testing framework is interesting and to my knowledge novel. The results are mainly theoretical, and proofs are provided.

Critical weaknesses

The paper is difficult to read; randomness appears at different levels but is just denoted by P.

On p.2 in the introduction, what does P refer to? Is it the probability that the procedure rejects this particular set of generators, or is it the probability that this particular set of generators is chosen from a collection of generators? The analysis suggests the latter but the former would be more informative.

Questions for authors

On p.2 in the introduction, what does P refer to? Is it the probability that the procedure rejects this particular set of generators, or is it the probability that this particular set of generators is chosen from a collection of generators? The analysis suggests the latter but the former would be more informative.

In the literature survey, the use of kernel Stein discrepancies for SDGs is not mentioned; see Xu, Wenkai, and Gesine D. Reinert. "A kernelised Stein statistic for assessing implicit generative models." Advances in Neural Information Processing Systems 35 (2022): 7277-7289. Is there a reason that this work is left out? It seems to give an alternative method to compare to.

In Section 3.1, how are the hypotheses and the indices related? What is I_\alpha? That seems to be a set which is available only after all tests have been carried out.

In Theorem 1, which probability does P refer to? Is it obtained under q_i?

In Algorithm 1, what is p_i? Does the algorithm always return a set of size N or N-1? Is this intended?

Section 3.2, the functions f should be measurable. There is not just one RKHS of real-valued functions; why do you phrase it like this?

Section 4.1: Here it is stated that the probability is with respect to the held-out data. Why would this be the interesting probability to measure? Why does the problem then not just reduce to a capture-recapture setting?

Stating that something holds with probability 1-\alpha is not the same as with high probability (whp); whp implies some asymptotics.

In the applications, q_i has to be estimated; how does the estimation error enter the theoretical results? In particular the p-value at the end of Section 5 is not theoretically validated as MMD is estimated in the stated expression, whereas in the theoretical result by Gretton et al. the expression cannot be random.

For the experimental results, what are the underlying data?

Is the recommendation to take a top-N approach or the multiple testing approach?

In the appendix, third line of the proof on p.12: should alpha be both on the left and on the right?

How does the 4th display follow? \hat{theta} is itself random, so the concentration inequaltiy for fixed \theta does not obviously apply. After the Thus, conclusion on that page; sup p_\theta is a number but p_B is random; how can this inequality hold?

On p.13 it seems that I_\alpha is different to the one earlier; the hyptheses are H_1, \ldots, H_N but what are the alternative hypotheses?

Table 2: these seem to be synthetic data; why are they labelled as real data?

How do you construct the 95% interval for Figure 4; how is the randomness of estimating q_i taken into account? Is this an asymptotic confidence interval? If so, how does that agree with the finite-n premise that motivated the paper?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.