acceptodds
Under review as a conference paper at ICLR 2027

When Does Training on Synthetic Data Provably Work?

Abstract

High-quality training data are a major bottleneck in machine learning, motivating increasing use of generated data. We study when a binary classifier trained on synthetic data transfers successfully to a real distribution, under a shared conditional label distribution. First, we derive finite-sample excess-error bounds when the real covariate distribution is absolutely continuous with respect to the synthetic distribution under a controlled density ratio. We also obtain a bound for Lipschitz hypothesis classes in terms of the Wasserstein distance between the two distributions. Conversely, when the real distribution contains mass outside the synthetic distribution and the hypothesis class is sufficiently expressive, synthetic risk alone cannot uniformly control real-world error: a classifier can be arbitrarily close to optimal on synthetic data while approaching worst-case error on the uncovered real region. We confirm our results experimentally in controlled low-dimensional settings as well as on an application-scale dataset: Using 32,200 images generated as counterparts to real ImageNet images from four state-of-the-art image generators, we show the existence of an adversarial classifier which exhibits up to 0.5% error rate on the synthetic while degenerating to 99.25% on the real domain. At the same time, across a diverse collection of ordinarily trained architectures, synthetic and real test errors remain positively correlated. These results distinguish the worst possible case from typical empirical behavior and suggest that models trained using synthetic data should be rigorously validated on real data.

Primary area:
learning theory
Keywords:
statistical learning theory, synthetic data, error bound, domain gap

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DwvTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the generalization of classifiers trained on synthetic data under covariate shift. It derives excess-risk bounds under density-ratio and Lipschitz assumptions, and characterizes worst-case failures when the real distribution contains regions not covered by the synthetic distribution. The theoretical results are illustrated through synthetic experiments and an ImageNet-based dataset generated using four image generators.

Critical strengths

The paper addresses a relevant problem and presents a reasonably clear theoretical analysis, supported by experiments on both controlled and realistic datasets.

Critical weaknesses
  1. The main theoretical contribution appears limited. Theorem 4 and Corollary 5 essentially characterize the non-identifiability of target-domain behavior outside the source distribution's support, which is closely related to existing impossibility results in domain adaptation. The paper does not clearly establish what is fundamentally new compared with these results. Furthermore, the guarantee in Theorem 7 depends on quantities that are generally unavailable in practice, including the Bayes classifier and the true conditional label distribution.

  2. The experimental evidence does not fully support the broader conclusions. The adversarial classifiers in Section 4.3 are trained using real data with deliberately inverted labels. While this confirms the existence of such classifiers, it does not demonstrate that these failures arise under ordinary synthetic-only training. Moreover, high real-versus-synthetic discrimination accuracy does not establish that the underlying distributions are mutually singular. The conclusions regarding support separation therefore seem overstated.

Questions for authors
  1. How do Theorem 4 and Corollary 5 differ substantively from existing domain adaptation impossibility results?

  2. Can the guarantee in Theorem 7 be made practically applicable without knowing the Bayes classifier or the true conditional label distribution?

  3. Do similar failure patterns arise when models are trained exclusively on synthetic data, without introducing real samples with inverted labels?

2 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for the critical and detailed engagement with our work. Several of the points were also raised in other reviews and motivated substantial changes, which we describe below. We answer point by point. (Theorem, Corollary, Section and Appendix numbers refer to the rebuttal version.)

W1 (novelty). We understand the concern and have gone back through the literature on impossibility results. Our work contains the following points that we could not find in the works we checked, which include the related work of our paper and further papers on domain adaptation:

  1. Necessity and sufficiency of domain separability for the discussed failure mode (good performance on synthetic, poor performance on real data) (Theorem 8).
  2. A bound on the log-likelihood of the failing classifier on the synthetic data, which is the training objective, and not only on its 0/1 error (Theorem 6).
  3. Explicit flip construction of a failing classifier from a classifier and a domain discriminator (Theorem 4).
  4. Explicit construction of a failing function even for Lipschitz models with a concrete distance and mass criterion (Theorem 10).

The work closest in title for the reviewer's question would be David et al. (2010), “Impossibility Theorems for Domain Adaptation”, but its setting differs: It does not treat training with the log-likelihood or Lipschitz models, and it contains no experiments. Its lower bound is also a constant, whereas ours is adaptive to the degree of domain separability.

Concerning the fact that the quantities are unobservable, we argue that population statements are common in learning theory, but we acknowledge that this is unsatisfactory from a practitioner's point of view. Since another review flagged this as well, we reformulated our negative results so that they depend on observable quantities, as detailed under W2.

W2 (discriminability vs. singularity and average vs. worst case). We agree that high discrimination accuracy between real and synthetic images does not establish that the distributions are mutually singular. This was an imprecision on our part. Another review made the same point, and we therefore decided to de-emphasize singularity which was possible because a very large density ratio and true singularity behave similar in practice and so our constructions work also without strict singularity. We reformulated the adversarial construction in terms of a domain classifier and its sensitivity and specificity (Theorem 4). This has two advantages. First, the strict division into singular and non-singular cases is replaced by a continuous dependence on the degree of domain separation. The updated result does not rely on the distributions being fully singular. Second, the construction now depends on observable quantities, which you and other reviewers flagged as a weakness of our results. The experiment with inverted labels in the submitted version is sufficient to show existence, but it required real labels. The experiment was updated to align with the reformulated theorem and real labels are not used directly anymore.

Concerning that ordinary training does not find the adversary, we agree, and we did not claim otherwise. The adversary shows existence, and the goal of our work is to characterize when this worst case can exist, since for safety-critical applications good average behavior is not always enough. Since another review made a similar point, we acknowledge that we did not communicate this clearly enough, and we now stress it more strongly, notably in the paragraph "Existence vs. selection" after Corollary 7. Since several reviews put a strong emphasis on ordinary training, we also extended the correlation experiment on ordinarily trained architectures (Sec. 4.1, Figure 2), but we do want to stress that it is not the main focus of this work.

AuthorsOpenReview ↗

Response 2/2

Q1. (cf. W1)

Q2. Only partially. Since another reviewer asked a similar question, we updated the corresponding section. In Sec. 4.1 and Appendix F, we show how a lower bound on the real mass outside the stability balls can be measured for a relaxed version of the bound. On our data, this bound is uninformative. We show that this is not a defect of the specific bound we give, but that, when the synthetic data lies far from the region where the discriminator predicts real, no bound on the excess error that holds uniformly over all Lipschitz models with a training risk close to the optimum can be smaller than sp−2errP(y∗)\text{sp} - 2 \text{err}_P(y^*), where sp\text{sp} is the specificity of a domain classifier (Corollary 11). We now estimate the quantities involved in both cases (guarantee (Theorem 9) and adversary Lipschitz construction (Theorem 10)) on our generated dataset which was previously not the case.

Q3. Yes, but to a much lesser degree than for the adversaries. In our correlation experiment on ordinarily trained architectures (Sec. 4.1, Figure 2, Appendix B.4) the real error is systematically higher than the synthetic error and grows roughly linearly with it. For the 30 architectures with the lowest synthetic error, which we retrained on a new split, the median gap between real and synthetic error is 8.6 to 17.7 points and the largest is 10.1 to 28.0 points depending on the generator (Table 6). For Stable Diffusion 3.5 Large, all 30 have a gap of at least 10 points, and for HunyuanImage 3.0, 7 of 30 have a gap of at least 20 points. However, no real error reaches 50% (chance level), whereas the adversaries reach gaps of 62 to 88 points (Table 3). So ordinary training shows a systematic gap but does not exhibit the worst case behavior in our experiments.

Reviewer_foidOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies synthetic-to-real transfer in binary classification under the assumption that the real and synthetic covariate distributions PXP_X and QXQ_X share the conditional label distribution p(y∣x)p(y\mid x). It distinguishes three cases by the relationship between PXP_X and QXQ_X: absolute continuity, partial singularity, and full singularity. The theoretical contributions include:

  • Excess classification-error bounds that transfer from QQ to PP under L∞L^\infty or LsL^s density-ratio assumptions (Theorems 1 and 3).
  • An existence result (Theorem 4, Corollary 5): if the hypothesis class can approximate "Bayes on the synthetic support, anti-Bayes elsewhere", a classifier can be near-optimal under QQ and near-worst-case on the singular part of PP.
  • A Lipschitz-based sufficient condition (Theorem 7) that bounds the real excess error by the real mass outside Lipschitz-certified balls around correctly classified synthetic points.

The theory is accompanied by controlled two-dimensional experiments and an ImageNet-derived benchmark (8 classes recreated by four image generators from VLM captions, 32,200 images in total). The experiments show strong real-versus-synthetic discriminability, construct adversarial classifiers with a large cross-domain error gap, and observe a positive correlation between real and synthetic error for 90 ordinarily trained architectures.

Critical strengths

1. Unified perspective on synthetic-to-real transfer

The paper systematically separates absolute continuity, partial singularity, and full singularity, and shows how the relationship between PXP_X and QXQ_X determines whether transfer guarantees are possible. The connection between hypothesis-class expressivity and worst-case failure is useful for understanding the limits of synthetic-only validation: behaviour on a QXQ_X-null set cannot be detected by any criterion computed from synthetic data. The authors also state explicitly that existence of an adversarial classifier does not imply that training finds it.

2. Meaningful distinction between worst-case and ordinary training behaviour

The experiments on ordinarily trained architectures find a positive association between synthetic and real error. This prevents the work from equating worst-case theoretical constructions with typical outcomes and is an important empirical observation. The paired-generation design, which recreates the same real images with four generators, isolates the domain gap from semantic content and is a useful benchmark design.

Critical weaknesses

1. The empirical evidence does not establish that the realistic experiments fall in the singular regime or that synthetic-only training fails (Sections 4.2–4.3, Figure 5, Appendix B.4).

(a) High domain discriminability does not establish mutual singularity. The paper uses the correct characterization

PX⊥QX  ⟺  ∃A∈B(X): PX(A)=1, QX(A)=0.P_X \perp Q_X \iff \exists A\in\mathcal{B}(\mathcal{X}):\ P_X(A)=1,\ Q_X(A)=0.

However, a finite-sample classifier that separates real from synthetic images with 94–99% accuracy (below 100%) establishes a large total-variation distance and strong discriminability, not mutual singularity or failure of absolute continuity. The result may also be confounded by low-level artifacts such as ImageNet JPEG compression and varying resolution versus uniformly sized generated images. Theorem 4 and Corollary 5 require a singular component, so the experiments do not show that the image-generation setting falls into the regime that motivates the negative results.

(b) The adversarial experiment demonstrates existence, not failure of synthetic-only training. The theory establishes the existence of a classifier with

err⁡Q(y^θ)−err⁡Q(y⋆)≤ε,\operatorname{err}_{Q}(\hat{y}_{\theta})-\operatorname{err}_{Q}(y^{\star})\leq\varepsilon,

while, in the fully singular setting,

err⁡P(y^θ)≥1−err⁡P(y⋆)−ε.\operatorname{err}_{P}(\hat{y}_{\theta})\geq 1-\operatorname{err}_{P}(y^{\star})-\varepsilon.

The corresponding experiment, however, trains the adversary on correctly labeled synthetic samples together with real samples with deliberately inverted labels. This is not equivalent to empirical risk minimization on synthetic data alone,

θ^Q∈arg⁡min⁡θ 1n∑i=1nℓ(θ,XiQ,YiQ).\hat{\theta}_{Q}\in\arg\min_{\theta}\ \frac{1}{n}\sum_{i=1}^{n}\ell\left(\theta,X_{i}^{Q},Y_{i}^{Q}\right).

The near-worst-case real error (99.25%) therefore shows that such a classifier can be constructed, not that standard synthetic-only training or its inductive biases tend to produce one. The text acknowledges this, but the abstract and discussion lean on the result more strongly.

(c) The shared conditional-label assumption is not verified. The theory assumes

P(Y=y∣X=x)=Q(Y=y∣X=x),P(Y=y\mid X=x)=Q(Y=y\mid X=x),

but this is not independently checked in the ImageNet experiments. Images generated from automatically produced captions may not preserve the original class or relevant semantic attributes. The observed real-versus-synthetic gap may therefore include conditional-label mismatch rather than arising solely from covariate shift.

2. The theory does not fully match the training objective, and Theorem 7 is only tested adversarially and is hard to evaluate in practice (Sections 3.1, 3.4, Appendix B.4).

(a) 0-1 error versus negative log-likelihood. Training is defined through the negative log-likelihood

RQ(θ)=EQ[−log⁡pθ(Y∣X)].R_{Q}(\theta)=\mathbb{E}_{Q}\left[-\log p_{\theta}(Y\mid X)\right].

Theorem 4 both assumes and concludes small 0-1 disagreement. A bound on the 0-1 excess error,

err⁡Q(y^θ)−err⁡Q(y⋆)≤ε,\operatorname{err}_{Q}(\hat{y}_{\theta})-\operatorname{err}_{Q}(y^{\star})\leq\varepsilon,

does not imply

RQ(θ)−inf⁡θ′RQ(θ′)≤ε,R_{Q}(\theta)-\inf_{\theta'}R_{Q}(\theta')\leq\varepsilon,

because a classifier can make nearly optimal hard predictions while assigning poor probabilities. The statement that minimizing the synthetic risk "cannot rule out this scenario" is therefore unproven for the risk the paper defines, and Corollary 5 inherits the issue. The underlying non-identifiability argument is plausibly repairable (for example, by assuming a sup-norm approximation of f⋆f^{\star} on the synthetic support, or by restating the claim for 0-1 ERM), but as written this is an overclaim.

(b) Theorem 7 states

err⁡P(y^θ)−err⁡P(y⋆)≤PX(X∉Uθ),\operatorname{err}_{P}(\hat{y}_{\theta})-\operatorname{err}_{P}(y^{\star})\leq P_{X}\left(X\notin U_{\theta}\right),

where UθU_{\theta} depends on the agreement between the learned classifier and the Bayes classifier, together with the Lipschitz constants LfL_{f} and L⋆L^{\star}. The experiment in Appendix B.4 trains a network on correctly labeled synthetic samples together with real samples with inverted labels, and selects checkpoints that maximize real excess error subject to a synthetic-error constraint. This shows that the bound is not violated by a constructed worst-case model. It does not show that Lipschitz-constrained models trained exclusively on synthetic data transfer reliably. In addition, the bound depends on the unknown Bayes classifier and true log-odds, and Figure 6 shows it saturating near 1 for large Lipschitz constants, which suggests it would be vacuous for high-dimensional image data.

Questions for authors

**Q1. ** The paper relies on the distinction between PX≪QXP_X\ll Q_X and PX⊥QXP_X\perp Q_X. Given that high domain-discrimination accuracy does not establish singularity, what additional evidence supports the applicability of Theorem 4 and Corollary 5 to the four image generators?

**Q2. ** How would you verify that caption-conditioned image generation preserves the assigned ImageNet labels? Were the generated images independently evaluated for class consistency? Can the resulting label-mismatch rate be quantified per generator?

**Q3. ** Is the V-JEPA adversary fine-tuned or linearly probed? Is the 99.25% real error measured on held-out real images that have no synthetic counterpart in training? Is the task binary or 8-way?

**Q4. ** The adversarial experiment uses real images with inverted labels. Can the authors characterize the frequency and severity of large real-domain failures when classifiers are trained exclusively on correctly labeled synthetic samples, with seeds and confidence intervals? Does the synthetic-real error correlation also hold in the low-error regime relevant to model selection?

**Q5. ** The bound in Theorem 7 depends on agreement with the unknown Bayes classifier and on the true log-odds. How can the coverage probability PX(X∉Uθ)P_{X}(X\notin U_{\theta}) be estimated or bounded from finite synthetic observations? Can the theorem be translated into a practically evaluable training or model-selection guarantee?

2 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for a detailed and constructive review. It led to several substantial improvements, namely new experiments and improved theoretical results, which sharpened the paper considerably. We answer point by point. (Theorem, Corollary and Appendix numbers refer to the rebuttal version.)

W1a (discriminability vs. singularity and image artifacts). We agree that high discriminator accuracy does not establish PX⊥QX.P_X \perp Q_X. This was an imprecision on our part and was also flagged in another review. We therefore decided to restate the negative results in a form that does not rely on singularity (Theorem 4, Corollary 5). This updated form is stated in terms of the sensitivity and specificity of a domain discriminator, which can be measured on held-out data, and it applies to any degree of separation. We thus also de-emphasized the 3-way split in the section names. Regarding format artifacts, we added a variant of the discrimination experiment that keeps only coarse content: both domains are downsampled to 64 pixels and back and JPEG-encoded a second time. The results decrease only slightly, so we think its unlikely the separation is due to such cues.

W1b (existence vs. ordinary training). We agree that the adversary shows existence only. This is the goal of the paper: it characterizes the severity of the worst case in different situations. Since this also was criticized in another review, we do acknowledge that we did not make this clear enough. We now stress this more strongly throughout the paper, notably in the paragraph "Existence vs. selection" after Corollary 7, where we explain why, for safety-critical applications, it is not enough that ordinarily trained models behave well in typical cases, since this does not rule out the worst case. Ordinary training is kept as a separate experiment at the end.

W1c (label consistency). Here, we partly disagree. The shared label map depends on human perception alone and not on anything the generators do. It is a fixed map independent of PXP_X and QXQ_X. This is due to our basic assumption and belief that what an image depicts does not depend on the process that generated it. In fact, PP and QQ are constructed from PXP_X and QXQ_X using the fixed p(y∣x)p(y|x), and not the other way around (obtaining e.g., PXP_X from PP by marginalization). Whether our generators preserve the label should therefore matter neither for the assumption of a shared label map nor for the construction of the adversary. We do, however, recognize that the intention of our experiment was indeed to replicate the images semantically, so checking for label preservation is an appropriate sanity check, and our generated dataset lacked such a validation. We therefore added a corresponding experiment (Sec. 4.1, end of 'Dataset' paragraph and more details in Appendix E) that compares the class of synthetic images with that of their real counterpart using three classifiers. We find that the labels are preserved to a high degree for all generators. Since the labels rest on human perception, we also checked samples of the data manually and found that all generators preserve the semantic content truthfully, though a complete check using human annotators was not feasible at this scale.

W2a (0-1 error vs. log-loss). We agree. This was a genuine gap in our argument. To close it, we added Theorem 6 and Corollary 7, where we repeat the construction of the adversarial model analogously in logit space rather than label space to cover the training objective (negative log-likelihood) and not just the 0/1-error.

W2b (Theorem 7, now Theorem 9). After reading your comment, we tested our bound on our generated dataset and we agree that the bound is vacuous for high-dimensional images. This was a shortcoming of the submitted version. To close this gap, we added a new paragraph and theorem (Theorem 10), where we show that this vacuousness is not a mere defect of our specific bound, but that the adversarial construction can still be carried out in the Lipschitz case (Sec. 4.1 'Lipschitz models', Appendix F) under a distance and mass criterion. We also added two experiments on the realistic data set that estimate bounds on the quantities involved in the guarantee (Theorem 9) and the Lipschitz adversary (Theorem 10). Neither was measured on real data in the submitted version, and we believe the updated paper is now stronger in that regard. Concerning the dependence of the bound on unknown quantities: we agree that this is unsatisfactory from a practitioner's point of view, but we think that population statements are quite common in learning theory and not problematic per se. Since this point appears in several reviews, we nevertheless shifted the adversarial construction to a version that is stated in terms of quantities that can be estimated from data (Theorem 4).

AuthorsOpenReview ↗

Response 2/2

Q1. (See W1a): The negative results no longer rely on singularity but on the measured sensitivity and specificity of a discriminator, for which we now report confidence bounds for all four generators (Sec. 4.1, Appendix C).

Q2. (See W1c): We added a label-preservation experiment per generator (Sec. 4.1, Appendix E). Among pairs whose real image the classifier labels correctly, the class is preserved in at least 98.0% of the cases for three generators and in 96.8% for HunyuanImage 3.0. Over all pairs, regardless of classification, the two zero-shot classifiers assign the same class to the real image and its synthetic counterpart in at least 99.5% of the cases for every generator (Appendix E).

Q3. For the experiment of the submitted version: the encoder was frozen and only a small attentive head was trained on top of it, so the adversary was neither fine-tuned nor linearly probed. The 99.25% real error was measured on held-out real images, and neither these images nor their synthetic counterparts were used in training. The task was binary (bear vs. big cat). This experiment has since been revised. (Sec. 4.1)

Q4. To answer the question about the low-error regime, we extended the correlation experiment and retrained the 30 architectures per generator with the lowest synthetic error on a new split (Sec. 4.1, Figure 2 (d)). None of the 30 comes close to the adversaries. No real error reaches 50% (chance level), and the largest gap between real and synthetic error is 10.1 to 28.0 points depending on the generator (Table 6), compared with 62 to 88 points for the adversaries (Table 3), so ordinary training does not produce the worst case, although the gap is systematic. Concerning the correlation effect: It is substantially weaker in the low error regime (Sec. 4.1, Figure 2 (d)). The R2R^2 among the best candidates goes as low as 0.05 for Stable Diffusion. In this low-error regime the synthetic error does not reliably identify the best model on real data. We also provide rank correlations for the low-error regime in (Table 7).

Q5. Coverage: PX(X∉Uθ)P_X(X \notin U_\theta) cannot be computed without the Bayes log-odds f*, but it can be bounded from below. The radius of each ball is the minimum of ∣f(x)∣/L0|f(x)|/L_0 and a term that depends on f∗f^*. Dropping the second term can only enlarge the balls, so the share of real test points outside the resulting balls is a lower bound on PX(X∉Uθ)P_X(X \notin U_\theta). This is explained now in detail in Sec. 4.1 and computed for our dataset. Model selection: no. This lower bound shows that the theorem gives no useful guarantee for our data, and the reason is indeed that Lipschitz constants of networks tend to be large in practice, so the stability radii are small, as you pointed out in your review. We incorporated this view point and this is now explained in the updated Section 3.5.

Reviewer_6CF3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies the effectiveness how models trained on synthetic data perform when tested on real data. The authors conduct theoritical analysis for three different cases. The first case when the real and synthetic covariate distributions match each other, the second case when there is a partial overlap between them but there are regions which are supported by the real distribution but not covered by the synthetic distribution, and the third (worst) case where the support regions of real distribution and synthetic distribution are disjoint.

Using Lebesgue decomposition, the authors suggest to split P_X (true distribution) and Q_X (generator distribution) into an absolutely continuous part and a singular part w.r.t. Q_X. Consequentially, the target error also can be split into err_P = err_{P^a} + err_{P^s}. When P is absolutely continuous w.r.t Q, theorem 3 shows that error is is bounded. While for the partial continuous case (P is a continuous part and singular part), theorem 4 shows that higher expressivity of F can lead to higher worst case error. for the singular case, there exist an adversarial classifier that has very small error rate on the synthetic data but doesn't generalize to the real data.

In addition to the theoretical analysis the authors have empirical experiments using a toy 2D classification dataset, and as well using 8 ImageNet classes Image classification with 32K images generated using 4 different state-of-art image generation models.

Main finding from the paper, is that authors demonstrate that there exist an adversarial classifier which can have very low (0.5%) error rate on synthetic data but at the same time have 99.25% error on the real data. Also, authors suggest that synthetic data should be used at very high sample count as the error rate goes down with the sample size (not surprisingly !)

Critical strengths
  • The paper touches an important topic with the increasing adoption of the synthetic data in model training.

  • The paper supports its theoretical analysis with empirical evaluation on both toy datasets and real world image classification dataset.

  • The toy dataset experiments and Figures 2 , and 3 are supportive of the theoretical analysis and easy to understand intutively.

Critical weaknesses

The real data experiment on ImageNet images is not convincing, specifically the authors used V-JEPA mode, authors picked it because it is not trained on ImageNet to rule data leakage, and show that a classifier is able to tell the difference between synthetic and real images.

It is not clear how this experiment relate to the paper theoritical analysis. Also authors say "This is surprising, given that many of the generative images are photorealistic. In contrast to the human eyes, it is very easy for a deep neural network to tell real and synthetic images a part."

I don't find this surprising. For two reasons at least: 1) many image generation models include steganographic image watermarking which are invisible for the human eye but can ease with the distinction between real and generated images. 2) conversely, computer vision models are suspect to adversarial examples where very small perturbation noise can have significant impact on the classification model results without being noticeable for a human eye.

That being said, I find the middle figure in Figure 5 (adversarial classifier) to be more related to the paper scope.

Questions for authors
  • Can this analysis/experiments be extended to other domains such as NLP models where the use of synthetic data is becoming more common?
  • Figure 2, shows that with the increasing sample size the error rate goes for the model trained on Q.. does this apply for all three cases ? Specifically it also hold for the fully singular case when the support of P and Q is disjoint ?!
1 comment

We thank the reviewer for the encouraging review. We answer the weaknesses and the questions point by point. (Theorem, Corollary, Section and Appendix numbers refer to the updated version.)

W1 (real-data experiment). We agree that we did not state the connection between the experiment and the theory precisely enough and in fact it can be criticized. We reported high separability as evidence for singular supports, which is not mathematically sound. Since this was flagged in more than one review, we de-emphasized singularity, because exact singularity and a very large density ratio make a difference in theory but hardly in practice. We instead reformulated our results in terms of the observable domain discriminator and its sensitivity and specificity (Theorem 4), so the distinction is now continuous and no longer a clear cut between singular and non-singular. The experiment now measures exactly the quantities that appear in the theorems, so the connection should be much clearer (Sec. 4.1). To test the concern about watermarks and other fine-grained cues, which you and another reviewer raised, we added an adapted pipeline that gives both domains the same format and reduces all images to a low resolution (downsampling to 64 pixels and back), so that only coarse content remains (Sec. 4.1, Appendix C). We found little difference between this version and the original version where this was not accounted for. We therefore believe that the separability does not stem from these details.

Q1. In principle, yes. Our results in terms of a domain discriminator (Theorems 4 and 6) make no assumption on the data type, so they also apply to discrete data such as text and to lower dimensions. However, they are informative only to the extent that a discriminator can separate real from synthetic data, i.e., that its sensitivity and specificity are high. We have not run experiments on language, and we expect that training an accurate domain discriminator is harder there than for images.

Q2. Not in general. If the synthetic data does not cover the real data, our theory shows a reduction of the error with more synthetic data only where the synthetic distribution has mass, and the data does not constrain the model on the uncovered real region (Theorem 4, Corollary 5). Successful training can still be possible if the geometry of the distributions and the inductive biases of the model are favorable. Appendix B.2 (Figure 4) shows this for fully disjoint supports, where training still works to some extent. A word seems to be missing from the question, so we hope we interpreted it correctly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment

We thank all reviewers for their detailed reviews. We made substantial changes to the paper, but we felt it was necessary to take every reviewer's concern into consideration. Below is a short overview, and the individual replies give details. All numbers refer to the updated version.

Theory. A core construction in the submitted version relied on the assumption of mutually singular distributions. Several reviews pointed out that this cannot be verified, and that our experimental evidence for this assumption was insufficient. We took this seriously and updated the result in a way that gets rid of this unverifiable assumption (Theorem 4, Corollary 5). Furthermore, the updated version is based on quantities that can be measured in practice, which also poses an improvement over the original version. We also, due to one of the reviews, added a version of our construction for the log-likelihood training loss rather than just the 0/1-error, which gives more practical relevance to the result, because this is the quantity one actually observes during training (Theorem 6, Corollary 7). A new two-sided bound now shows that domain separability is not only sufficient (that part was already covered), but also necessary for failure (Theorem 8). Also, following another suggestion, we extended a version of our construction to the case of Lipschitz functions, which shows that restricting the model to Lipschitz functions does not prevent failure under a condition that we evaluate on our data (Theorem 10, Corollary 11).

Experiments. The submitted adversary was trained with real images with inverted labels. The new one is built from the discriminator and a classifier trained on synthetic data only, so it does not need real labels directly. We also report confidence bounds for the sensitivity and specificity of the discriminator, a variant of the experiment that removes format cues and fine detail, a check that the generators preserve the class, measurements of the quantities in the Lipschitz results, and an extended study of ordinarily trained architectures (retraining the 30 best on a new split).

Presentation. We clarified that the adversary shows existence and not that training selects it (paragraph "Existence vs. selection"). The toy experiments moved to the appendix to keep the focus on the realistic dataset. And the importance of full singularity was de-emphasized in the structure of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.