- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses potential issues that can arise from validating a model on mixed real synthetic data. They argue that existing models do not seperately evaluate the impact of incorporating synthetic data into their validation loop. While training on synthetic and testing on real data is generally more effective, some datasets (especially medical ones) are too small to test the model alone.
Using two existing models, FTIR and LIBS, the authors demonstrate their multi-step model evalutation process to determine whether test AUC determined from mixed data is reliable.
- Substitution Utility: Compare Test AUC on Synthetic vs Real data
- Attribution: Checks if the generated data is more accurate than it's permutation or random noise 3.Empirical Membership risk: tests proximity of generate samples to real samples (goal is to have low attack accuracy)
Importantly, each of these gates only refutes a model, but does not guarantee validity because the gates are constructed with superficial prespecified bounds.
The experiments show that FTIR can have the same AUC for models that fail and pass these gates. This is evidence that additional tests like those proposed in this paper are necessary. They also apply post-hoc corrections to improve the FTIR model. The LIBS model could not be perfectly corrected without failing the membership risk gate.
Critical strengths
This paper clearly demonstrates the risks of testing a model on mixed data. They are able to identify clear examples of when a model may seem to perform well, but this response is actually comming from zero-signal controls, memorization etc. These concerns are motivated by real medical model examples (FTIR and LIBS), and clearly shown in Figure 1. They identify an easy to follow three gate process to validate whether a model is falling into these traps caused by the use of synthetic data. The authors are careful to state throughout the work that these tests do not show formal guarantees of medical validity, generalization or differential privacy; but rather act as a refutation process to reject invalid models.
Critical weaknesses
Privacy: The weakest component of this paper is the privacy aspect. It is emphazied throughout that they are not computing any privacy guarantees, but why not? The membership inference attacks they implement are commonly and naturally avoided through incorporation of differential privacy, especially when using sensitive medical data. To my understanding, they only tested on unprotected models, to see if the inclusion of synthetic data introduces vulnerability to membership inference attacks. It would be very interesting to see how one could incorporate privacy prserving techniques into this framework, for a more fair comparison. This may be beyond the scope of this work, but it merits discussion, especially since the lack of privacy guarantees is mentioned so frequently. I also think the assumption made in the privacy attacks conducted (e.g. access to training/testing data, access to generator etc.), should be described more explicitely in the main text.
Clarity Issues: -Line 292, AUC_perm and AUC_noise are not clearly defined, this criteria was hardest to follow during reading -Table 2: Unclear what column TOST is, what does it stand for? -Line 465: "unstable LIBS diffusion conditioning remains unexplained." Can you highlight what you might try in the future to address this?
Impact: While the three gates proposed in this work seem to be effective to address the problem, they feel like an incomplete solution. It would be more encouraging if there was evidence that these three gates are exhaustive. If they are not exhaustive, clearly describe what other methods/considerations should be taken.
Questions for authors
-
How would the empirical membership risk gate be affected if the model was trained using differential privacy? Would the model then automatically pass this gate? Discuss how existing privacy techniques could overlap with this work, besides simply identifying the common risk of data leakage from synthetic data usage.
-
Is it truly impossible to guarantee validity of a model in this case? If so, why is it impossible? Are these tests exhaustive? What could improve them?
10 comments
Weakness (Impact) + Q2
2. Assumptions of the attack
The revised Section 6 states the threat model. The attack targets the synthetic cohort itself, not a downstream classifier, and asks whether the cohort reveals which patients trained the generator. The adversary holds the synthetic cohort and the spectrum of a candidate patient in the audited representation, without access to the generator, its training code, other patients' data or a reference population. Members are the patients who trained the generator of a split, non-members are its held-out patients, and the statistic is the distance to the closest synthetic record. White-box, shadow-model and likelihood-ratio attacks were not evaluated, and the revised Sections 6 and 9 say so.
3. The source of the LIBS full-spectrum signal
This finding concerns the uncorrected representative mixture under a full-spectrum attack (Section 7.4), and it is separate from the post hoc corrections of Section 7.3. The submission traced the signal to the PCA-Gaussian rows, and a new experiment confirms the cause by removing it (Table R8). Fitting the principal-component basis on the training donors places every PCA-Gaussian row inside the span of their spectra. When the basis is fitted instead on the 40 unlabeled donors of the deposit, who are never members and never test patients, the full-spectrum attack falls to chance over the whole ROC curve and at a 1% false-positive rate, while the class-wise means and variances are still estimated from the training donors. The full-spectrum copy rate of the 50% mixture falls below the 0.10 threshold as well, and the mixture keeps its equivalence and meets the task-space thresholds, whereas PCA-Gaussian alone loses some utility and narrowly misses the margin. This remedy needs no noise. It separates the estimation of the subspace from the labeled training data, the dependence moves to the donors who supply the basis, and it therefore applies wherever an unlabeled reference set is available.
Table R8. LIBS, basis fitted on the training donors of each split (as in the paper) or on the 40 unlabeled donors. Substitution uses the primary classifier, and the task-space attack uses the four element features. The full-spectrum attack is the five-split mean with its permutation p-value (null 95th percentile 0.54 to 0.55). All rows come from the same rerun over five splits and five seeds, in which the rows with the training-donor basis differ from Table 2 by about 0.001 in substitution and by up to 0.03 in task-space attack AUC (see Table R6 in our reply to Reviewer 4fAz).
| Set | Basis from | Substitution Δ [90% CI] | Task-space attack | Full-spectrum attack | TPR at 1% FPR (× chance) | Full-spectrum copy rate |
|---|---|---|---|---|---|---|
| PCA-Gaussian | training donors | +0.014 [−0.015, +0.043] ✓ | 0.530 | 0.704 (p = 0.001) | 6.9 | 0.265 |
| PCA-Gaussian | unseen donors | −0.025 [−0.050, −0.003] | 0.504 | 0.489 (p = 0.654) | 0.9 | 0.138 |
| 50% mixture (candidate) | training donors | −0.006 [−0.030, +0.020] ✓ | 0.495 | 0.671 (p = 0.001) | 5.4 | 0.131 |
| 50% mixture | unseen donors | +0.001 [−0.017, +0.021] ✓ | 0.494 | 0.489 (p = 0.666) | 0.9 | 0.076 |
4. The prespecified thresholds
The gates are operational tolerances and not validated risk limits, as the paper states. Table R9 collects the context that the paper reports for each of them.
Table R9. The three thresholds and their context in the paper.
| Threshold | Value | Context |
|---|---|---|
| Equivalence margin | ±0.05 AUC | Two to two and a half times the split-to-split standard deviation of the real-trained classifiers (Section 3), with conclusions re-evaluated from 0.03 to 0.10 (Appendix G) |
| Attack AUC | below 0.55 | Above the 95th percentile of the permutation null of the five-split mean, which is 0.52 for FTIR and 0.54 for LIBS (Appendix I) |
| Copy rate | below 0.10 | Held-out real patients score 0.060 for FTIR and 0.035 for LIBS against the training fold (Appendix I) |
Clarity and Ethics addressed
The role of the corrections, AUC_perm and AUC_noise (line 292). The post hoc corrections are not used to improve a candidate. They are studied on both cohorts as a test case for the attribution control, both representative candidates are uncorrected, and Section 7.3 now opens with a sentence that says this. The revised Section 6 defines AUC_perm as the substitution AUC of Spectral-LDM samples with permuted class labels, which keep the distribution of the generator and lose their class association. AUC_noise is the substitution AUC of noise rows, drawn as absolute values of standard Gaussian variables and normalized like generated spectra. Both controls pass through the same correction, classifier and evaluation as the arm that they are compared with. Section 6 also states the criterion in two parts. An uncorrected arm must exceed both controls in substitution AUC, and a corrected arm must in addition exceed both in the gain that the correction produces, so that an arm already strong before the correction is not credited for it. None of the ten corrected LIBS arms that satisfy equivalence and the task-space privacy thresholds shows generator-specific gain, and the four built from the nonlinear generators fall below the corrected controls in absolute AUC as well.
TOST (Table 2). Two one-sided tests, the standard procedure for equivalence. An arm receives "yes" when the 90% confidence interval of its AUC difference from the real-trained classifier lies inside ±0.05. The revision introduces the acronym in Section 6 and in the caption of Table 2.
LIBS conditioning (line 465). The revised Section 9 names what we would examine, which is class separability in the latent space of the autoencoder (the generated samples reproduce only part of the real class differences, Appendix D), the instability across seeds, the guidance weight selected on FTIR and applied to LIBS unchanged, and a generator restricted to the emission lines used by the classifier. These are directions for future work.
6. Are the tests exhaustive, and can validity be guaranteed? (impact, question 2)
No to both, and the revised Section 9 says so. An empirical test can refute a claim and cannot certify it. A finite set of attacks bounds leakage from below, and only a guarantee on the mechanism bounds it from above, at the cost shown in Table R7. Equivalence likewise holds for the sampled patients, the stated classifier and the stated margin. The three tests are not exhaustive, but they are not redundant either, since each rejects a set that the other two accept. The margin alone rejects FTIR PCA-Gaussian, the membership test alone rejects SMOTE, and the attribution control alone rejects the corrected LIBS arms built from the nonlinear generators. Neither standard fidelity metrics nor mixed training can stand in for these tests (new Section 7.5, Table R2 in our reply to Reviewer tBw2). Section 9 now lists what to add before a synthetic cohort is shared, namely attacks in every exposed representation, stronger attack families, attribute-inference and reconstruction attacks, external validation, and checks under secondary classifiers.
7. The flag for ethics review
Both cohorts are public, de-identified deposits on figshare under the CC BY 4.0 license. The study is a secondary analysis, recruitment, consent and ethical approval were obtained by the original investigators, and no new data from human subjects were collected. The membership analyses audit our own synthetic artifacts and target no deployed system and no third-party data. Because the synthetic spectra are computed from public data, they cannot reveal more about any donor than the deposits do, and the LIBS signal is a methodological finding about representations and not a new exposure. The artifacts carry a synthetic-data designation, a provenance manifest and use restrictions, and they are not intended for clinical use (Section 8). The revised Ethics statement adds these points.
Changes in the revision. Section 6 (threat model, definitions, two-part criterion, TOST), Table 2 caption, Section 7.3 (role of the corrections), Section 7.4 and the new Appendix P (differential privacy, basis fitted on unseen donors, Tables 22 and 23), Section 9 (audits and guarantees, tests to add, LIBS directions) and the Ethics statement. Our comment to all reviewers maps every change onto the paper. We would be glad to clarify any point further.
Weakness 1 (privacy) + Q1
We thank the reviewer for the careful reading and for noting that the paper scopes its claims throughout. The paper audits synthetic cohorts that generators produce from two patient cohorts, one measured by ATR-FTIR and one by LIBS. We ran two new experiments in response, and the revision adopts every clarity suggestion. ✓ marks a 90% interval inside the ±0.05 margin. Section numbers are those of the submission and stay valid in the revision, which adds Section 7.5 (the submitted 7.5 becomes 7.6) and Appendices N to P.
1. Differential privacy and the membership test (weakness on privacy, question 1)
Why the submission reported no guarantee. None of the evaluated generators uses a differentially private mechanism, so there was no guarantee to report. The audit measures the realized risk of a specific synthetic cohort, whereas a guarantee is a property of the mechanism that produced it. The two are complementary, and the revised Sections 7.4 and 9 say how they fit together.
Would a model trained with differential privacy pass the gate automatically? In theory, only for very small privacy budgets. Under ε-differential privacy every membership test obeys a trade-off between true and false positives (Kairouz, Oh and Viswanath, ICML 2015), which caps the attack AUC at e^ε / (1 + e^ε). This cap equals our gate value of 0.55 only when ε is about 0.2 and rises quickly from there (Table R7). At the budgets used in practice the guarantee does not imply that the gate is passed, and passing the gate certifies no value of ε.
What happens in practice. We built a differentially private PCA-Gaussian for FTIR, in which every statistic that the generator needs is released through the Gaussian mechanism at the patient level, and passed it through the unchanged audit (Table R7). At every budget from 1 to 32 it passes the membership test and fails the utility test by a wide margin, because the noise required for the guarantee removes the class structure and pushes the samples away from the data. Differential privacy thus moved the failure from the privacy test to the utility test, and the audit is what shows this cost. The result covers one mechanism and about 400 training patients per split.
How privacy techniques fit into the framework. The audit is agnostic to the generator, so a private generator is one more arm. Substitution checks whether the class structure survived the noise, and the membership test measures realized risk in each representation, which stays informative where the formal cap is close to one. The attribution control matters as well, since a post hoc correction that inserts real class statistics would void a privacy guarantee unless those statistics were privatized too.
Table R7. Differentially private PCA-Gaussian on FTIR. The guarantee is (ε, δ) with δ = 10⁻⁵ at the patient level, from five Gaussian-mechanism releases with equal budgets (class counts, mean, second moments, and class-wise means and variances in the private subspace) composed under zero-concentrated differential privacy. The cap is the value for pure ε. The last row runs the same code without noise, and the real-trained AUC is 0.817.