acceptodds
Under review as a conference paper at ICLR 2027

Beyond Augmentation Performance: A Utility, Attribution, and Privacy Audit for Synthetic Clinical Spectra

Abstract

Generative models for small clinical datasets are commonly evaluated by adding synthetic samples to real training data and measuring downstream performance. This mixed-training benchmark can confound generator quality with class rebalancing, does not isolate the contribution of generated samples, and provides no evidence of patient-level privacy. We address these limitations with a structured empirical audit that evaluates three distinct properties: utility through train-on-synthetic, test-on-real substitution under prespecified equivalence testing; attribution through reverse transfer and zero-signal controls; and privacy through membership-inference and copy-rate gates. On ATR-FTIR, a latent-diffusion generator with near-chance synthetic-to-real transfer matches its corrected counterpart in augmentation AUC, showing that mixed-training performance can conceal generator failure. Across ATR-FTIR (\(n=584\)) and LIBS (\(n=112\)), linear PCA-Gaussian matches or exceeds the individual nonlinear generators, while mixtures improve substitution stability. Post-hoc corrections create apparent utility even from label-shuffled or noise inputs, whereas SMOTE shows strong utility yet suffers from near-perfect membership discrimination under the evaluated distance-based attack. In the end we produce fully synthetic artifacts with a classifier passing equivalence criteria with a one trained on real data. However, a post-hoc full-spectrum attack detects membership signal in the assembled LIBS artifact (\(AUC=0.58\), \(p=0.004\)), showing that empirical privacy depends on the adversarial representation. Thus, augmentation performance alone does not establish generator utility; utility, attribution, and privacy require separate evaluation. Our audit is a falsification framework, not a formal privacy guarantee, clinical validation, or certification of release safety.

Primary area:
datasets and benchmarks
Keywords:
synthetic data release, train-on-synthetic-test-on-real, membership inference, equivalence testing (TOST), latent diffusion, privacy–utility trade-off, clinical spectroscopy, FTIR, LIBS, data anonymisation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_47WNOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses potential issues that can arise from validating a model on mixed real synthetic data. They argue that existing models do not seperately evaluate the impact of incorporating synthetic data into their validation loop. While training on synthetic and testing on real data is generally more effective, some datasets (especially medical ones) are too small to test the model alone.

Using two existing models, FTIR and LIBS, the authors demonstrate their multi-step model evalutation process to determine whether test AUC determined from mixed data is reliable.

  1. Substitution Utility: Compare Test AUC on Synthetic vs Real data
  2. Attribution: Checks if the generated data is more accurate than it's permutation or random noise 3.Empirical Membership risk: tests proximity of generate samples to real samples (goal is to have low attack accuracy)

Importantly, each of these gates only refutes a model, but does not guarantee validity because the gates are constructed with superficial prespecified bounds.

The experiments show that FTIR can have the same AUC for models that fail and pass these gates. This is evidence that additional tests like those proposed in this paper are necessary. They also apply post-hoc corrections to improve the FTIR model. The LIBS model could not be perfectly corrected without failing the membership risk gate.

Critical strengths

This paper clearly demonstrates the risks of testing a model on mixed data. They are able to identify clear examples of when a model may seem to perform well, but this response is actually comming from zero-signal controls, memorization etc. These concerns are motivated by real medical model examples (FTIR and LIBS), and clearly shown in Figure 1. They identify an easy to follow three gate process to validate whether a model is falling into these traps caused by the use of synthetic data. The authors are careful to state throughout the work that these tests do not show formal guarantees of medical validity, generalization or differential privacy; but rather act as a refutation process to reject invalid models.

Critical weaknesses

Privacy: The weakest component of this paper is the privacy aspect. It is emphazied throughout that they are not computing any privacy guarantees, but why not? The membership inference attacks they implement are commonly and naturally avoided through incorporation of differential privacy, especially when using sensitive medical data. To my understanding, they only tested on unprotected models, to see if the inclusion of synthetic data introduces vulnerability to membership inference attacks. It would be very interesting to see how one could incorporate privacy prserving techniques into this framework, for a more fair comparison. This may be beyond the scope of this work, but it merits discussion, especially since the lack of privacy guarantees is mentioned so frequently. I also think the assumption made in the privacy attacks conducted (e.g. access to training/testing data, access to generator etc.), should be described more explicitely in the main text.

Clarity Issues: -Line 292, AUC_perm and AUC_noise are not clearly defined, this criteria was hardest to follow during reading -Table 2: Unclear what column TOST is, what does it stand for? -Line 465: "unstable LIBS diffusion conditioning remains unexplained." Can you highlight what you might try in the future to address this?

Impact: While the three gates proposed in this work seem to be effective to address the problem, they feel like an incomplete solution. It would be more encouraging if there was evidence that these three gates are exhaustive. If they are not exhaustive, clearly describe what other methods/considerations should be taken.

Questions for authors
  1. How would the empirical membership risk gate be affected if the model was trained using differential privacy? Would the model then automatically pass this gate? Discuss how existing privacy techniques could overlap with this work, besides simply identifying the common risk of data leakage from synthetic data usage.

  2. Is it truly impossible to guarantee validity of a model in this case? If so, why is it impossible? Are these tests exhaustive? What could improve them?

10 comments
AuthorsOpenReview ↗

Weakness 1 (privacy) + Q1

We thank the reviewer for the careful reading and for noting that the paper scopes its claims throughout. The paper audits synthetic cohorts that generators produce from two patient cohorts, one measured by ATR-FTIR and one by LIBS. We ran two new experiments in response, and the revision adopts every clarity suggestion. ✓ marks a 90% interval inside the ±0.05 margin. Section numbers are those of the submission and stay valid in the revision, which adds Section 7.5 (the submitted 7.5 becomes 7.6) and Appendices N to P.

1. Differential privacy and the membership test (weakness on privacy, question 1)

Why the submission reported no guarantee. None of the evaluated generators uses a differentially private mechanism, so there was no guarantee to report. The audit measures the realized risk of a specific synthetic cohort, whereas a guarantee is a property of the mechanism that produced it. The two are complementary, and the revised Sections 7.4 and 9 say how they fit together.

Would a model trained with differential privacy pass the gate automatically? In theory, only for very small privacy budgets. Under ε-differential privacy every membership test obeys a trade-off between true and false positives (Kairouz, Oh and Viswanath, ICML 2015), which caps the attack AUC at e^ε / (1 + e^ε). This cap equals our gate value of 0.55 only when ε is about 0.2 and rises quickly from there (Table R7). At the budgets used in practice the guarantee does not imply that the gate is passed, and passing the gate certifies no value of ε.

What happens in practice. We built a differentially private PCA-Gaussian for FTIR, in which every statistic that the generator needs is released through the Gaussian mechanism at the patient level, and passed it through the unchanged audit (Table R7). At every budget from 1 to 32 it passes the membership test and fails the utility test by a wide margin, because the noise required for the guarantee removes the class structure and pushes the samples away from the data. Differential privacy thus moved the failure from the privacy test to the utility test, and the audit is what shows this cost. The result covers one mechanism and about 400 training patients per split.

How privacy techniques fit into the framework. The audit is agnostic to the generator, so a private generator is one more arm. Substitution checks whether the class structure survived the noise, and the membership test measures realized risk in each representation, which stays informative where the formal cap is close to one. The attribution control matters as well, since a post hoc correction that inserts real class statistics would void a privacy guarantee unless those statistics were privatized too.

Table R7. Differentially private PCA-Gaussian on FTIR. The guarantee is (ε, δ) with δ = 10⁻⁵ at the patient level, from five Gaussian-mechanism releases with equal budgets (class counts, mean, second moments, and class-wise means and variances in the private subspace) composed under zero-concentrated differential privacy. The cap is the value for pure ε. The last row runs the same code without noise, and the real-trained AUC is 0.817.

εCapAttack AUCCopy rateNN ratioSubstitutionΔ [90% CI]
10.7310.4890.00016.80.492−0.325 [−0.377, −0.274]
40.9820.5040.0008.40.432−0.385 [−0.440, −0.331]
8> 0.9990.5030.0006.00.520−0.297 [−0.349, −0.247]
16> 0.9990.5120.0004.90.580−0.237 [−0.286, −0.188]
32> 0.9990.5040.0004.40.585−0.232 [−0.279, −0.185]
no DPnone0.5210.0141.070.784−0.034 [−0.063, −0.007]
AuthorsOpenReview ↗

Weakness (Impact) + Q2

2. Assumptions of the attack

The revised Section 6 states the threat model. The attack targets the synthetic cohort itself, not a downstream classifier, and asks whether the cohort reveals which patients trained the generator. The adversary holds the synthetic cohort and the spectrum of a candidate patient in the audited representation, without access to the generator, its training code, other patients' data or a reference population. Members are the patients who trained the generator of a split, non-members are its held-out patients, and the statistic is the distance to the closest synthetic record. White-box, shadow-model and likelihood-ratio attacks were not evaluated, and the revised Sections 6 and 9 say so.

3. The source of the LIBS full-spectrum signal

This finding concerns the uncorrected representative mixture under a full-spectrum attack (Section 7.4), and it is separate from the post hoc corrections of Section 7.3. The submission traced the signal to the PCA-Gaussian rows, and a new experiment confirms the cause by removing it (Table R8). Fitting the principal-component basis on the training donors places every PCA-Gaussian row inside the span of their spectra. When the basis is fitted instead on the 40 unlabeled donors of the deposit, who are never members and never test patients, the full-spectrum attack falls to chance over the whole ROC curve and at a 1% false-positive rate, while the class-wise means and variances are still estimated from the training donors. The full-spectrum copy rate of the 50% mixture falls below the 0.10 threshold as well, and the mixture keeps its equivalence and meets the task-space thresholds, whereas PCA-Gaussian alone loses some utility and narrowly misses the margin. This remedy needs no noise. It separates the estimation of the subspace from the labeled training data, the dependence moves to the donors who supply the basis, and it therefore applies wherever an unlabeled reference set is available.

Table R8. LIBS, basis fitted on the training donors of each split (as in the paper) or on the 40 unlabeled donors. Substitution uses the primary classifier, and the task-space attack uses the four element features. The full-spectrum attack is the five-split mean with its permutation p-value (null 95th percentile 0.54 to 0.55). All rows come from the same rerun over five splits and five seeds, in which the rows with the training-donor basis differ from Table 2 by about 0.001 in substitution and by up to 0.03 in task-space attack AUC (see Table R6 in our reply to Reviewer 4fAz).

SetBasis fromSubstitution Δ [90% CI]Task-space attackFull-spectrum attackTPR at 1% FPR (× chance)Full-spectrum copy rate
PCA-Gaussiantraining donors+0.014 [−0.015, +0.043] ✓0.5300.704 (p = 0.001)6.90.265
PCA-Gaussianunseen donors−0.025 [−0.050, −0.003]0.5040.489 (p = 0.654)0.90.138
50% mixture (candidate)training donors−0.006 [−0.030, +0.020] ✓0.4950.671 (p = 0.001)5.40.131
50% mixtureunseen donors+0.001 [−0.017, +0.021] ✓0.4940.489 (p = 0.666)0.90.076

4. The prespecified thresholds

The gates are operational tolerances and not validated risk limits, as the paper states. Table R9 collects the context that the paper reports for each of them.

Table R9. The three thresholds and their context in the paper.

ThresholdValueContext
Equivalence margin±0.05 AUCTwo to two and a half times the split-to-split standard deviation of the real-trained classifiers (Section 3), with conclusions re-evaluated from 0.03 to 0.10 (Appendix G)
Attack AUCbelow 0.55Above the 95th percentile of the permutation null of the five-split mean, which is 0.52 for FTIR and 0.54 for LIBS (Appendix I)
Copy ratebelow 0.10Held-out real patients score 0.060 for FTIR and 0.035 for LIBS against the training fold (Appendix I)
AuthorsOpenReview ↗

Clarity and Ethics addressed

The role of the corrections, AUC_perm and AUC_noise (line 292). The post hoc corrections are not used to improve a candidate. They are studied on both cohorts as a test case for the attribution control, both representative candidates are uncorrected, and Section 7.3 now opens with a sentence that says this. The revised Section 6 defines AUC_perm as the substitution AUC of Spectral-LDM samples with permuted class labels, which keep the distribution of the generator and lose their class association. AUC_noise is the substitution AUC of noise rows, drawn as absolute values of standard Gaussian variables and normalized like generated spectra. Both controls pass through the same correction, classifier and evaluation as the arm that they are compared with. Section 6 also states the criterion in two parts. An uncorrected arm must exceed both controls in substitution AUC, and a corrected arm must in addition exceed both in the gain that the correction produces, so that an arm already strong before the correction is not credited for it. None of the ten corrected LIBS arms that satisfy equivalence and the task-space privacy thresholds shows generator-specific gain, and the four built from the nonlinear generators fall below the corrected controls in absolute AUC as well.

TOST (Table 2). Two one-sided tests, the standard procedure for equivalence. An arm receives "yes" when the 90% confidence interval of its AUC difference from the real-trained classifier lies inside ±0.05. The revision introduces the acronym in Section 6 and in the caption of Table 2.

LIBS conditioning (line 465). The revised Section 9 names what we would examine, which is class separability in the latent space of the autoencoder (the generated samples reproduce only part of the real class differences, Appendix D), the instability across seeds, the guidance weight selected on FTIR and applied to LIBS unchanged, and a generator restricted to the emission lines used by the classifier. These are directions for future work.

6. Are the tests exhaustive, and can validity be guaranteed? (impact, question 2)

No to both, and the revised Section 9 says so. An empirical test can refute a claim and cannot certify it. A finite set of attacks bounds leakage from below, and only a guarantee on the mechanism bounds it from above, at the cost shown in Table R7. Equivalence likewise holds for the sampled patients, the stated classifier and the stated margin. The three tests are not exhaustive, but they are not redundant either, since each rejects a set that the other two accept. The margin alone rejects FTIR PCA-Gaussian, the membership test alone rejects SMOTE, and the attribution control alone rejects the corrected LIBS arms built from the nonlinear generators. Neither standard fidelity metrics nor mixed training can stand in for these tests (new Section 7.5, Table R2 in our reply to Reviewer tBw2). Section 9 now lists what to add before a synthetic cohort is shared, namely attacks in every exposed representation, stronger attack families, attribute-inference and reconstruction attacks, external validation, and checks under secondary classifiers.

7. The flag for ethics review

Both cohorts are public, de-identified deposits on figshare under the CC BY 4.0 license. The study is a secondary analysis, recruitment, consent and ethical approval were obtained by the original investigators, and no new data from human subjects were collected. The membership analyses audit our own synthetic artifacts and target no deployed system and no third-party data. Because the synthetic spectra are computed from public data, they cannot reveal more about any donor than the deposits do, and the LIBS signal is a methodological finding about representations and not a new exposure. The artifacts carry a synthetic-data designation, a provenance manifest and use restrictions, and they are not intended for clinical use (Section 8). The revised Ethics statement adds these points.

Changes in the revision. Section 6 (threat model, definitions, two-part criterion, TOST), Table 2 caption, Section 7.3 (role of the corrections), Section 7.4 and the new Appendix P (differential privacy, basis fitted on unseen donors, Tables 22 and 23), Section 9 (audits and guarantees, tests to add, LIBS directions) and the Ethics statement. Our comment to all reviewers maps every change onto the paper. We would be glad to clarify any point further.

Reviewer_4fAzOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose an auditing of the effects of mixed training of generated samples in medical spectroscopy data and propose an alternative scheme. Specifically, they explore issues of substitution utility, attribution and empirical privacy and do so using using two datasets of FTIR and LIBS with extensive experimentations comparing diffusion models, GANs, PCA-Gaussian generators, and SMOTE controls across multiple patient-level splits and random seeds to support their findings that augmentation performance does not necessary does not translate to generator utility.

Critical strengths
  1. The most notable strength is the first finding of the mixed training setting obscuring the ineffectiveness of synthetic training and consequently generator failure. This is an important finding and strongly contributes to my positive impression.

  2. The analysis of the synthetic AUC as well as the attack AUC, Copy rate of the PCA-Gaussian next to the non-linear generators is a compelling and interesting result as well.

Critical weaknesses
  1. The third finding of the PCA-Gaussian generator might be rather features used in the classifier. For example, in Table 2 FTIR seems to be 235 feature full-spectrum while LIBS seems to be 4 features. It could be that a reduced feature set following the PCA-Gaussian is more easily able to produce better performance.
Questions for authors
  1. Do the authors have any comment on the benefits of their input size and the effectiveness of the simple PCA-Gaussian generator? Could the authors explain why a similar 4 feature classifier was not possible for FTIR as in LIBS?
4 comments
AuthorsOpenReview ↗

Combined response to Q1/W1 part 1

We thank the reviewer for the supportive assessment and for a question that the submission answered only in appendix tables. We ran the experiment that the question suggests, and it has two outcomes. PCA-Gaussian does not owe its standing to a small feature set. Your intuition that the task representation matters is nevertheless correct in a form that the main text did not state, and the revision now states it. Table entries are differences in pooled out-of-fold AUC between a classifier trained on synthetic data and the same classifier trained on real data, and ✓ marks a 90% interval inside the ±0.05 margin. Section numbers are those of the submission and stay valid in the revision, which adds Section 7.5 (the submitted 7.5 becomes 7.6) and Appendices N to P.

What differs between the two cohorts

Table 2 did not say that the cohorts use different primary classifiers. FTIR uses XGBoost on the 235-point spectrum, and LIBS uses logistic regression on four integrated element intensities (Ca, K, Mg and Na). The revised caption states the classifier and the feature space of each cohort.

Does the feature set explain the result for PCA-Gaussian?

On FTIR the primary classifier already uses the full spectrum, and PCA-Gaussian matches the diffusion model there. Table R5 extends the comparison to four further FTIR classifiers, and Table R6 to full-spectrum XGBoost on LIBS. Across these seven combinations of cohort and classifier, PCA-Gaussian has the smallest worst-case gap to real-trained performance among the generators and mixtures. It is also the only one of them whose interval lies inside the margin when the LIBS classifier sees all 24,488 channels, although the lower end of that interval is close to the margin. Its standing therefore does not depend on a reduced feature set.

Your intuition does hold in one specific form. Classifiers on principal components favor PCA-Gaussian by construction, because its samples lie in a low-dimensional principal subspace that such a classifier recovers, and with four components it even exceeds the real-trained classifier slightly. More generally, a very small feature set makes the test easy to pass. With four selected spectral points every generator is equivalent, WGAN-GP included, although the zero-signal controls still fail.

Table R5. FTIR. LR is logistic regression, the four points have the largest class separation inside each training set, and the principal components (PCs) are fitted on each training set.

ArmXGBoost, 235 points (primary)LR, 235 pointsLR, 4 pointsLR, 4 PCsLR, 20 PCs
Real-trained AUC0.8170.8460.7320.7040.827
Spectral-LDM−0.037−0.213−0.022 ✓−0.043−0.067
WGAN-GP−0.091−0.136−0.006 ✓−0.008 ✓−0.108
PCA-Gaussian−0.032−0.073+0.009 ✓+0.032 ✓−0.016 ✓
25% mixture (candidate)−0.006 ✓−0.217−0.006 ✓−0.044−0.052
50% mixture−0.015 ✓−0.234−0.007 ✓−0.025 ✓−0.044
SMOTE (control)+0.023 ✓−0.040+0.015 ✓+0.001 ✓+0.001 ✓
Permuted control−0.292−0.319−0.190−0.171−0.267
Noise control−0.288−0.334−0.239−0.210−0.327

Table R6. LIBS. The second column comes from a rerun. Arms with PCA-Gaussian rows vary slightly between runs, by about 0.03 in split-mean AUC under XGBoost and 0.001 under the primary classifier, because the randomized PCA solver used for the 24,488-channel spectra was not seeded, and the revision discloses this.

ArmLR, 4 element intensities (primary)XGBoost, full spectrum [90% CI]
Real-trained AUC0.8900.889
Spectral-LDM−0.327−0.534 [−0.640, −0.425]
WGAN-GP−0.452−0.324 [−0.432, −0.218]
PCA-Gaussian+0.013 ✓−0.009 [−0.044, +0.025] ✓
25% mixture−0.033−0.087 [−0.145, −0.035]
50% mixture (candidate)−0.006 ✓−0.044 [−0.087, −0.006]
SMOTE (control)+0.007 ✓−0.012 [−0.044, +0.017] ✓
AuthorsOpenReview ↗

Combined response to Q1/W1 part 2

Why a four-feature classifier was not used for FTIR

Three reasons led us to keep the full spectrum. First, each cohort keeps the feature space of its source study, which was four integrated element intensities for LIBS and the whole preprocessed fingerprint spectrum for FTIR. Second, the four LIBS features are physically defined. LIBS is an atomic emission technique whose lines are narrow and belong to single elements, whereas mid-infrared absorption of plasma is molecular, with broad and overlapping bands of proteins, lipids, carbohydrates and nucleic acids, and it offers no agreed four-band summary for this task. Third, the parsed FTIR deposit lacks the wavenumber axis of the instrument (Section 3), so features defined by band position were not available to us. Table R5 shows what a data-driven alternative costs. Selecting the four most discriminative points inside each training set lowers the real-trained AUC from above 0.8 to about 0.73, and the test then no longer separates the generators from one another.

Classifier dependence and downstream inductive bias (now in Section 7.2)

The equivalence of the representative candidates holds under the primary classifier of each cohort and not under every classifier. Under full-spectrum logistic regression, which has the highest real-trained AUC on FTIR, no synthetic set reaches equivalence, and the FTIR mixture falls short by about 0.2. The mixture also misses the margin under the principal-component classifiers, and the LIBS mixture misses it under full-spectrum XGBoost. The submission reported the logistic-regression results in its full grid (Table 8, now Table 6) and noted in Appendices A.9 and D that linear classifiers transfer worse than trees on diffusion samples, but the main text did not draw the consequence for the candidates.

Table R5 also shows a pattern. For linear classifiers the gap grows with the number of features that the classifier uses, from four points to twenty components to the full spectrum, and it does so for every generator, whereas a tree ensemble on the same 235 points loses far less. One reading is that a linear model on the full spectrum depends on covariance across many points, which the generators reproduce only in part, while trees rely on a few points at a time. We have not tested this explanation directly.

Substitution utility is thus conditional on the classifier, in the same way that the privacy result is conditional on the representation. XGBoost has been the primary FTIR classifier since the start of the project, before any classifier was trained on synthetic data alone, and the LIBS classifier follows the source study, so the confirmatory analysis does not rest on a choice made after seeing these results.

Changes in the revision. Table 2 caption (classifier and feature space), Section 7.2 (classifier dependence of the candidates, robustness of PCA-Gaussian), Sections 8 and 9 (artifacts eligible for the stated classifier and representation) and the new Appendix O (Tables 20 and 21, which add intervals to Tables R5 and R6). Our comment to all reviewers maps every change onto the paper. We are grateful for the question, which made the paper more precise about what its candidates do and do not establish.

Reviewer_4fAzOpenReview ↗

Thank you for the response

I thank the authors for their detailed responses to my questions. While I choose to not increase my score already positive initial evaluation, I will maintain it and wish the authors all the best for their submission.

AuthorsOpenReview ↗

You are very welcome

We hope that we addressed your concerns regarding the paper's soundness. Feel free to ask any remaining questions to the current revision if you have them. During the first response phase we have run extensive extra tests on the data, so we are confident, that we could give you more information if that would be required. A residual window of ~1 day still gives us plenty opportunities to continue the discussion

Reviewer_tBw2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that for small clinical spectroscopy datasets, augmentation performance under mixed real and synthetic training cannot isolate generator quality, citing a buggy latent diffusion model (LDM) that performs at chance under TSTR yet matches its corrected version under mixed training. Building on this, the authors propose an audit that evaluates substitution utility, attribution, and privacy as separate gates. They report that a linear PCA Gaussian baseline is competitive with nonlinear generators, that post hoc corrections produce utility even from zero signal inputs, and that privacy conclusions depend on the attack representation.

Critical strengths
  • The zero signal controls establish an explicit floor that any claim of generator utility must exceed. By passing Gaussian noise and label shuffled samples through the same pipeline, the paper defines a lower bound that a generator must clear before its score can be credited to the generator. Under post hoc correction this floor rises to the level of real data training. This shows concretely that corrected scores should not be read as evidence of generator quality.
  • The statistical evaluation is careful and well designed. The study uses patient level splits, TOST equivalence testing with a prespecified margin, multiplicity correction, and margin sensitivity analysis. In small cohorts where leakage and overinterpretation are common, this design increases confidence in the reported numbers themselves.
Critical weaknesses
  • The motivating problem is narrow and the methodological novelty is limited. Separate evaluation of fidelity, utility, and privacy, together with TSTR, is already standard practice in synthetic data research. It appears both in method papers on synthetic data generation [1] and in established benchmark frameworks [2]. Judging generators by mixed training performance alone applies only to some of the spectroscopy studies the paper cites, such as McHardy et al. (2023). The problem is therefore not compelling as a general ML question, and apart from the zero signal controls the contribution largely reconfirms what is already known.
  • The central demonstration relies on low quality generators and does not establish the need for the proposed audit. The broken LDM is immediately exposed by basic distance based fidelity diagnostics, with a nearest neighbour ratio of 9.8 compared with 0.89 after correction. The demonstration thus shows only that standard fidelity checks suffice, not that the proposed audit is needed. Even the corrected LDM at the center of the demonstration is easily separated from real spectra by a real versus synthetic discriminator (AUC 0.998, Appendix A.2). The demonstration therefore contrasts two low quality versions of the same generator.

Reference

  • [1] H. Zhang et al. Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space. In International Conference on Learning Representations (ICLR), 2024.
  • [2] Z. Qian et al. Synthcity: a benchmark framework for diverse use cases of tabular synthetic data. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2023.
Questions for authors
  • Can you provide a generator that performs well under mixed training evaluation and passes standard fidelity measures (discriminator AUC, nearest neighbour ratio, alpha precision and beta recall [3], and so on) yet fails only TSTR or the attribution check?
  • In Figure 1(a), every generator shows no augmentation gain over real data alone (0.796) and falls below SMOTE (0.823). If no generative augmentation helps in this setting and the two LDM configurations are also indistinguishable, is this not a consequence of the setting rather than a limitation of the benchmark?

Reference

  • [3] A. M. Alaa et al. How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models. In International Conference on Machine Learning (ICML), 2022.
9 comments
AuthorsOpenReview ↗

Weakness 1 Addressed

We thank the reviewer for the careful reading and for recognizing the zero-signal controls and the statistical design. The two questions led to new experiments that sharpen the paper in two places. The first shows that standard fidelity metrics and the mixed benchmark both accept synthetic sets that carry no class information. The second maps mixed training across data-scarcity regimes. It confirms your reading that scarcity lets the benchmark register a harmful generator, and it shows that in no regime does mixed training credit the corrected generator for the class structure that substitution and reverse transfer measure. AUC values are pooled out-of-fold over the 487 FTIR patients with the primary XGBoost classifier, as in Table 2, and ✓ marks a 90% interval inside the ±0.05 margin. Section numbers are those of the submission and stay valid in the revision, which adds Section 7.5 (the submitted 7.5 becomes 7.6) and Appendices N to P.

W1. Novelty and scope

We agree that train-on-synthetic evaluation, fidelity metrics and privacy metrics are established. We had not cited TabSyn, Synthcity or Alaa et al., and the revised Section 2 cites all three. TabSyn reports fidelity metrics that include α-precision and β-recall, a detection test, machine-learning efficiency (training on synthetic and testing on real data) and distance to the closest record. Synthcity adds a wider privacy suite, differentially private generators, and an augmentation use case scored by mixed training against an unaugmented baseline.

The audit adds three elements that we did not find in either framework. It replaces a score with a decision, which is equivalence with the real-trained classifier inside a prespecified margin, with uncertainty estimated at the patient level. It adds the zero-signal controls that you credit, which test whether utility comes from the generator or from statistics inserted after generation. It also treats membership risk as a gate that utility cannot offset, audited in more than one representation of the same artifact. The submission already contrasts this with an aggregate benchmark (Appendix I). The harmonic-mean score of Sella et al. (2025) is zero for the LIBS candidate although its rows train an equivalent classifier, and controls that contain no individual donor receive a clearly nonzero identifiability score. The new experiment under Q1 adds that the fidelity suite of these frameworks does not replace any of the three tests.

On scope, we re-read the five spectral-generation studies cited in Section 2. Each of them evaluates generated spectra by adding them to real training data, and we found neither synthetic-only training nor a privacy analysis in any of them (Table R1). The practice examined in Section 4 is therefore the norm in this application area and not specific to McHardy et al. Clinical spectroscopy motivates our cohorts, but the failure modes arise from the evaluation design and not from spectra. Label-agnostic fidelity metrics cannot see a label permutation by construction, the class-conditional ones moved by less than 0.06 under it, and the task-space and full-spectrum attacks disagreed on the same artifact. We therefore expect these failures in other small cohorts evaluated the same way, and the revised Section 2 says so. We submitted to the datasets and benchmarks area because the contribution concerns evaluation design, falsification controls and audited artifacts with provenance records.

Table R1. How the five studies cited in Section 2 evaluate generated spectra.

StudySpectraEvaluationSynthetic-only trainingPrivacy analysis
McHardy et al. (2023)ATR-FTIR of serum, cancerCNN on real plus WGAN spectra, against real onlyNoNo
Chung et al. (2024)Sensor spectra of hydrogelsCNN on real plus generated minority samplesNoNo
García-Esteban et al. (2023)Radiative heat-transfer spectraNetwork on real plus conditional-WGAN spectraNoNo
Flanagan and Glavin (2024)Raman, solvent mixtures and SARS-CoV-2 serumTwo deep models on real plus blended or VAE spectra in growing batches, against real onlyNoNo
Yao et al. (2024)Bacterial RamanClassifier on real plus diffusion spectra, against real onlyNoNo
AuthorsOpenReview ↗

W2. The demonstration relies on low-quality generators

Both observations are correct, and the paper does not present the corrected generator as a high-fidelity model, since it fails equivalence on its own (Table 2). The failed configuration had two defects at once. Its samples lay far from the data, which the nearest-neighbor ratio detects, and they carried no class information, which that ratio cannot detect. The second defect also occurs without the first, as the answer to Q1 shows. The only result that passed the equivalence test was a mixture of two different generators

Two further results address the concern that the demonstration contrasts two weak versions of one generator. First, the insensitivity of mixed training is not specific to that pair. At full cohort size the mixed benchmark places all seventeen synthetic sets that we evaluated within about 0.02 of real-only training, including noise and off-manifold WGAN-GP samples, while their substitution AUC ranges from chance to above the real-trained value (Table R2, and Table 18 in the new Appendix N for all seventeen sets). Second, higher fidelity does not make the audit redundant but changes which test does the rejecting. The only set that a discriminator cannot tell from real spectra is SMOTE, which fails the membership test. The most realistic generator by the discriminator and by α-precision is PCA-Gaussian, which narrowly fails equivalence, whereas the less realistic 25% mixture passes.

The discriminator values are now computed under one protocol for all sets, with cross-validation on equal-sized real and synthetic samples and early stopping on a validation part of the training folds. They are lower than those of Appendix A.2 and keep the same ordering, and the revised A.2 reports them.

Table R2. Fidelity, mixed training and the audit on FTIR over five patient-level splits. Fidelity is measured against real training spectra, with held-out real patients as the reference. Disc. is a cross-validated real-versus-synthetic discriminator, for which 0.5 means indistinguishable, and α-precision and β-recall follow Alaa et al. (2022). Mixed training adds minority-class rows at 1.0× the class deficit, and real-only training gives 0.817. The last column names the audit tests that reject each set, and the new Appendix N adds attack AUC and copy rate.

Synthetic setDisc.α-prec.β-recallNN ratioMixedSubstitutionRejected by
Held-out real patients (reference)0.4740.9160.4921.01
25% mixture (candidate)0.9090.7620.4400.970.8210.811 ✓none
25% mixture, labels permuted0.9090.7620.4400.970.8030.497substitution
Spectral-LDM0.9920.6830.4380.940.8150.780substitution, copy rate
Spectral-LDM, unconditional (w = 0)0.9940.5100.4580.850.7970.492substitution, copy rate
PCA-Gaussian0.8080.8560.4351.070.8250.785substitution (narrowly)
WGAN-GP0.9980.0110.0005.60.8160.725substitution
Noise (control)0.9940.0000.00028.30.8250.529substitution
Spectral-LDM, covariance-corrected0.9370.6480.8060.760.8210.799 ✓attribution, copy rate
Permuted Spectral-LDM, covariance-corrected (control)0.9410.6250.8260.750.8260.780substitution, copy rate
Noise, covariance-corrected (control)0.8190.5960.1461.290.8260.786substitution (narrowly)
SMOTE (control)0.4840.8070.8220.310.8260.840 ✓membership
AuthorsOpenReview ↗

Q1. A generator that passes mixed training and standard fidelity measures, yet fails only TSTR or the attribution check

Short answer to Q1: yes. Table R2 shows the substitution case on FTIR, and Table R2 together with the LIBS grid shows the attribution case. On these cohorts the discriminator separates every generator from real spectra, with AUC between 0.81 and 1.00, including the mixture that trains an equivalent classifier. It accepts only SMOTE, which fails the membership test. As an absolute criterion it would therefore reject the mixture that passes the audit and accept the set that leaks, so fidelity can be compared here only in relative terms. With this understanding, Table R2 contains the following cases.

Same fidelity as the candidate, fails only substitution. The 25% mixture with permuted class labels consists of the same spectra as our representative candidate. Every label-agnostic metric is identical by construction, the privacy thresholds are met, and mixed training stays inside the equivalence margin, whereas substitution AUC drops to chance. A silent loss of conditioning gives the same picture. Diffusion samples drawn without conditioning keep the discriminator AUC and β-recall of the conditional model, mixed training again stays inside the margin, and substitution falls to chance.

Passes mixed training and equivalence, fails attribution. After covariance correction, Spectral-LDM passes mixed training and equivalence, with better discriminator AUC and β-recall than the uncorrected model. The label-permuted copy of the same samples, corrected in the same way, obtains the same fidelity scores and the substitution AUC of the uncorrected generator, although it carries no generator class signal. The correction adds about 0.26 AUC to these permuted samples and about 0.02 to the generator's own samples, so the utility of the corrected arm is the utility of the correction. Fidelity metrics, mixed training and substitution all accept this arm, and the permuted control exposes that its gain came from the correction. Its copy rate also exceeds the 0.10 threshold. On LIBS, where we ran the audit tests and not the fidelity suite, the attribution control is the only test that rejects four corrected arms built from the nonlinear generators. They satisfy equivalence and both task-space privacy thresholds (attack AUC 0.46 to 0.52, copy rate at most 0.03) yet stay below the zero-signal controls that received the same correction (Section 7.3, and Table 8, now Table 6).

Passes the discriminator, mixed training and substitution, fails only membership. This is SMOTE, whose training patients the attack identifies almost perfectly.

The opposite error also occurs. The most realistic generator fails equivalence while a less realistic mixture passes, and Appendix I reports the same disagreement for an aggregate quality score on LIBS.

Class-conditional fidelity does not repair the first case. Computed class by class, the discriminator, α-precision, β-recall and sliced Wasserstein distance change by less than 0.06 under label permutation, while substitution AUC changes by more than 0.25. The gap between real and synthetic spectra dominates these metrics and leaves little room for the class signal that a classifier needs. Authenticity, the third metric of Alaa et al., flags SMOTE and to a lesser degree the corrected diffusion samples, which is consistent with its role as a check on copying. The revised paper reports this analysis in the new Section 7.5 (Table 3), directly after the privacy results, and in the new Appendix N (Tables 18 and 19).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Comment to all reviewers: summary of the revision

We thank the three reviewers and the area chair. Every point of the reviews led to a new experiment or to a clarification, and the revised PDF contains all of them. This comment shows where each change sits in the paper. The second part gives the outcome for each point, the corrections that we made, and what stayed as submitted. Our replies under each review carry the arguments and the result tables.

New analyses. Standard fidelity metrics and mixed training are set against the audit on seventeen FTIR synthetic sets. Mixed training is followed from the full cohort down to one tenth of the training patients. All generators are evaluated under seven combinations of cohort and classifier. A differentially private generator passes through the unchanged audit. The cause of the LIBS full-spectrum membership signal is confirmed by removing it with a basis fitted on unlabeled donors.

What they show. Neither mixed training nor standard fidelity metrics can stand in for the three tests of the audit. Substitution utility is conditional on the classifier, as empirical privacy is conditional on the representation. A formal privacy guarantee and the audit complement each other, since the guarantee bounds the risk and the audit shows what it costs in utility.

Table S1. Numbering. Whatever is not listed keeps its number, and the main text has ten pages.

SubmissionRevision
Section 7.5, calibrationSection 7.6
Tables 3 to 5, which repeated Tables 15, 11 and 16removed
Tables 6 to 15Tables 4 to 13
Table 16Tables 14 to 16
not presentnew Section 7.5, Table 3, Appendices N to P, Tables 18 to 23

Table S2. Changes by location in the revised paper.

WhereChangeRaised by
Abstract, Section 1Claim tied to the full cohort. Fidelity, mixture and LIBS findingsall
Section 2TabSyn, Synthcity, Alaa et al. cited. Practice of the five spectral studiestBw2
Section 3, Table 1Calibration set of the frozen thresholdauthors
Section 4New paragraph on mixed training with few real patientstBw2
Section 6TOST and both controls defined. Criterion in two parts. Threat model47WN
Table 2Caption names the classifier and feature space of each cohort4fAz, 47WN
Section 7.2Mixtures and their stability. New paragraph on classifier dependence4fAz
Section 7.3New opening sentence on the role of the corrections47WN
Section 7.4Remedy for the LIBS signal. New paragraph on differential privacy47WN
Section 7.5 (new), Table 3Fidelity metrics and mixed training against the audittBw2
Section 8Eligibility stated for the primary classifier4fAz
Section 9Linear baseline and mixtures. Audits and guarantees. Tests to add. LIBS directionsall
Section 10Mixture result and LIBS remedy47WN
Ethics statementLicenses of the deposits. Scope of the membership analyses47WN
Reproducibility statementRun-to-run variation of LIBS arms with a PCA fitauthors
Appendix A.2, A.4Discriminator values. Count for each part of the criteriontBw2, 47WN
Appendix A.1, A.5, A.6, A.8, A.9Repeated text and tables replaced by pointers. Unique sentences moved to Appendices D, H and Kauthors
Appendix L, Tables 14 to 16Scarcity analysis rebuilttBw2
Appendix N (new), Tables 18, 19Seventeen sets under fidelity, privacy, mixed training and substitutiontBw2
Appendix O (new), Tables 20, 21Classifier dependence with intervals4fAz
Appendix P (new), Tables 22, 23Privacy bound, private generator, basis from unlabeled donors47WN
AuthorsOpenReview ↗

summary of the revision pt. 2

PointOutcomeWhere
tBw2, noveltyFrameworks cited. The audit adds a decision rule, zero-signal controls and a membership gateSection 2
tBw2, scopeAll five cited spectral studies evaluate by mixed training aloneSection 2
tBw2, low-quality generators, Q1Yes. Fidelity metrics and mixed training accept sets without class informationSection 7.5, Appendix N
tBw2, Q2Confirmed in part. With few real patients mixed training registers harm and never the learned class signalSection 4, Appendix L
4fAz, feature setNot confirmed. PCA-Gaussian has the smallest worst-case gap over seven combinationsSection 7.2, Appendix O
4fAz, four features on FTIRFeature spaces follow the source studies. With four selected points every generator is equivalentAppendix O
47WN, guarantees, Q1A private generator passes the membership test and fails utility at every budgetSection 7.4, Appendix P
47WN, attack assumptionsThreat model statedSection 6
47WN, clarityControls, criterion and TOST defined. Directions for LIBS conditioning namedSections 6 and 9, Table 2
47WN, exhaustiveness, Q2Not exhaustive and not redundant. An audit refutes and cannot certifySection 9
47WN, ethics flagPublic deposits under CC BY 4.0. The analyses audit our own artifactsEthics statement

Strengthened. Three findings of the submission now rest on more evidence. The insensitivity of mixed training holds for seventeen synthetic sets and extends to standard fidelity metrics. The LIBS full-spectrum signal has a confirmed cause and a remedy that preserves equivalence. The linear baseline is the most robust generator across seven combinations of cohort and classifier.

Conditions made explicit. Two claims now carry their condition. The insensitivity of mixed training is stated for the full cohort (abstract, Sections 4 and 9), and the equivalence of the artifacts for the primary classifier of each cohort (abstract, Sections 7.2 and 8).

Table S4. Corrections. No entry of Table 2 changes.

WhereSubmissionRevision
Appendix L, scarcity gainsGenerators and SMOTE fitted on the full training foldFitted on the subset alone, five subset draws. Smaller gains
Sections 6 and 7.3, Appendix A.4Attribution rule on absolute AUC, corrected arms judged by gainCriterion in two parts, with the count for each part
Appendix A.2, discriminator AUC0.998, 0.882, 0.933, 0.9040.992, 0.808, 0.909, 0.839 under one protocol
Table 1, threshold calibration20 to 40 real patients81 real patients
Section 7.1, LIBS PCA-GaussianSplit-mean AUC 0.9150.922, as in Tables 6 and 8
Appendix A.4, gains of the controls0.24 to 0.430.18 to 0.43
Section 2, Flanagan and GlavinAdversarial networkVariational autoencoder
Table 7, captionpre-registered marginprespecified margin

Withdrawn. The submitted scarcity gains are replaced by Tables 14 to 16. The corrected arms no longer "fail attribution" and instead "show no generator-specific gain". The sentence of Section 2 on what earlier studies emphasize gives way to what the five cited studies do. A value for logistic regression on twenty components in Appendices A.9 and D is superseded by Table 20. Sentences that repeated a scope statement were removed for the page limit, and every such statement remains in the abstract, in Section 1, 6, 8 or 9, or in the Ethics statement.

Unchanged. The cohorts, splits, seeds, generators and classifiers, the equivalence margin and the privacy thresholds, every entry of Table 2, both figures and the representative candidates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.