- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies membership inference leakage in tabular diffusion models (ClavaDDPM and Tabdiff) in white-box and black-box settings. It uses two attacks from MIDST challenge - Tartan Federer and Ensembe approaches. They vary factors such as training set size and steps, diffusion timestamps, batch size and ratio of synthetic to training data. For clavaDDPM, they also weaken attacker assumptions through shadow models, mismatched shadow configs, and shifted or noisy shadow-model data. Experiments are conducted on Berka and Diabetes datasets, reporting TPR at FPR of 0.1 and DCR, along with other heuristic privacy metrics. The paper's main contribution is an empirical study, as it takes SOTA MIAs and uses them to measure how much models leak under distinct settings and what factors are important. The privacy risk follows the usual overfitting factors like more training, bigger models and less training data mean more leakage. An interesting finding is that generating more synthetic rows relative to training set sharply raises attack success, while DCR could stay flat.
Critical strengths
This performs a broad and controlled privacy study that uses SOTA attacks rather than distance heuristics. It covers two tabular diffusion models under white-box and black box settings, and a variety of training and release choices. The high-level effects are clear in the plot results, and it gives practical and concrete evidence that heuristic metrics can miss leakage. The attacker resources experiments are useful, as it weakens the common argument that idealized attack settings overstate practical risk.
Critical weaknesses
Some concerns are the following:
- quantitative conclusions lack statistical support, and some factors are confounded. Multiple runs and deviations over those would be helpful to gain confidence in empirical results, as some conclusions are drawn based on small differences. A quantitative correlation between DCR and MIA success would be beneficial.
- Scope of general claims: the ensemble attack, experiments on attacker compute, shadow model and data mismatch are run only on ClavaDDPM. TabDiff seems to behave different on batch size results. The conclusion that large compute, perfect knowledge and IID data are not prerequisites for successful attacks seems to rest only on one model.
- The paper mentions Yao et al ("The DCR delusion: Measuring the privacy risk of synthetic data"), which appears to have already ran the MIDST attacks on ClavaDDPM and found no correlation between DCR and MIA success. It would help clarify what are the differences between these results.
Overall, I think this is a useful and timely empirical study. However, the main contributions lack novelty or empirical surprises, that are sufficient to justify high-quality submission.
Questions for authors
- Could the authors report numebr of seeds, standard deviations, data splits, especially for the shadow-count and data-mismatch results? It would clarify whether the smaller effects are real.
- It would be useful to see a quantitative association between DCR and MIA success across settings, such as a rank correlation
- Could the authors report TPR at lower FPRs or full ROC curves for the main levers? I would like to know whether the conclusions hold at the operating points most relevant to worst-case privacy
6 comments
The paper mentions Yao et al... It would help clarify what are the differences between these results.
We evaluate a different DCR formulation. Yao et al. evaluate a percentile-based privacy test comparing synthetic-to-training and holdout-to-training nearest-neighbor distances, as well as a continuous measure based on mean synthetic-to-training distance. In contrast, we use the holdout-based DCR introduced by Zhang et al. [1], which measures the proportion of synthetic records closer to training than holdout records, thereby accounting for differences in local data density. We show that even with this additional consideration, DCR does not reliably reflect membership leakage.
We systematically investigate whether this density aware DCR tracks changes in membership leakage across different training and synthesis configurations. While Yao et al. establish that DCR is unreliable as a standalone privacy measure, we further examine whether it reflects the direction and magnitude of changes in attack-measured risk. Our results show that changes in DCR do not consistently correspond to changes in MIA success.
We also extend the analysis to other empirical privacy metrics (Appendix E). Metrics such as NNDR, Hitting Rate, and ε-Identifiability Risk [2-6] are used in the literature to assess privacy risks in synthetic data, including comparisons between generative models. For example, Lee and Min (AAAI 2025) use NNDR and DCR to compare the privacy protection of CG-TGAN and TabDDPM. However, our experiments demonstrate that these metrics do not necessarily reflect membership leakage. Across model architectures, NNDR remains relatively stable (0.28–0.33), while MIA success increases from approximately 12% to 26% TPR@10% FPR. Similarly, increasing the synthetic dataset size from 1× to 1M leaves NNDR nearly unchanged (approximately 0.35), while MIA success increases from 24% to 35%. (Figure 9)
It would be useful to see a quantitative association between DCR and MIA success across settings, such as a rank correlation.
We thank the reviewer for this suggestion. We computed Spearman's rank correlation between MIA success (TPR@10% FPR) and the absolute deviation of DCR from the ideal baseline across the reported training dataset sizes. Correlations are calculated separately for 5 independent seeds for the TF attack for Clavaddpm and tabdiff and 3 independent seeds for Ensemble.
| Attack | ClavaDDPM (Berka) | ClavaDDPM (Diabetes) | TabDiff (Berka) | TabDiff (Diabetes) |
|---|---|---|---|---|
| TF (BB) | 0.850 | 0.900 | -0.050 | -0.500 |
| Ensemble (BB) | 1.000 | 0.800 | -- | -- |
While DCR shows strong positive correlations for ClavaDDPM, this relationship does not hold for TabDiff. Moreover, rank correlation does not capture the magnitude of membership leakage. These results further highlight the limitations of DCR as a general privacy measure.
Could the authors report TPR at lower FPRs or full ROC curves for the main levers?
We thank the reviewer for this suggestion. We evaluated our white-box attacks at 1% FPR across the main experimental settings and observed that the trends reported at 10% FPR generally persist. As an example, we report results across training dataset sizes for TabDDPM and TabDiff on Berka and Diabetes, averaged over 5 independent runs.
TPR@1% FPR (%) across training dataset sizes
| Dataset | Model | 5K | 10K | 20K | 50K | 100K |
|---|---|---|---|---|---|---|
| Berka | ClavaDDPM | 44.8 | 39.1 | 30.4 | 5.9 | 3.1 |
| Berka | TabDiff | 5.8 | 3.2 | 3.4 | 3.1 | -- |
| Diabetes | ClavaDDPM | 74.3 | 50.7 | 1.0 | -- | -- |
| Diabetes | TabDiff | 29.67 | 14.33 | 6.33 | -- | -- |
These results support our findings at 10% FPR, showing that increasing training dataset size generally reduces membership leakage even at stricter operating points. We also observed consistent trends for the other main experimental levers.
We report representative numerical results here. We would be happy to provide additional results if the reviewer finds them useful.
[1] Zhang, Hengrui, et al. “Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space.” International Conference on Learning Representations (ICLR), 2024
[2] Zhao, Zilong, et al. "CTAB-GAN: Effective Table Data Synthesizing." Asian Conference on Machine Learning (ACML). PMLR, 2021.
[3] Lee, Seungcheol, and Moohong Min. "CG-TGAN: Conditional Generative Adversarial Networks with Graph Neural Networks for Tabular Data Synthesizing." Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 17, 2025.
[4] Trudslev, Frederik Marinus, et al. "PrivEval: A Tool for Interactive Evaluation of Privacy Metrics in Synthetic Data Generation." Proceedings of the VLDB Endowment, vol. 18, no. 12, 2025, pp. 5271–5274.
[5] Lautrup, Anton D., et al. "SynthEval: A Framework for Detailed Utility and Privacy Evaluation of Tabular Synthetic Data." Data Mining and Knowledge Discovery, vol. 39, no. 1, Article 6, 2025.
[6] Yoon, Jinsung, Lydia N. Drumright, and Mihaela van der Schaar. “Anonymization Through Data Synthesis Using Generative Adversarial Networks (ADS-GAN).” IEEE Journal of Biomedical and Health Informatics, 24(8), 2378–2388, 2020.
Dear Reviewer Pn86,
We thank you for your feedback and recognizing the breadth of our empirical evaluation, the use of state-of-the-art attacks, and the practical significance of our findings on privacy metrics and attacker resource requirements. Below, we address your questions.
On the number of seeds. We report the number of independent runs and maximum standard errors (SE) of TPR@10% FPR for each experimental setting below.
Experimental setup: Across independent seeds, data splits are randomized and constructed from scratch, target and shadow models are retrained, and challenge points are independently sampled. Attack classifiers are also independently trained.
For the distribution-mismatch experiments, IID and non-IID adversary datasets are independently generated for each seed, following the same data-splitting and model-training procedures described above.
For the Ensemble attack, we use 3 independent seeds due to its computational cost, varying data splits, target and shadow models, challenge points, and attack training.
Additionally, during the rebuttal period, we repeated the TabDiff training dataset size experiments 5 times for the white-box attack on both Berka and Diabetes. All datasets were independently constructed, and models were retrained from scratch. The resulting mean TPR@10% FPR differed from the originally reported results by at most 0.01, with standard errors below 0.02.
The conclusion that large compute, perfect knowledge and IID data are not prerequisites for successful attacks seems to rest only on one model. Our intention in reporting the attacker-compute, shadow-count, and data-mismatch experiments on ClavaDDPM was to challenge the assumption that favorable attacker conditions, such as substantial compute, perfect knowledge, and distribution-matched data, are necessary for successful MIAs. These experiments demonstrate that substantial membership leakage remains possible even when these assumptions are relaxed.
For example, in the ClavaDDPM/Berka noisy adversary data experiment, the adversary's data differs substantially from the target's training distribution. We quantify this mismatch through differences in the correlation matrices of numerical features, mutual-information matrices of categorical features, and statistical tests comparing feature marginals.
Despite this distribution mismatch, both white-box and black-box attacks remain effective. (TPR at 10% FPR, reported as mean ± standard error across 5 runs.)
Even when all feature marginals differ significantly, TF-WB achieves 39.16% TPR and TF-BB achieves 24.78% TPR, compared with 10% at chance level. Similarly, our shadow-count and model-configuration mismatch experiments demonstrate that successful attacks remain possible with limited attacker compute and imperfect knowledge of the target model.
We acknowledge that the attacker-resource and data-mismatch results are specific to ClavaDDPM and do not establish architecture-independent behavior. However, they demonstrate that typical assumptions about the attacker access are not always necessary for successful membership inference.
We would also like to emphasize that batch size is the only training or synthesis factor for which we observe substantially different trends between ClavaDDPM and TabDiff.