- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies a timely problem (knowledge transfer in PD synthesis data), develops a useful benchmark, and provides some interesting findings. Data is a bottleneck for frontier model training and DP text synthesis is an important method to leverage sensitive corpora currently. However, current evaluations fail to adequately account for knowledge elicitation, superficial matching, and distillation effects. To address these issues, the authors design grounded factual QA tasks to evaluate knowledge transfer by testing model performance on either fictional or post-cutoff knowledge. The generator and downstream model use the same base checkpoint to eliminate distillation effects. Moreover, the authors design an automated framework for continuous data re-generation triggered by new corpus release. The data and evaluation harness are open-sourced.
Critical strengths
- The paper studies a timely problem, develops a useful benchmark, and has some interesting findings, e.g. DP synthesis methods fail to transfer knowledge from original corpus.
- The automated re-generation of data and the reusable, scalable framework design of the benchmark seems particularly nice from an architecture perspective.
- The analysis is rigorous and substantial.
- The writing is clear.
Critical weaknesses
- In abstract, "even at ε = 100", would be better to directly explain what does ε mean here (privacy constraint).
- The major finding is a bit counterintuitive to me. If non-private synthesis transfers knowledge well, why DP synthesis with extremely weak privacy constraint cannot transfer knowledge? Is there any key mechanism in the DP method leads to this outcome, or could there be other underlying factors influencing the results besides privacy? Is it caused by the noise in DP-SGD, as in Appendix E?
Questions for authors
Overall, the problem is timely, the benchmark seems useful, the method seems solid, and the findings are interesting. I appreciate the authors' hard work. The major finding is a bit counterintuitive to me. If non-private synthesis transfers knowledge well, why DP synthesis with extremely weak privacy constraint cannot transfer knowledge? Is there any key mechanism in the DP method leads to this outcome, or could there be other underlying factors influencing the results besides privacy? Is it caused by the noise in DP-SGD, as in Appendix E?
3 comments
An intuitive explanation of why DP synthesis fails to transfer facts
We have taken an initial step toward understanding this gap through empirical investigation. In our ablation experiment, we first found that direct DP fine-tuning on the original corpus performs comparably to the evaluators trained on DP synthetic corpus, so the bottleneck lies in the private training itself. Then, our clipping vs. noising ablation shows that clipping does not hurt much, while noising is the main cause of the degradation.
We thus attempt to explain this phenomenon from a SNR perspective. In GEMINON, a fact supported by ~200 out of 200K records appears in only ~0.5 examples per batch (of 512), and each supporting record contains only a few QA-relevant tokens. Thus, the gradient signal is weak and infrequent, so DP noise can obscure it even at large . However, clipping itself only bounds the magnitude of individual contributions without introducing random signals, so as a result, learning signals that are weak can still accumulate. On the other hand, the style and structure of our corpus contribute signals in every batch, so they can still be learned under DP, as reflected in the MAUVE scores and the qualitative examples. This explains why DP synthetic text can look distributionally similar to the real corpus while transferring few facts.
We emphasize that this is just an intuitive explanation rather than a theoretical characterization. We hope that this counterintuitive findings can motivate further research on DP training dynamics, the learnability of tasks under DP, and improving DP synthesis.
Please let us know if this answers your question or if any clarification you would like before the discussion period ends tomorrow. If your concerns have been addressed, we would greatly appreciate it if you would consider updating your score.
W1: Clarifying ϵ in the abstract
Thank you for the suggestion. We agree that the meaning of ϵ should be made explicit for readers less familiar with differential privacy, and we will revise the abstract accordingly to clarify.
W2/Q: Counterintuitive findings
We thank the reviewer for this question. Indeed, we find that the results are counterintuitive, especially given the substantial knowledge transfer by non-private synthesis. Nevertheless, we believe that uncovering this surprising phenomenon is indeed a contribution of our work. Existing evaluations suggested that DP synthesis methods perform well, yet our results reveal a substantial and previously overlooked gap, even at a weak privacy regime. We believe this finding challenges the current understanding of the DP synthesis methods and deserves attention from the broader research community.
In particular, ϵ=100 is an extremely weak privacy regime, e.g., randomized response with ϵ=100 only lies with probability in the order of 10−44. The fact that these well established DP synthesis methods still fail at this level suggests that the problem is not a fundamental limit of DP. Rather, it highlights how far these methods are from achieving comparable utility at meaningful privacy regimes, e.g., ϵ=1. We hope that understanding the underlying causes and developing methods to overcome this gap will motivate substantial follow-up research.
Our primary goal in this work is to rigorously identify this empirical phenomenon, rather than to provide a complete mechanistic explanation. Nevertheless, we have also taken initial steps toward understanding the failure, and we are happy to engage with your question via our own hypotheses regarding a mechanistic explanation for this phenomenon, please see the response below.
Overall, it seems to us that the main concern in your review is this counterintuitive finding, which, as we argue above, is indeed the paper's central contribution. We also appreciate your positive comments on the timeliness of the work, the usefulness of the benchmark, and the soundness of the methodology. If our responses address your concern, we would be grateful if you would consider raising your score.