- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a new method for private synthetic text generation using LLMs. The problem is timely and important. The proposed method builds on Aug-PE and proposes a lightweight evaluator that uses previously privatized voting results to guide additional evolution steps without further access to the private corpus. This enables more iterations without additional privacy cost and improves generation quality under the same privacy budget.
Critical strengths
- The paper is well-presented, easy to follow and intuitive.
- The proposed method meaningfully improves Aug-PE while being simple and computationally efficient.
- Experiments include diverse datasets and informative ablations.
Critical weaknesses
-
My biggest concern is the choice of baselines. The paper mostly compares against older methods such as Aug-PE and CTCL, but leave out many newer methods in both inference-only and fine-tuning branches. For example, see [1-6] for the methods that should have been compared against. Also see [7] for a survey. With the current status, I lean toward rejecting the paper. If the authors provide a comprehensive comparison with these baselines, I will increase my score. Otherwise, I am unsure that the survey of previous methods are informative enough for acceptance.
-
What are hyperparameters used to compute MAUVE? MAUVE is known to be highly sensitive to hyperparameters such as scaling factors or number of clusters. For example, [2] and [3] show that under MAUVE scaling factor 5, many of methods based on instruction-tuned models show very low MAUVE score, reflecting their low diversity. Also in Figure 6, the authors should clarify if the number of clusters is growing with the data size or is fixed.
-
What is the reason behind choosing a relatively old LLM, i.e., Llama 3.2 3B-Instruct, as the main model in experiments? How do the results change if a newer reasoning model is used instead?
-
What is the privacy-utility trade-off of the method? In particular, how does SelfEvo-PE perform in and cases?
[1] Private prediction for large-scale synthetic text generation
[2] Clustering and Median Aggregation Improve Differentially Private Inference
[3] EPSVec: Efficient and Private Synthetic Data Generation via Dataset Vectors
[4] ACTG-ARL: Differentially Private Conditional Text Generation with RL-Boosted Control
[5] MAPLE: Metadata Augmented Private Language Evolution
[6] DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuning
[7] How to DP-fy Your Data: A Practical Guide to Generating Synthetic Data With Differential Privacy
Questions for authors
Please see weaknesses.
6 comments
Response to Reviewer 9GsG (2/2)
Weakness 2: MAUVE hyperparameters
We use the default hyperparameters of the mauve-text library [1]. In particular, the scaling factor is set to a constant value at 5. This configuration is consistently used to evaluated all methods in our experiments.
[1] https://pypi.org/project/mauve-text/
LLM backbone
The reason we chose a relatively old LLM because the cutoff date of its training data is before the date of the datasets we used in our experiments. This ensures that the LLM has not seen any of the private data during its pretraining, which is important for the DP guarantee.
In Appendix C.7.3, we presented the result with a slightly larger LLM (Qwen-2.5-7B), where SelfEvo-PE outperforms Aug-PE across different privacy budgets.
Additionally, we conduct a new experiment using a closed-source model (GPT6-luna) on the BioRxiv dataset.
| Method | FID | MAUVE | Acc |
|---|---|---|---|
| Aug-PE | 67.37 | 0.248 | 0.625 |
| SelfEvo-PE | 60.17 | 0.279 | 0.670 |
SelfEvo-PE also outperforms Aug-PE in this case. This confirms that the gain of SelfEvo-PE is from the better noise reduction and selection performance, regardless of the LLM backbone. We added this new result to Appendix C.7.4.
Performance without DP
While the main point of SelfEvo-PE is to enhance PE through better selection and noise reduction, we do not expect SelfEvo-PE to improve over Aug-PE in the non-private setting. We added this new result to Appendix C.5.3.
| Dataset | Method | FID | MAUVE | Acc |
|---|---|---|---|---|
| BioRxiv | Aug-PE | 28.91 | 0.954 | 73.253 |
| BioRxiv | SelfEvo-PE | 27.71 | 0.931 | 73.919 |
| Congressional Speech | Aug-PE | 34.92 | 0.952 | 60.067 |
| Congressional Speech | SelfEvo-PE | 36.54 | 0.857 | 57.367 |
| OpenReview | Aug-PE | 42.21 | 0.461 | 43.126 |
| OpenReview | SelfEvo-PE | 42.87 | 0.354 | 41.637 |
Private Evolution is not able to perform with , as there is no signal to guide the evolution. Instead, we can consider the zero-shot baseline that was already included in the main paper.
Dear Reviewer,
Thank you very much for your time and helpful suggestions. We hope our response addresses your concerns. As we are approaching the rebuttal deadline, please let us know if you have any further questions or concerns.
Sincerely, Authors of SelfEvo-PE
Response to Reviewer 9GsG (1/2)
Thank you for taking the time to review our work and share your suggestions.
Weakness 1: Baselines
We thank the reviewer for the suggestion. Unfortunately, most of the mentioned papers [1,2,5,6] do not release their code. Our paper already cited and discussed [1], [2], [4], [5], [6], and [7]. We have updated the paper to include [3] as well. The survey [7] categorizes the methods into three main branches based on the setting differences and their strenghts: (1) Private Inference [1,2], (2) Private Evolution [5], and (3) Fine-tuning [4,6,7] (please see Table 10, Page 62).
First of all, our method is built on the private evolution branch, which leverages black-box LLM APIs only and does not require any access to the model weights or fine-tuning. Therefore, the fine-tuning methods [4,6,7] are not directly comparable to our method. While MAPLE [5] is a private evolution method, it enhances the random_api which is orthogonal to our method. Additionally, MAPLE requires an additional assumption of a small public (donated) in-distribution dataset, which is not required in our method. [3] also does not require fine tuning, but it requires the control of inference (forward pass) to manipulate the hidden states of the model, which is not available in most standard LLM APIs.
We implement the Private Inference methods [1,2] only, as they share the closest setting with our method. Private Inference was designed to generate a small number of synthetic samples (e.g., <25 samples) mainly for in-context learning. These methods add noise to the softmax output of each token, so the required noise scale is increased with both the number and the length of the generated samples. However, we found that to generate a few hundred samples, the noise is too large that makes each token probability distribution close to uniform. The below table shows the results of [1] and [2] on the BioRxiv dataset with ϵ=1.0. We added this new result to Appendix C.7.5.
[1] Private prediction for large-scale synthetic text generation
[2] Clustering and Median Aggregation Improve Differentially Private Inference
[3] EPSVec: Efficient and Private Synthetic Data Generation via Dataset Vectors
[4] ACTG-ARL: Differentially Private Conditional Text Generation with RL-Boosted Control
[5] MAPLE: Metadata Augmented Private Language Evolution
[6] DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuning
[7] How to DP-fy Your Data: A Practical Guide to Generating Synthetic Data With Differential Privacy