acceptodds
Withdrawn from ICLR 2027

DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Differentially private (DP) synthetic data generation plays a pivotal role in developing large language models (LLMs) on private data, where data owners cannot provide eyes-on access to individual examples. Generating DP synthetic data typically involves a difficult trade-off. On one hand, DP finetuning methods train an LLM as a synthetic data generator with formal privacy guarantees, yet it still requires the raw content of private examples for model training. However, methods that avoid direct exposure to private data are bounded by an off-the-shelf, un-finetuned model, whose outputs often lack domain fidelity. Can we train an LLM to generate high-quality synthetic text without eyes-on access to individual private examples?In this work, we introduce Differentially Private Reinforcement Fine-Tuning (DP-RFT), an online reinforcement learning algorithm for synthetic data generation with LLMs. DP-RFT leverages DP-protected nearest-neighbor votes from an eyes-off private corpus as a reward signal for on-policy synthetic samples generated by an LLM. The LLM iteratively learns to generate synthetic data to maximize the expected DP votes through Proximal Policy Optimization (PPO). We evaluate DP-RFT for long-form and domain-specific synthetic data generation, such as news articles, meeting transcripts, and medical article abstracts. Our experiments show that DP-RFT closes the gap between private evolution and DP finetuning methods in terms of the fidelity and downstream utility of the generated synthetic data, while respecting the private data boundary.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Differential Privacy, Synthetic Data Generation, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gTyfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DP-RFT, a differentially private reinforcement fine-tuning framework for generating synthetic text without directly exposing individual private examples to the language model. The method uses DP-protected similarity rewards derived from a private corpus to guide on-policy generation through Proximal Policy Optimization (PPO). Experiments on four datasets, including PubMed, BBC News, WildChat, and QMSum, show improvements in synthetic data fidelity and downstream language modeling performance over Aug-PE.

Critical strengths
  1. The paper addresses an important challenge in privacy-preserving synthetic text generation: adapting language models to private data distributions without directly exposing individual private examples. The proposed eyes-off setting has practical value for privacy-sensitive applications.

  2. DP-RFT combines differentially private similarity rewards with on-policy reinforcement fine-tuning, enabling the generator to learn from private data feedback without directly accessing the original samples. This provides an alternative to conventional DP fine-tuning and iterative prompting-based approaches.

Critical weaknesses
  1. POPri (Hou et al., ICML 2025) has already explored optimizing synthetic text generation using differentially private feedback. The authors explicitly discuss this closely related work in the Related Work section and even cite it as part of the basis for their downstream evaluation setup. However, POPri is not included in the experimental comparisons. Since the main methodological distinction lies in adopting on-policy PPO instead of DPO, the absence of a direct comparison makes it difficult to establish the advantages of the proposed approach and substantially weakens its novelty claims.

  2. The paper does not clearly define the similarity function, its value range, or how the additional Jaccard and KL-based terms are incorporated into the reward. More importantly, Figure 2 clips similarity scores using Si,jS_{i,j}=\min(Si,j,cS_{i,j},c), whereas Appendix F assumes ∣Si,j∣|S_{i,j}| ≤\leq cc to derive the sensitivity bound $\Delta_2$$\leq$ csc\sqrt{s}). This clipping operation only limits the upper bound and does not guarantee the assumed absolute-value bound if similarity scores can be negative. The authors should clarify the score range and clipping mechanism, as these assumptions are essential to the validity of the claimed differential privacy guarantee.

Questions for authors
  1. Could the authors clarify the key methodological differences and include a comparison with POPri under matched settings? If such a comparison is infeasible, please explain why.

  2. DP-RFT introduces both raw similarity rewards and PPO-based optimization. Can the authors provide controlled ablations to distinguish their individual contributions? For example, how does PPO with the original binary voting rewards compare with PPO using raw similarity rewards, and can raw similarity rewards improve Aug-PE without PPO?

Then see the weaknesses. A clear and convincing response to these concerns could lead me to reconsider my final recommendation.

Reviewer_71SEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DP-RFT, a differentially private reinforcement fine-tuning framework for synthetic text generation. DP-RFT uses DP similarity scores between generated and private documents as rewards and optimizes the generator through PPO. Experiments show that DP-RFT generally improves synthetic data fidelity and downstream utility over Aug-PE.

Critical strengths
  • This paper is clearly written and easy to follow.

  • The proposed approach of leveraging RL fine-tuning is intuitive and provides an alternative to existing private evolution methods.

Critical weaknesses
  • The technical novelty over prior work is unclear. Hou et al. (2025) proposed using DP similarity feedback to fine-tune language models for synthetic data generation through DPO. The main distinction of DP-RFT appears to be the use of PPO instead of DPO. Could the authors further clarify the technical novelty of this work?

  • I recommend including Hou et al. (2025) as a baseline in the experiments, since both methods follow a similar framework of using DP similarity feedback to train synthetic data generators.

  • I also suggest considering [1] as an additional baseline, as it also provides a training-free approach to DP synthetic text generation.

[1] Sun, Zhonghao, et al. "Dpga-textsyn: Differentially private genetic algorithm for synthetic text generation." Findings of the Association for Computational Linguistics: ACL 2025. 2025.

  • The similarity-based reward may not adequately capture the full private data distribution. I recommend further evaluating the diversity and coverage of the generated data.

  • The downstream evaluation is relatively limited. The experiments mainly evaluate next-token prediction accuracy using GPT-2 and BERTSmall. I recommend evaluating additional downstream tasks and more modern language models to show the practical utility of the generated synthetic data.

Questions for authors

See Weaknesses.

Reviewer_vVxNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DP-RFT to train a generative model on private text. The problem in this setting is a "eyes-off" setting, where the generative model should not be provided with the private text, while also making sure model weights are updated. The authors propose to use PPO to train a generative model. The reward signals are computed by embedding similarity between generated document and private text. The reward signals are then DP protected by injecting noise. One main contribution is that the authors provide a mechanism to prevent reward hacking by using another GPT-4o model.

Critical strengths
  1. The paper addresses an interesting problem setting: eyes off model training, where the LLM that is being trained never looks at the private text.
  2. From the experimentation results, an interesting and surprising finding is that for small models like BertSmall, DP-RFT beats DP-FT (where LLM has access to private text), and shows that in low-data settings, the proposed method has really good value.
Critical weaknesses
  1. The paper lacks any formal privacy analysis and threat model analysis. For instance, the method passes synthetic documents to GPT-4o (OpenAI API) every step of training. After some training steps, the synthetic documents could well resemble the private text. Are we assuming a post-processing DP property here?
  2. The experiments while surfacing interesting findings, overall does bring out the gaps in the method. On FID, Aug-PE is better most of the times. The authors do mention that Aug-PE directly optimize for FID, but there is a big gap.
  3. The paper does not provide any diversity analysis in generated text, since the rewards are based on similarity, there is a high chance that there is some form of mode collapse too, and existing reward hacking mitigation does not account for that.
Questions for authors
  1. There are some decisions that are not ablated fully: threshold of 6, reward hacking is shown only on one dataset and one privacy budget.
  2. Are there any hidden computation costs here for embedding computation every training step?
  3. Every experimentation number is a single point estimate, could you provide statistical variance estimates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.