- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DP-RFT, a differentially private reinforcement fine-tuning framework for generating synthetic text without directly exposing individual private examples to the language model. The method uses DP-protected similarity rewards derived from a private corpus to guide on-policy generation through Proximal Policy Optimization (PPO). Experiments on four datasets, including PubMed, BBC News, WildChat, and QMSum, show improvements in synthetic data fidelity and downstream language modeling performance over Aug-PE.
Critical strengths
-
The paper addresses an important challenge in privacy-preserving synthetic text generation: adapting language models to private data distributions without directly exposing individual private examples. The proposed eyes-off setting has practical value for privacy-sensitive applications.
-
DP-RFT combines differentially private similarity rewards with on-policy reinforcement fine-tuning, enabling the generator to learn from private data feedback without directly accessing the original samples. This provides an alternative to conventional DP fine-tuning and iterative prompting-based approaches.
Critical weaknesses
-
POPri (Hou et al., ICML 2025) has already explored optimizing synthetic text generation using differentially private feedback. The authors explicitly discuss this closely related work in the Related Work section and even cite it as part of the basis for their downstream evaluation setup. However, POPri is not included in the experimental comparisons. Since the main methodological distinction lies in adopting on-policy PPO instead of DPO, the absence of a direct comparison makes it difficult to establish the advantages of the proposed approach and substantially weakens its novelty claims.
-
The paper does not clearly define the similarity function, its value range, or how the additional Jaccard and KL-based terms are incorporated into the reward. More importantly, Figure 2 clips similarity scores using =\min(), whereas Appendix F assumes to derive the sensitivity bound $\Delta_2$$\leq$ ). This clipping operation only limits the upper bound and does not guarantee the assumed absolute-value bound if similarity scores can be negative. The authors should clarify the score range and clipping mechanism, as these assumptions are essential to the validity of the claimed differential privacy guarantee.
Questions for authors
-
Could the authors clarify the key methodological differences and include a comparison with POPri under matched settings? If such a comparison is infeasible, please explain why.
-
DP-RFT introduces both raw similarity rewards and PPO-based optimization. Can the authors provide controlled ablations to distinguish their individual contributions? For example, how does PPO with the original binary voting rewards compare with PPO using raw similarity rewards, and can raw similarity rewards improve Aug-PE without PPO?
Then see the weaknesses. A clear and convincing response to these concerns could lead me to reconsider my final recommendation.