acceptodds
Under review as a conference paper at ICLR 2027

Measuring and Mitigating Private Data Leakage in Automated Prompt Optimization

Abstract

Automated prompt optimization (APO) has emerged as a lightweight method for adapting a large language model (LLM) to perform well on a particular task. APO edits and refines input prompt text, a sample-efficient and flexible alternative to parameter-based fine-tuning. For tasks that involve interacting with users' private data, further prompt quality improvements are possible by performing APO directly on real users' examples rather than public or synthetic data. However, this exposes the risk of private information leaking into the optimized prompt. In this paper, we demonstrate that private data leakage can indeed occur when APO is run using training data containing private information. We introduce empirical privacy auditing methods to quantify this leakage based on carefully calibrated matching of token windows between training data examples and the optimized prompt. We then present some simple modifications to APO, specifically the meta-prompts used by the mutator and autorater models, that significantly reduce private data leakage without hampering the quality of the optimized prompt. Comparison of different leakage mitigation strategies in terms of their privacy-utility trade-offs shows that private data leakage through APO can be mitigated with minimal impact on the performance of the optimized prompt.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
privacy, prompt optimization, language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_piXJOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses an important and relatively underexplored problem, whether automated prompt optimization (APO) algorithms can inadvertently memorize and expose private training examples in their optimized prompts. It shows the empirical demonstration of private training-data leakage and develops a practical auditing framework and mitigation strategies. However, the paper lacks rigorous differential privacy protection and has low novelty of the mitigation techniques.

Critical strengths

S1. Generally speaking, the paper did a reasonable well evaluation, experiments on two APO algorithms, ProTeGi and GEPA, using Enron emails and LongLaMP-3 product reviews. The results suggest the proposed techniques are practical alternatives or complements to computationally expensive formal differential privacy approaches.

Critical weaknesses

W1. The caption of Figure 4 states "The PM+PA strategy, which uses the privacy-preserving mutator meta-prompt plus the privacy-preserving autorater feedback yields the best empirical epsilon of 0 with minimal impact on the test score". However, as shown in Equation 6, a lower bound of zero does not imply that the true privacy loss is zero, or even small.

W2. The three mitigation strategies (remove PII from training examples, instruct the mutator not to copy private information, instruct the autorater not to include private information in its feedback) are conceptually simple and the paper does not propose a new privacy-aware optimization objective or a principled mechanism for controlling leakage.

W3. The paper lacks experimental comparisons with established privacy-preserving APO methods, particularly DP-OPT.

Questions for authors

W1 and W3 in the above weakness section.

Reviewer_RoDEOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the leakage of private training content into prompts generated by automated prompt optimization (APO). To quantify this leakage, it introduces a prompt-inspection audit based on exact and approximate text matching against training and held-out examples, reporting both leakage density and an empirical lower bound on privacy loss, (\epsilon_{\mathrm{lo}}). To mitigate this risk, the authors evaluate privacy-aware instructions in the mutator and autorater meta-prompts, as well as PII redaction of the training data. Experiments with ProTeGi and GEPA on Enron and LongLaMP-3, conducted over six random seeds, show that combining the two meta-prompt interventions (PM+PA) reduces measured textual leakage with modest utility losses relative to the undefended baseline. The results support a practical mitigation of observed leakage.

Critical strengths
  1. The paper highlights a relatively novel and practically important privacy risk in APO. Its focus on private training content leaking into the final optimized prompt is compelling. Figure 1 provides concrete evidence of this failure mode, showing substantial passages from training examples copied verbatim into optimized prompts. These examples establish that the risk occurs in practice and can involve considerably more than isolated identifiers, motivating closer attention to the privacy of the prompts produced by APO.

  2. The defenses are lightweight, easy to integrate, and evaluated through a useful set of ablations. The experiments examine PM and PA individually and in combination, with and without PII redaction, alongside a malicious control and six random seeds. This design helps assess the contribution of each intervention and reveals cases where redaction is ineffective or counterproductive. PM+PA is particularly practical: it can be incorporated into existing APO pipelines through changes to the meta-prompts, with little additional computational overhead. Across both optimizers and datasets, it reduces measured textual leakage while incurring relatively small utility losses.

Critical weaknesses
  1. The audit provides evidence of textual copying, but its coverage of privacy leakage is not sufficiently validated. Exact and approximate 10-token matching can identify copied passages, yet may miss paraphrases that preserve private facts or disclosures involving short identifiers. Complementary checks using more advanced text similarity metrics such as METEOR/BERTScore etc. may help identify such cases. This would clarify whether PM+PA reduces the disclosure of private information or mainly reduces lexical overlap. The audit also needs clearer documentation and statistical justification. In the redaction experiments, it is unclear whether matching uses original or redacted examples, and whether the held-out cohort receives the same preprocessing. These choices affect both matching and exclusivity filtering. Moreover, the pooled Clopper–Pearson intervals require justification because examples within each run share one optimized prompt. The resulting epsilon lower bound should not be interpreted as a privacy guarantee: a value of zero can coexist with residual training matches, as Table 1 shows.

  2. The defense is technically quite basic, and its advantage over simpler alternatives remains unclear. PM and PA primarily instruct the mutator and autorater not to reproduce private information, while redaction removes PII before optimization. The underlying APO algorithms are unchanged. Such lightweight interventions can be useful, but the methodological contribution is limited and therefore depends heavily on convincing experimental comparisons. The ICL ablation in Appendix A.1.3 is particularly relevant. Removing the clause that permits demonstrations substantially reduces GEPA's measured leakage, but the corresponding task utility is not reported. Without this comparison, it is difficult to establish why PM+PA is preferable to removing that clause or permitting only abstract or synthetic examples.

  3. The adverse redaction results are left largely unexplained. Adding redaction increases both training hits and matching density in ProTeGi–LongLaMP-3, whereas in ProTeGi–Enron the epsilon lower bound increases despite a slight decrease in density. Clarifying the audit representation and examining the affected passages would help distinguish changes in copying behavior from changes in background matches or filtering.

Questions for authors
  1. Coverage of the audit. The 10-token matching protocol may miss paraphrases that preserve private facts while changing their wording. Could you validate its coverage using a small manually annotated set containing synonymous expressions, more substantial paraphrases, short identifiers, and benign similarities? Complementary lexical or semantic similarity measures could help identify candidate disclosures, followed by checks of the underlying entities and facts. Do the reported gains from PM+PA persist under these additional checks?

  2. Incomplete ICL/non-ICL comparisons. The Baseline−ICL ablation substantially reduces measured leakage, but its task utility is not reported, and the corresponding PM+PA condition is missing. Could you provide a matched comparison crossing the presence or absence of the demonstration-generation clause with the presence or absence of PM+PA, reporting both utility and leakage? Please also verify whether removing the clause actually eliminates demonstrations from the resulting prompts. These results would clarify whether PM+PA provides additional value without ICL and whether retaining ICL with PM+PA offers a better trade-off than simply removing the clause.

  3. Explanation of the redaction results. Why does adding PII redaction to PM+PA worsen some reported privacy results? Are prompts audited against original or redacted examples, and is the held-out cohort processed identically? An examination of the matched passages and cohort hit rates would help distinguish increased copying from changes in preprocessing, background matches, or exclusivity filtering.

  4. Interpretation of a zero epsilon lower bound. A zero lower bound does not establish an absence of leakage, and Table 1 reports residual training matches in conditions where the bound is zero. Could you revise the “perfect privacy” wording and interpret the results jointly using the reported hit counts, matching densities, and epsilon lower bounds? Please clarify the scope of the resulting privacy claims, particularly when these measures give different impressions of a defense's effectiveness.

Reviewer_GNodOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper measures private data leakage in APO and considers countermeasures to mitigate it. An exclusive prompt-inspection audit pipeline is developed, which uses a held-out set of the same size to estimate how many matches would occur naturally in the text itself, and then excludes duplicate templated content. This paper considers two APO algorithms, ProTeGi and GEPA, and uses two datasets to evaluate the mitigation strategies.

Critical strengths
  • The problem of measuring private data leakage in APO appears to be unexplored.
  • The idea of using an exclusivity filter has some merit.
Critical weaknesses
  • Since APO methods are in heavy use today by LLaaS providers, evaluating only two APO methods in the paper seems limited. More APO frameworks are expected to be evaluated to demonstrate the applicability of the inspection method and the validity of the measurement conclusions.

  • The threat model is not clearly formulated. The methods to mitigate private data leakage are underexplored, and only some simple strategies are considered. It is unclear how previous methods for privacy-preserving training would perform (they may not be specific to APO, but could be adapted for it).

  • The writing needs to be significantly improved.

Questions for authors

Please see the Weaknesses.

1 comment

Thank you for the clarifications and revisions. The explicit threat model helps address my concern about the problem formulation, and I appreciate the efforts to improve the presentation. However, my concerns about the empirical scope and mitigation evaluation remain.

First, the optimizer-independent applicability of the audit does not establish the generality of the empirical findings. Second, the computational cost and feasibility analysis of DP-OPT is useful, but it does not establish the relative privacy-utility performance of the proposed interventions.

Overall, I appreciate the clarifications, but the response does not sufficiently resolve these substantive concerns. I therefore maintain my current overall assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.