acceptodds
Withdrawn from ICLR 2027

When Does Memorization Become Extraction? Frequency-Conditioned Privacy Auditing of Fine-Tuned Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models are increasingly fine-tuned on sensitive data, where the same person's information may appear multiple times across records. This can cause models to memorize sensitive information, but memorization does not necessarily mean that the information can be recovered. A more serious privacy risk arises when the model begins to reproduce that information during generation. In this paper, we study how often sensitive information must appear in the fine-tuning data before it becomes recoverable. Across four language models, we vary the frequency of email addresses and phone numbers from to and compare standard fine-tuning, gradient clipping, and differentially private training. We find that the number of repetitions required before sensitive information becomes recoverable varies substantially across models. Gradient clipping can delay extraction, but it does not consistently prevent leakage at high repetition. DP-SGD at produces zero observed extraction through repetition across all four models under the attacks we test, although the utility cost varies by model. These findings show that practical privacy risk is exposure-dependent and that the transition from memorization to extraction varies across models and training mechanisms.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM privacy, memorization, data extraction, differential privacy, fine-tuning, membership inference, gradient clipping

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_rWPNOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how often sensitive information needs to appear in fine-tuning data before a language model starts reproducing it. The authors distinguish between memorization, which measures whether a model has learned something from its training data, and extraction, which happens when the model actually generates that information.

To investigate this, the authors create 800 canaries containing email addresses and phone numbers from the AI4Privacy dataset. They add these canaries to the fine-tuning data, repeating them at different frequencies (from 1 to 50 times), with different canaries assigned to each frequency. They also keep a separate set of unseen canaries for comparison.

They evaluate four small language models (LLaMA-3.2-1B, Gemma-2-2B, Gemma-3-1B, and VaultGemma-1B) using LoRA fine-tuning under three conditions: standard training, gradient clipping, and DP-SGD. They measure memorization using loss-based metrics, Exposure scores, and membership inference, and test extraction using greedy and stochastic decoding. They also evaluate how the different training methods affect model performance.

The main finding is that extraction increases with repetition, but the frequency at which this happens varies considerably across models. Gradient clipping can delay extraction in some cases, but it does not consistently prevent leakage at higher repetition frequencies. DP-SGD prevents extraction in all the tested settings, although it also leads to substantial drops in model performance. The authors also find that memorization and membership-inference scores are not always good predictors of which examples can actually be extracted.

Generally the authors argue that directly testing extraction at different repetition frequencies gives a better picture of privacy risks than relying only on memorization or membership-inference metrics.

Critical strengths
  1. The paper addresses an important question for privacy in language models. The distinction between memorization and actual extraction is relevant, especially when sensitive information appears multiple times in the training data.

  2. The experimental setup is well controlled. The authors test six repetition frequencies across four models and compare standard training, gradient clipping, and DP-SGD. Including a clipping-only condition is useful to better understand the effect of clipping separately from DP noise.

  3. The authors evaluate privacy using different metrics, including both memorization and direct extraction. They also use three training seeds and test both greedy and stochastic decoding. I also appreciate that they broadly acknowledge the limitations of their experiments.

Critical weaknesses

Weaknesses

  1. Most findings confirm what previous work has already shown: repetition increases extraction, DP reduces leakage at a utility cost, and memorization does not necessarily imply extraction. In particular, Szep et al. (2026) already study PII memorization during fine-tuning and the role of frequency. The paper adds a controlled comparison across repetition frequencies, models, and training methods, but I find the new insights rather limited. For example, the authors show that extraction thresholds differ across models without investigating why.

  2. The experiments only consider four small models (1-2B), two types of sensitive information (emails and phone numbers), and LoRA fine-tuning. Given how much the results vary even across these four models, it is unclear how well the findings generalize to larger models or other settings.

  3. The paper motivates the problem with sensitive information appearing across different records, but the experiments repeat identical canary examples. In practice, the same information may appear in different contexts. Also, different canaries are used at each frequency, so the experiments do not directly show how increasing repetition affects the same sensitive value.

  4. Despite the title, the paper mainly studies extraction as a function of repetition. Section 4.3 compares memorization scores with extraction, but the AUROC results are pooled across frequencies. Since frequency itself affects extraction, it is unclear whether these metrics predict which individual examples are vulnerable beyond simply reflecting their repetition frequency.

  5. DP-SGD results in zero observed extraction, but also substantially reduces model performance (Tab 2). It would be useful to compare against non-private models with similar utility to better understand how much of the reduction in extraction is specific to DP training rather than weaker task learning.

Minor comments:

  • The LoRA configuration should be described in the main experimental setup (currently stands only in the Appendix).
  • Section 4.3 compares membership-inference AUC from Table 6 with extraction-prediction AUROC from Table 3, although these results come from different training configurations. This comparison should be clarified.
  • The extraction attacks use the exact context preceding each sensitive value. It would be useful to discuss how this affects the applicability of the findings to more realistic attacks.
Questions for authors
  1. Have the authors tested extraction on the 1,000 unseen reference canaries? This would help establish whether the sensitive values are recoverable without being included in fine-tuning, especially given that the prompts contain the exact preceding context.

  2. How should the DP guarantee be interpreted when the same sensitive value appears up to 50 times? Is the guarantee at the example level or the individual level?

  3. What is the main new insight from the controlled frequency experiments that was not established by Szep et al. (2026)?

Reviewer_dpsFOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper explores how the repetition frequency of sensitive information during fine-tuning affects its transition from statistical memorization to practical extraction. The authors conduct experiments across four language models using email and phone-number canaries at different repetition frequencies. They compare non-private fine-tuning, gradient clipping, and DP-SGD, and further examine whether statistical privacy metrics can predict extraction risks. The results suggest that extraction behavior varies across models, while DP-SGD substantially reduces observed leakage under the evaluated settings, albeit with notable utility costs.

Critical strengths

The paper addresses a practically relevant privacy concern in language model fine-tuning. Evaluating statistical memorization alongside direct extraction is a meaningful perspective. The experiments also cover multiple models, repetition frequencies, and training mechanisms, providing useful empirical observations about privacy leakage.

Critical weaknesses
  1. The frequency groups contain different canaries rather than evaluating the same sensitive values under different repetition levels. This makes it difficult to fully disentangle the effect of repetition from differences in the inherent extractability of individual examples. The distributional checks provided do not sufficiently address this concern. Moreover, the experiments are limited to relatively small models and two types of sensitive information, while the extraction attacks cover only a limited range of strategies.

  2. The paper combines frequency-conditioned extraction, privacy mechanism comparisons, membership inference, and privacy-budget evaluations, but these components are not sufficiently integrated into a coherent experimental narrative. Although the authors distinguish the matched experiments from the broader audits, the latter use different model-specific training configurations, making it harder to interpret the results collectively. A clearer organization linking each research question to its experimental evidence and conclusions would substantially strengthen the paper.

  3. Table 6 extends beyond the page boundary, with some values overlapping the page number and several rows of experimental results cut off in the submitted PDF. This directly affects the readability and verifiability of the reported results. The manuscript would benefit from a thorough formatting and consistency check.

Questions for authors
  1. Could the authors evaluate the same canaries at multiple repetition frequencies, or provide an equivalent within-canary control to show that the reported frequency effect is not driven by differences in the inherent extractability of the canaries?

  2. Which comparisons across the frequency experiments, privacy mechanisms, membership-inference audits, and privacy-budget evaluations are intended to support common conclusions, given their different model-specific training configurations? Could the authors clarify this with matched comparisons or an explicit account of the relevant controls?

  3. Could the authors provide a complete, legible version of Table 6 and confirm the results currently cut off by the page boundary, so that the reported findings can be independently assessed?

Reviewer_ytSeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how the frequency of sensitive information in fine-tuning data affects its vulnerability to extraction from language models. The authors evaluate different training methods, including gradient clipping and DP-SGD, and examine the relationship between membership inference signals and successful extraction. Experiments across multiple models show that extraction risk generally increases with frequency. The results also suggest that membership inference scores are not reliable indicators of extraction risk.

Critical strengths

S1. How repeated sensitive data in fine-tuning sets translates into leakage is a practically relevant question.

S2. The study controls the canary frequency and compares matched training conditions across several models.

Critical weaknesses

W1. Limited novelty and new insights. The main findings are largely established in prior work: repetition increases memorization and extraction, PII frequency affects leakage during fine-tuning, and membership inference does not fully capture extraction risk. The experimental design also closely follows the Secret Sharer methodology (Carlini et al., 2019), which studies canary repetition and differential privacy. Moreover, closely related work on PII extraction from fine-tuned models with and without differential privacy [1] is not discussed. While the paper brings these elements together in a controlled evaluation, it offers limited new insights beyond existing findings.

W2. Each canary is inserted as a complete training example repeated verbatim, and extraction uses the same instruction and preceding context as training. This setup primarily measures discoverable memorization rather than extraction under realistic attacker capability, weakening the paper's distinction between memorization and extraction. In particular, the motivating scenario of sensitive information appearing across different records is not evaluated. A more realistic setting would vary the contexts containing the same sensitive value and consider attackers without access to the exact training context. The current results therefore do not fully support claims about practical extraction risk.

W3. Membership inference results are aggregated across frequency levels, preventing a direct comparison of when membership becomes detectable and when extraction succeeds. The paper should report membership inference performance, such as TPR at low FPR, and extraction rates at each frequency level under matched settings. Moreover, the claim that WBC achieves strong membership inference but poorly predicts extraction relies on results from different training configurations; under the matched condition, its membership inference performance is already weak. The prediction analysis also pools frequency levels, making it difficult to separate predictive ability from the effect of repetition.

W4. DP-SGD uses the same batch size and learning rate as non-private training. However, it is known that DP-SGD typically requires different hyperparameter tuning to achieve competitive utility. Consequently, the reported absence of extraction and privacy–utility trade-off may reflect poor model training rather than the effectiveness of DP-SGD. A properly tuned DP baseline or a utility-matched non-private comparison is needed. Furthermore, the reported ϵ=5\epsilon = 5 guarantee applies at the record level, whereas repeated canaries require a much weaker group-privacy bound. The absence of extraction for frequently repeated canaries therefore cannot be directly attributed to the stated privacy guarantee.

[1] Analyzing Leakage of Personally Identifiable Information in Language Models. IEEE S&P 2023.

Questions for authors

Please see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.