- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how often sensitive information needs to appear in fine-tuning data before a language model starts reproducing it. The authors distinguish between memorization, which measures whether a model has learned something from its training data, and extraction, which happens when the model actually generates that information.
To investigate this, the authors create 800 canaries containing email addresses and phone numbers from the AI4Privacy dataset. They add these canaries to the fine-tuning data, repeating them at different frequencies (from 1 to 50 times), with different canaries assigned to each frequency. They also keep a separate set of unseen canaries for comparison.
They evaluate four small language models (LLaMA-3.2-1B, Gemma-2-2B, Gemma-3-1B, and VaultGemma-1B) using LoRA fine-tuning under three conditions: standard training, gradient clipping, and DP-SGD. They measure memorization using loss-based metrics, Exposure scores, and membership inference, and test extraction using greedy and stochastic decoding. They also evaluate how the different training methods affect model performance.
The main finding is that extraction increases with repetition, but the frequency at which this happens varies considerably across models. Gradient clipping can delay extraction in some cases, but it does not consistently prevent leakage at higher repetition frequencies. DP-SGD prevents extraction in all the tested settings, although it also leads to substantial drops in model performance. The authors also find that memorization and membership-inference scores are not always good predictors of which examples can actually be extracted.
Generally the authors argue that directly testing extraction at different repetition frequencies gives a better picture of privacy risks than relying only on memorization or membership-inference metrics.
Critical strengths
-
The paper addresses an important question for privacy in language models. The distinction between memorization and actual extraction is relevant, especially when sensitive information appears multiple times in the training data.
-
The experimental setup is well controlled. The authors test six repetition frequencies across four models and compare standard training, gradient clipping, and DP-SGD. Including a clipping-only condition is useful to better understand the effect of clipping separately from DP noise.
-
The authors evaluate privacy using different metrics, including both memorization and direct extraction. They also use three training seeds and test both greedy and stochastic decoding. I also appreciate that they broadly acknowledge the limitations of their experiments.
Critical weaknesses
Weaknesses
-
Most findings confirm what previous work has already shown: repetition increases extraction, DP reduces leakage at a utility cost, and memorization does not necessarily imply extraction. In particular, Szep et al. (2026) already study PII memorization during fine-tuning and the role of frequency. The paper adds a controlled comparison across repetition frequencies, models, and training methods, but I find the new insights rather limited. For example, the authors show that extraction thresholds differ across models without investigating why.
-
The experiments only consider four small models (1-2B), two types of sensitive information (emails and phone numbers), and LoRA fine-tuning. Given how much the results vary even across these four models, it is unclear how well the findings generalize to larger models or other settings.
-
The paper motivates the problem with sensitive information appearing across different records, but the experiments repeat identical canary examples. In practice, the same information may appear in different contexts. Also, different canaries are used at each frequency, so the experiments do not directly show how increasing repetition affects the same sensitive value.
-
Despite the title, the paper mainly studies extraction as a function of repetition. Section 4.3 compares memorization scores with extraction, but the AUROC results are pooled across frequencies. Since frequency itself affects extraction, it is unclear whether these metrics predict which individual examples are vulnerable beyond simply reflecting their repetition frequency.
-
DP-SGD results in zero observed extraction, but also substantially reduces model performance (Tab 2). It would be useful to compare against non-private models with similar utility to better understand how much of the reduction in extraction is specific to DP training rather than weaker task learning.
Minor comments:
- The LoRA configuration should be described in the main experimental setup (currently stands only in the Appendix).
- Section 4.3 compares membership-inference AUC from Table 6 with extraction-prediction AUROC from Table 3, although these results come from different training configurations. This comparison should be clarified.
- The extraction attacks use the exact context preceding each sensitive value. It would be useful to discuss how this affects the applicability of the findings to more realistic attacks.
Questions for authors
-
Have the authors tested extraction on the 1,000 unseen reference canaries? This would help establish whether the sensitive values are recoverable without being included in fine-tuning, especially given that the prompts contain the exact preceding context.
-
How should the DP guarantee be interpreted when the same sensitive value appears up to 50 times? Is the guarantee at the example level or the individual level?
-
What is the main new insight from the controlled frequency experiments that was not established by Szep et al. (2026)?