acceptodds
Under review as a conference paper at ICLR 2027

TRAP: Understanding and Mitigating Privacy Memorization in Language Models

Abstract

Fine-tuning a language model on sensitive records can leave it able to reproduce them. We ask when this memorization arises and how to prevent it without knowing in advance which spans are sensitive. Our starting point is that most memorization scores and attacks share one statistical core: whether the model assigns a token more probability than some reference would. Taking as the reference a model trained on the complementary half of the same corpus gives the Target Reference Advantage (TRA), a per-token signal that separates what a model fit to a particular record from what it learned across records, and is cheap and differentiable. We then study what drives memorization during fine-tuning: it keeps growing well past the validation minimum, is larger on small datasets and at higher learning rates, and higher when the underlying task is harder. Early stopping removes much of it, but because it is chosen by aggregate validation loss it helps least for rare, hard-to-predict spans embedded in otherwise learnable text, which is exactly what sensitive information tends to be. We therefore introduce TRAP, a one-sided penalty on tokenwise TRA that acts only where the target model pulls ahead of its reference. On student essays with annotated personal information and clinical cases with patient identifiers, TRAP brings memorization near the level of an untrained model at little utility cost, where generic regularizers barely move and differential privacy gives up most of what fine-tuning bought.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Training data memorization, privacy leakage, extraction attacks, sensitive information extraction, membership inference, language models, mitigation methods, LoRA fine-tuning, differential privacy, privacy-utility tradeoff

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_F7QHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how language models memorize sensitive information during fine-tuning and proposes TRAP, a regularization method to reduce memorization without requiring annotations of sensitive tokens. The authors introduce Target Reference Advantage (TRA), which measures the difference in token likelihood between two models trained on complementary halves of the data. They then use this signal to penalize tokens that appear to be memorized during training. The paper also investigates how factors such as training duration, learning rate, dataset size, task difficulty, and model size affect memorization. Experiments on three datasets show that TRAP achieves a better trade-off between memorization and model utility than differential privacy, LoRA, and several other regularization methods.

Critical strengths
  1. The paper addresses an important problem in fine-tuning language models on sensitive data. One of the main strengths of TRAP is that it does not require annotations of sensitive tokens, which can be difficult to obtain reliably in practice.

  2. Unlike general regularization methods that affect all training data, TRAP specifically targets tokens that the model is more likely to have memorized, based on the difference between the target and reference models. This is an interesting approach to reducing memorization while preserving model utility.

  3. Section 3 provides a thorough analysis of the factors that influence memorization. I particularly liked the finding that early stopping does not necessarily protect rare, hard-to-predict sensitive information. The experimental evaluation is also quite extensive, comparing TRAP against several baselines, including differential privacy and LoRA. I appreciate that the authors compare privacy–utility trade-off curves rather than just individual configurations.

  4. The paper provides a useful unified perspective on existing memorization measures by framing them as likelihood comparisons against different references. TRA is also a practical contribution, as it approximates counterfactual memorization at a much lower computational cost and can be directly used during training.

Critical weaknesses
  1. TRAP relies on differences between models trained on complementary subsets of the data. However, sensitive information appearing in both subsets may be learned by both models, resulting in a small TRA and potentially escaping the penalty. The synthetic datasets enforce unique sensitive values, leaving the effectiveness of TRAP under repeated sensitive information unclear. Additionally, the released model is trained on only half of the available data, and the proposed solution for using both halves is not experimentally validated.

  2. The evaluation primarily relies on likelihood-based memorization scores and membership-inference-style tests rather than direct extraction attacks. Although these metrics provide evidence of memorization, they do not establish whether sensitive information can be recovered through realistic prompting. The absence of adaptive attacks and direct extraction measurements limits the strength of the privacy conclusions.

  3. Several key experiments use relatively small evaluation sets, including only 20 member spans for MedCase and the rare-minority setting and 48 held-out sensitive spans for the essays. Many experiments also rely on single-seed runs. Given the dependence of TRA on the random data partition, additional seeds, larger evaluation sets, and uncertainty estimates would help establish the reliability of the reported improvements.

  4. Most experiments involve models with 1B parameters or fewer, with the 8B evaluation restricted to one dataset. It remains unclear whether the observed privacy–utility trade-offs generalize to larger models and more realistic fine-tuning scenarios.

Questions for authors
  1. How would TRAP behave when the same sensitive information appears in multiple documents, potentially across both training subsets? Since both models could memorize the same information, TRA might remain small even when sensitive information is memorized. Could the authors clarify how TRAP handles this case or provide experiments with different levels of duplication?

  2. The authors mention the possibility of training two models, each defended against the other, to avoid leaving half the data unused. Could the authors provide preliminary results for this approach or discuss how they expect it to affect the privacy–utility trade-off and computational cost?

  3. The evaluation focuses mainly on likelihood-based memorization scores. Could the authors provide results from direct secret-extraction attacks, even on a subset of the experiments? This would help establish whether the improvements in memorization scores also translate into reduced information leakage.

  4. Several experiments use relatively small evaluation sets and limited random seeds. How sensitive are the results to the random data partition and training initialization? Additional seeds or uncertainty estimates for the main results would help strengthen confidence in the findings.

  5. Most experiments use models with 1B parameters or fewer, with the 8B evaluation limited to MedCase. Could the authors discuss whether they expect TRAP to behave similarly on larger models and what challenges might arise when scaling the approach?

Reviewer_JHWSOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a method (TRA) to measure memorization by comparing the target model's token probabilities against a reference model trained on the complementary half of the dataset. The authors also do controlled experiments to show when and why are certain spans memorized during fine-tuning. The paper further proposes TRAP, a mitigation method applied during training that penalizes a token only if its TRA exceeds a specific threshold.

Critical strengths
  • Practical relevance: understanding the dynamics of memorization in a fine-tuning setting are practically very relevant and the authors make an effort in that direction. The overall idea of capturing memorization during finetuning, understanding when it happens, and proposing a mitigation could be a useful contribution to the memorization research community and practitioners.
Critical weaknesses
  • The writing of the paper is very hard to parse: for example, the first few paragraphs and even the abstract are not easy to read. The language is unclear, especially in the abstract. I believe this paper can be improved significantly in guiding the reader.

  • Methodology lacks rigor and prior literature on memorization:

    • The authors have missed citing prior works that use a similar notion of measuring memorization by looking at model's probabilities [1] and comparing it with the baseline [2]. The proposed method isn't entirely novel. Take a look at Figure 1a in [2], where the authors include a reference-based check to evaluate memorization.
    • In the when and why sensitive spans are memorized during fine-tuning part, the authors establish the connection to extended training, the complexity of the task and examples, etc. However, there is already some prior work which shows how memorization during fine-tuning changes as we over-train [3] as well as the connection to task/example complexity [4], which this work has completely missed citing or discussing.
    • The authors propose a mitigation method but do not compare it with some of the existing mitigation methods such as MemFreeDecoding, that use a similar notion of mitigation [5].

[1] "Measuring memorization in language models via probabilistic extraction".
[2] "QUANTIFYING MEMORIZATION ACROSS NEURAL LANGUAGE MODELS".
[3] "An Empirical Analysis of Memorization in Fine-tuned Autoregressive Language Models".
[4] "Does Learning Require Memorization? A Short Tale about a Long Tail".
[5] "Preventing Generation of Verbatim Memorization in Language Models Gives a False Sense of Privacy"

Questions for authors
  • Why exactly is TRA useful as a memorization measurement strategy if we already have methods that work in practice, including extraction attacks (some of which use reference models) and probabilistic memorization? Can the authors comment on this theoretically and empirically?
  • How does the proposed mitigation method compare against the existing baselines to mitigate memorization? This could be in terms of performance, overall utility degradation, etc.
  • The claims on memorization and training dynamics overlap with what we know already (such as over-training, task complexity, etc). How do the results in this paper challenge or expand what we know already?
Reviewer_LxJPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1(−1)
Read review
Paper summary

This paper studies the memorization of sensitive information during language-model fine-tuning. The authors introduce Target Reference Advantage (TRA), which measures the tokenwise log-likelihood difference between a target model and a reference model trained on a disjoint data subset. They use this measure alongside existing membership-inference scores to investigate how training duration, task difficulty, subgroup rarity, learning rate, and dataset and model size affect sensitive-span memorization. Based on these observations, they propose TRAP, a tokenwise regularizer that penalizes likelihood advantages exceeding a threshold, without requiring sensitive-span annotations during training. Experiments on student essays, synthetic applicant records, and clinical cases with synthetic identifiers compare TRAP with LoRA, differentially private training, and other regularizers. The authors report improved trade-offs between held-out predictive loss and sensitive-span membership-inference scores.

Critical strengths
  1. An important practical problem. The paper addresses how to learn from sensitive fine-tuning data while limiting disclosure of private information. The goal of preserving useful task performance while reducing sensitive-information leakage is consequential for deploying language models in settings where training records contain personal information.
Critical weaknesses
  1. The methodological novelty and additional insight are insufficiently established. TRAP closely relates to AdaFlood (Bae et al., TMLR 2024), which derives example-specific loss thresholds from auxiliary models trained with the corresponding examples held out. Under particular parameter choices, TRAP’s tokenwise objective matches AdaFlood with an uncorrected reference threshold. Although the complete training procedures and applications differ, the paper does not adequately distinguish its contribution from this prior approach. The empirical characterization also provides limited additional conceptual insight: several experimental takeaways confirm expected behavior, while the experiments do not establish the proposed explanation that models first learn shared structure and subsequently memorize example-specific information. Together, these limitations leave the paper’s substantive advance unclear.

  2. The experiments do not convincingly establish the claimed practical privacy–utility advantage. The main defense evaluation relies on sensitive-span membership nAUC and average held-out cross-entropy. These measurements provide limited evidence about sensitive-information recovery or attack performance at low false-positive rates, adaptive attacks are explicitly left untested. The small evaluation sets also limit confidence in comparisons near chance. Moreover, the paper does not compare against DP-LoRA or simple preprocessing methods such as automatic PII redaction, leaving its added value over relevant alternatives unresolved. Baseline strength is also unclear in Figure 2b, where the reported LoRA configurations do not reach TRAP’s higher-utility region and TRAP+LoRA is absent. Since the main contribution rests on improved trade-offs, these gaps prevent the reported curves from providing compelling evidence of an advantage over relevant, adequately tuned alternatives.

Questions for authors

For the method understanding:

  1. How does TRAP differ substantively from a token-level application of AdaFlood with reference-derived thresholds, particularly given their objective equivalence under specific parameter choices? What additional capability or insight do TRAP’s theoretical and empirical results establish beyond this adaptation?

  2. How does TRAP distinguish unwanted memorization from useful knowledge that appears only in the target subset? What evidence shows that penalizing positive TRA preserves generalization for rare subgroups, beyond preserving average held-out cross-entropy?

  3. If the same sensitive association occurs in both target and reference data, both models could assign it high probability while TRA remains small. Under what assumptions does TRAP protect such information, and how do the reported experiments address this case?

For experiments:

  1. What complete LoRA configuration grid was evaluated for MedCase/Pythia-1B, including learning rates, ranks, target modules, and training budgets? Why does the reported LoRA frontier contain no configurations below approximately 3.1 held-out cross-entropy, and what evidence establishes adequate tuning and convergence? Was TRAP+LoRA evaluated in this setting? If so, what were the results; if not, why was it omitted despite being evaluated on other datasets?

  2. How does TRAP compare with (i) DP-LoRA, combining differentially private optimization with LoRA, and (ii) simple preprocessing, such as removing or masking sensitive information using an automatic PII detector? These comparisons would help establish whether TRAP provides a meaningful privacy–utility advantage over private parameter-efficient fine-tuning and a simpler preprocessing approach.

  3. Does the learning-rate trend persist at comparable held-out utility? The matched-training-loss analysis does not by itself establish the conclusion that smaller learning rates reduce memorization at the same task performance.

  4. Could the authors report TPR at low FPRs for TRAP and the main baselines, alongside nAUC and uncertainty estimates? Aggregate AUC can conceal differences in attack performance at these thresholds. Do TRAP’s reported advantages persist under this evaluation? Given the small nonmember sets in some experiments, please clarify which FPR levels can be reliably assessed.

For practical deployment:

  1. How should practitioners select TRAP’s penalty weight, threshold, and iteration count without sensitive-span annotations? In the reported experiments, what data and criteria were used to select these hyperparameters and construct the frontiers, and was the final privacy evaluation kept independent of that selection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.