- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how language models memorize sensitive information during fine-tuning and proposes TRAP, a regularization method to reduce memorization without requiring annotations of sensitive tokens. The authors introduce Target Reference Advantage (TRA), which measures the difference in token likelihood between two models trained on complementary halves of the data. They then use this signal to penalize tokens that appear to be memorized during training. The paper also investigates how factors such as training duration, learning rate, dataset size, task difficulty, and model size affect memorization. Experiments on three datasets show that TRAP achieves a better trade-off between memorization and model utility than differential privacy, LoRA, and several other regularization methods.
Critical strengths
-
The paper addresses an important problem in fine-tuning language models on sensitive data. One of the main strengths of TRAP is that it does not require annotations of sensitive tokens, which can be difficult to obtain reliably in practice.
-
Unlike general regularization methods that affect all training data, TRAP specifically targets tokens that the model is more likely to have memorized, based on the difference between the target and reference models. This is an interesting approach to reducing memorization while preserving model utility.
-
Section 3 provides a thorough analysis of the factors that influence memorization. I particularly liked the finding that early stopping does not necessarily protect rare, hard-to-predict sensitive information. The experimental evaluation is also quite extensive, comparing TRAP against several baselines, including differential privacy and LoRA. I appreciate that the authors compare privacy–utility trade-off curves rather than just individual configurations.
-
The paper provides a useful unified perspective on existing memorization measures by framing them as likelihood comparisons against different references. TRA is also a practical contribution, as it approximates counterfactual memorization at a much lower computational cost and can be directly used during training.
Critical weaknesses
-
TRAP relies on differences between models trained on complementary subsets of the data. However, sensitive information appearing in both subsets may be learned by both models, resulting in a small TRA and potentially escaping the penalty. The synthetic datasets enforce unique sensitive values, leaving the effectiveness of TRAP under repeated sensitive information unclear. Additionally, the released model is trained on only half of the available data, and the proposed solution for using both halves is not experimentally validated.
-
The evaluation primarily relies on likelihood-based memorization scores and membership-inference-style tests rather than direct extraction attacks. Although these metrics provide evidence of memorization, they do not establish whether sensitive information can be recovered through realistic prompting. The absence of adaptive attacks and direct extraction measurements limits the strength of the privacy conclusions.
-
Several key experiments use relatively small evaluation sets, including only 20 member spans for MedCase and the rare-minority setting and 48 held-out sensitive spans for the essays. Many experiments also rely on single-seed runs. Given the dependence of TRA on the random data partition, additional seeds, larger evaluation sets, and uncertainty estimates would help establish the reliability of the reported improvements.
-
Most experiments involve models with 1B parameters or fewer, with the 8B evaluation restricted to one dataset. It remains unclear whether the observed privacy–utility trade-offs generalize to larger models and more realistic fine-tuning scenarios.
Questions for authors
-
How would TRAP behave when the same sensitive information appears in multiple documents, potentially across both training subsets? Since both models could memorize the same information, TRA might remain small even when sensitive information is memorized. Could the authors clarify how TRAP handles this case or provide experiments with different levels of duplication?
-
The authors mention the possibility of training two models, each defended against the other, to avoid leaving half the data unused. Could the authors provide preliminary results for this approach or discuss how they expect it to affect the privacy–utility trade-off and computational cost?
-
The evaluation focuses mainly on likelihood-based memorization scores. Could the authors provide results from direct secret-extraction attacks, even on a subset of the experiments? This would help establish whether the improvements in memorization scores also translate into reduced information leakage.
-
Several experiments use relatively small evaluation sets and limited random seeds. How sensitive are the results to the random data partition and training initialization? Additional seeds or uncertainty estimates for the main results would help strengthen confidence in the findings.
-
Most experiments use models with 1B parameters or fewer, with the 8B evaluation limited to MedCase. Could the authors discuss whether they expect TRAP to behave similarly on larger models and what challenges might arise when scaling the approach?