- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether fine-tuning can restore access to previously learned private associations without requiring genuine private supervision. The authors propose ReGap, a Generate–Select–Adapt framework that generates candidate associations from the target model, selects them based on answer-token likelihood, and performs LoRA fine-tuning using the selected pseudo-supervision. Experiments across six models from the GPT-2, OPT, and Qwen3 families demonstrate improvements of 6–21 percentage points in target-association recovery. The paper further presents several control experiments and a gradient-based theoretical analysis to investigate the recovery phenomenon and its limitations.
Critical strengths
This paper studies privacy risks induced by subsequent model adaptation instead of only examining privacy leakage at a fixed checkpoint. It explores whether previously learned private information can be re-accessed without genuine private supervision, which is a relevant research direction. The authors perform multiple experiments to investigate the recovery phenomenon. Specifically, the unseen-identity control shows that recovery can happen without directly reusing target identities or exact target answers, and the prior-exposure control highlights the importance of the model’s training history. I value the experiments on opaque random bindings, non-Enron associations, and NPO unlearning, which offer useful insights into settings where ReGap is less effective and help define the boundary of the presented findings.
Critical weaknesses
- My main concern relates to the construction of the Enron-derived person-email benchmark. As described in Appendix A.1, person identifiers are deterministically derived from email local parts, introducing substantial lexical and compositional relationships between the queries and their corresponding answers. Although the authors acknowledge this limitation and provide relevant controls, I believe including a simple rule-based prediction baseline would help clarify the difficulty of the benchmark and strengthen the interpretation of the recovery results.
- The matched prior-exposure experiment in Section 5.5 provides valuable evidence that recovery depends on whether the model previously encountered the target associations. However, the exposed and unexposed checkpoints already exhibit different recovery rates before adaptation, and the same adapter may affect their parameter states differently. Furthermore, the unseen-identity experiment rules out direct identity reuse and exact-answer replay, but the adaptation data still preserve the underlying person-email task structure. Therefore, the observed improvements may reflect an interaction between previously learned information and the reinforcement of shared prediction patterns.
- The experiments primarily focus on person-email associations, where the relationship between identities and target attributes can be relatively direct. It remains unclear whether ReGap would achieve similar recovery improvements for less structured private information or associations involving more complex dependencies or simple reasoning. The existing boundary experiments partially address this concern, but they either show limited recovery or involve additional factors such as exact target-answer replay. Evaluation on a less structurally predictable association task would provide stronger evidence for the broader applicability of the approach.
Questions for authors
-
Could the authors include a simple rule-based or lexical prediction baseline, or otherwise quantify how many Enron-derived associations are predictable from the person identifiers alone, to clarify the difficulty of the benchmark?
-
Can the authors further distinguish recovery attributable to prior exposure from improvements caused by reinforcing the shared person-email prediction structure, for example, through controls that account for differences in pre-adaptation recovery rates?
-
Would ReGap still recover previously learned information on a less structurally predictable private-association task, with controlled prior exposure and without replaying exact target answers?
2 comments
Thanks for the reviewer's reply, but I don't think it is enough to resolve my confusion, and I will maintain my current score.
Response to Reviewer Ss6A
Thank you for raising these three points, which directly shaped our new controls.
Q1. Predictability from names alone. Using eight standard name-formatting rules plus single-component copying and full-component concatenation (detailed in our response to Reviewer dUw2 Q1), local-part coverage reaches 17/100 (first rule), 23/100 (8-rule union), and 89/100 (with single-component copying), without predicting the domain. Jaccard overlap falls in [0.75, 1] for 99/100 targets, and edit-distance analyses similarly confirm lexical concentration. This motivated our non-lexical random-binding experiments in Setting A (General Response), where recovery persists without name-spelling cues (+18.50 pp over Random).
On the 11 rule-uncovered targets (7 parser-supported; Table 49, Appendix G), Content-NLL recovers 4/7 on Qwen3-1.7B (vs. 2/7 baseline; 3, 2, 1 Random) and 3, 3, 2 on Qwen3-4B (vs. 1/7 baseline; 2, 2, 2 Random). For Graniere (Qwen3-1.7B), Content-NLL corrects baseline [email protected] to [email protected] across 3/3 seeds; for Jennifer P Toomey (Qwen3-4B), it recovers [email protected] in 2/3 seeds. Neither correct answer appeared in selected supervision, and Content-NLL never selected the target queries.
Q2. Distinguishing prior exposure from shared task-pattern learning. In Setting B (General Response), two branches with randomized, mutually exclusive exposure histories train independent LoRA adapters on the exact same 50 experimenter-constructed auxiliary pairs (disjoint from evaluation identities and answers):
Exposed accuracy rises by +19.17 pp on average, whereas unexposed associations remain at 0/200. On the jointly baseline-wrong subset, exposed-side post-adaptation accuracy averages 36.06% (34/86, 46/117, and 39/133 vs. 0% unexposed). This comparison matches decoded baseline failure status, though exposed and unexposed items still differ in baseline answer likelihood. In Repetition 1, 33 of the 34 newly correct items also had a correct memorization-stage prediction. Meanwhile, the 150 shared associations gain +20.89 pp and the 50 jointly unexposed controls yield zero hits (despite improved stopping and answer probabilities). These results supplement our earlier cross-checkpoint observation (42% -> 63% vs. 13% -> 13%) and support a contribution of prior exposure under identical auxiliary supervision.
Q3. Recovery on less predictable associations without target-answer replay. Yes, in the six-digit random-binding regime. In Setting A (General Response), mean full-email accuracy increases from 45.92% (Baseline) and 54.50% (Random) to 73.00% (Content-NLL) and 74.17% (Content+EOS), beating Random in all 6/6 branches. Jointly excluding permutation swap blocks with post-selection replay retains 164, 162, and 158 identities per branch and preserves +18.63 and +19.95 pp gains over Random. Per-target restoration trajectories from historical failure (Memory correct -> Baseline wrong, with recovery Adapted correct) are:
Content-NLL restores 50.97% of lost hits vs. 17.02% for Random (48.14% vs. 15.82% after replay exclusion), with opposite-branch cross-binding hits occurring in only 1/200 queries (already present at baseline). We also report boundaries transparently: when scaling six-digit bindings to 3,000 items across four models (Table 42), unexposed controls remain at 0.00%. Crucially, this extension separates content from full-email recovery: on Qwen3-1.7B, gains appear in six-digit correctness (62.00% baseline -> 73.53% Content-NLL vs. 69.87% Random), not complete-email recovery (0.00% after Content-NLL vs. 0.20% baseline, 0.33% Random); furthermore, on complete-email accuracy, selector differences favor Random on Small (73.60% vs. 74.87%) and OPT-1.3B (80.13% vs. 85.00%). Meanwhile, the 12-character opaque control shows little recovery.