acceptodds
Under review as a conference paper at ICLR 2027

Sleeping Secrets: How Fine-Tuning Reawakens Privacy Risks in Language Models

Abstract

Beyond adapting Large Language Models (LLMs) to specialized applications, fine-tuning has recently been shown to recover private information that is no longer accessible through direct queries. Previous fine-tuning recovery attacks, however, require genuine private supervision drawn from the same distribution, i.e., the previous training dataset. We argue that such recovery remains possible without such impractical knowledge. We show that LLM-generated candidates can provide sufficient supervision to recover previously learned private associations. Based on this, we propose ReGap, a data-free attack that recovers private associations using task structure, filters them by answer-token likelihood, and updates the target model via low-rank adaptation. Specifically, ReGap requires neither target answers nor auxiliary genuine private supervision. Across six GPT-2, OPT, and Qwen3 models, ReGap improves target-association recovery by 6-21 percentage points over the post-training target model. Recovery remains substantial even when the adaptation identities are disjoint from all memorized and evaluation identities, with no exact target answers appearing in the generated or selected supervision. Moreover, the same trained adapters increase recovery from 42% to 63% on a previously exposed checkpoint, but produce no gain on a matched checkpoint that never encountered the targets. This contrast shows that adaptation alone is insufficient to explain the observed recovery and that prior target exposure strongly affects post-adaptation recoverability. Our findings highlight that routine model customization can reawaken latent privacy risks, warranting urgent attention from the academic and industrial communities.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM, Privacy, Memorization, Fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Ss6AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether fine-tuning can restore access to previously learned private associations without requiring genuine private supervision. The authors propose ReGap, a Generate–Select–Adapt framework that generates candidate associations from the target model, selects them based on answer-token likelihood, and performs LoRA fine-tuning using the selected pseudo-supervision. Experiments across six models from the GPT-2, OPT, and Qwen3 families demonstrate improvements of 6–21 percentage points in target-association recovery. The paper further presents several control experiments and a gradient-based theoretical analysis to investigate the recovery phenomenon and its limitations.

Critical strengths

This paper studies privacy risks induced by subsequent model adaptation instead of only examining privacy leakage at a fixed checkpoint. It explores whether previously learned private information can be re-accessed without genuine private supervision, which is a relevant research direction. The authors perform multiple experiments to investigate the recovery phenomenon. Specifically, the unseen-identity control shows that recovery can happen without directly reusing target identities or exact target answers, and the prior-exposure control highlights the importance of the model’s training history. I value the experiments on opaque random bindings, non-Enron associations, and NPO unlearning, which offer useful insights into settings where ReGap is less effective and help define the boundary of the presented findings.

Critical weaknesses
  1. My main concern relates to the construction of the Enron-derived person-email benchmark. As described in Appendix A.1, person identifiers are deterministically derived from email local parts, introducing substantial lexical and compositional relationships between the queries and their corresponding answers. Although the authors acknowledge this limitation and provide relevant controls, I believe including a simple rule-based prediction baseline would help clarify the difficulty of the benchmark and strengthen the interpretation of the recovery results.
  2. The matched prior-exposure experiment in Section 5.5 provides valuable evidence that recovery depends on whether the model previously encountered the target associations. However, the exposed and unexposed checkpoints already exhibit different recovery rates before adaptation, and the same adapter may affect their parameter states differently. Furthermore, the unseen-identity experiment rules out direct identity reuse and exact-answer replay, but the adaptation data still preserve the underlying person-email task structure. Therefore, the observed improvements may reflect an interaction between previously learned information and the reinforcement of shared prediction patterns.
  3. The experiments primarily focus on person-email associations, where the relationship between identities and target attributes can be relatively direct. It remains unclear whether ReGap would achieve similar recovery improvements for less structured private information or associations involving more complex dependencies or simple reasoning. The existing boundary experiments partially address this concern, but they either show limited recovery or involve additional factors such as exact target-answer replay. Evaluation on a less structurally predictable association task would provide stronger evidence for the broader applicability of the approach.
Questions for authors
  1. Could the authors include a simple rule-based or lexical prediction baseline, or otherwise quantify how many Enron-derived associations are predictable from the person identifiers alone, to clarify the difficulty of the benchmark?

  2. Can the authors further distinguish recovery attributable to prior exposure from improvements caused by reinforcing the shared person-email prediction structure, for example, through controls that account for differences in pre-adaptation recovery rates?

  3. Would ReGap still recover previously learned information on a less structurally predictable private-association task, with controlled prior exposure and without replaying exact target answers?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Ss6A

Thank you for raising these three points, which directly shaped our new controls.

Q1. Predictability from names alone. Using eight standard name-formatting rules plus single-component copying and full-component concatenation (detailed in our response to Reviewer dUw2 Q1), local-part coverage reaches 17/100 (first rule), 23/100 (8-rule union), and 89/100 (with single-component copying), without predicting the domain. Jaccard overlap falls in [0.75, 1] for 99/100 targets, and edit-distance analyses similarly confirm lexical concentration. This motivated our non-lexical random-binding experiments in Setting A (General Response), where recovery persists without name-spelling cues (+18.50 pp over Random).

On the 11 rule-uncovered targets (7 parser-supported; Table 49, Appendix G), Content-NLL recovers 4/7 on Qwen3-1.7B (vs. 2/7 baseline; 3, 2, 1 Random) and 3, 3, 2 on Qwen3-4B (vs. 1/7 baseline; 2, 2, 2 Random). For Graniere (Qwen3-1.7B), Content-NLL corrects baseline [email protected] to [email protected] across 3/3 seeds; for Jennifer P Toomey (Qwen3-4B), it recovers [email protected] in 2/3 seeds. Neither correct answer appeared in selected supervision, and Content-NLL never selected the target queries.

Q2. Distinguishing prior exposure from shared task-pattern learning. In Setting B (General Response), two branches with randomized, mutually exclusive exposure histories train independent LoRA adapters on the exact same 50 experimenter-constructed auxiliary pairs (disjoint from evaluation identities and answers):

Rep.Exposed (Pre -> Post)Net Gain (Gains / Losses)Unexposed (Pre -> Post)
1114/200 -> 144/200+15.0 pp (+34 / -4)0/200 -> 0/200
283/200 -> 129/200+23.0 pp (+46 / -0)0/200 -> 0/200
367/200 -> 106/200+19.5 pp (+39 / -0)0/200 -> 0/200

Exposed accuracy rises by +19.17 pp on average, whereas unexposed associations remain at 0/200. On the jointly baseline-wrong subset, exposed-side post-adaptation accuracy averages 36.06% (34/86, 46/117, and 39/133 vs. 0% unexposed). This comparison matches decoded baseline failure status, though exposed and unexposed items still differ in baseline answer likelihood. In Repetition 1, 33 of the 34 newly correct items also had a correct memorization-stage prediction. Meanwhile, the 150 shared associations gain +20.89 pp and the 50 jointly unexposed controls yield zero hits (despite improved stopping and answer probabilities). These results supplement our earlier cross-checkpoint observation (42% -> 63% vs. 13% -> 13%) and support a contribution of prior exposure under identical auxiliary supervision.

Q3. Recovery on less predictable associations without target-answer replay. Yes, in the six-digit random-binding regime. In Setting A (General Response), mean full-email accuracy increases from 45.92% (Baseline) and 54.50% (Random) to 73.00% (Content-NLL) and 74.17% (Content+EOS), beating Random in all 6/6 branches. Jointly excluding permutation swap blocks with post-selection replay retains 164, 162, and 158 identities per branch and preserves +18.63 and +19.95 pp gains over Random. Per-target restoration trajectories from historical failure (Memory correct -> Baseline wrong, with recovery Adapted correct) are:

Rep.Lost HitsRandom RestoredContent-NLL RestoredContent+EOS Restored
123131 (13.42%)105 (45.45%)115 (49.78%)
219044 (23.16%)101 (53.16%)106 (55.79%)
322132 (14.48%)120 (54.30%)120 (54.30%)

Content-NLL restores 50.97% of lost hits vs. 17.02% for Random (48.14% vs. 15.82% after replay exclusion), with opposite-branch cross-binding hits occurring in only 1/200 queries (already present at baseline). We also report boundaries transparently: when scaling six-digit bindings to 3,000 items across four models (Table 42), unexposed controls remain at 0.00%. Crucially, this extension separates content from full-email recovery: on Qwen3-1.7B, gains appear in six-digit correctness (62.00% baseline -> 73.53% Content-NLL vs. 69.87% Random), not complete-email recovery (0.00% after Content-NLL vs. 0.20% baseline, 0.33% Random); furthermore, on complete-email accuracy, selector differences favor Random on Small (73.60% vs. 74.87%) and OPT-1.3B (80.13% vs. 85.00%). Meanwhile, the 12-character opaque control shows little recovery.

Thanks for the reviewer's reply, but I don't think it is enough to resolve my confusion, and I will maintain my current score.

Reviewer_h5YvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether previously learned private associations can become more recoverable through fine-tuning on model-generated supervision, without access to target answers or genuine auxiliary private records. The experimental setting starts with models that have learned person–email associations, then applies non-target continual training on WikiText-2 to reduce direct recoverability, with only the resulting checkpoint available to the attacker. The proposed method, ReGap, generates candidate question–answer pairs from this checkpoint, selects candidates with the highest average answer-token log-likelihood under the same frozen model, and performs LoRA fine-tuning on the selected pseudo-supervision. Across six GPT-2, OPT, and Qwen3 models, ReGap improves recovery by 6–21 percentage points over the pre-adaptation checkpoint, while its selection rule provides an additional, model-dependent improvement over randomly selected generated examples. Unseen-identity experiments demonstrate recovery without exact target-answer replay, and a matched prior-exposure control supports a role for earlier target exposure, although these behavioral results do not establish an internal memory-restoration mechanism or exclude contributions from shared person–email regularities.

Critical strengths
  1. The paper provides an insightful gradient-alignment analysis explaining how fine-tuning on model-generated pseudo-supervision can reduce the loss on previously learned private associations, even without overlapping target identities or answers. This theoretical perspective offers a meaningful explanation for how auxiliary updates can improve the recoverability of private information without directly using the target data.

  2. The proposed Generate–Select–Adapt framework is simple and lightweight, leveraging model-generated supervision and LoRA adaptation without requiring genuine private records. Its consistent recovery improvements of 6–21 percentage points across six model settings demonstrate the practical effectiveness of the approach and highlight potential privacy risks introduced by subsequent model customization.

Critical weaknesses
  1. The main benchmark derives person identities from email local parts, introducing strong lexical and compositional relationships between queries and target answers. For example, [email protected] -> Alice Smith. Such structural similarities may naturally facilitate the gradient alignment underlying ReGap, while the sharply reduced recovery under opaque random bindings further suggests that the method depends heavily on these regularities, limiting its generalizability to other types of private associations.

  2. The theoretical motivation for likelihood-based candidate selection relies on an assumed relationship between model confidence and target-gradient alignment, which is not fully established in the main experiments. Moreover, randomly selected model-generated supervision already improves recovery across all six model settings, and ReGap's additional gains are relatively small in some cases, making the independent contribution of its selection strategy less convincing (Sections 4.2 and 5.4).

  3. The main experiments construct hidden associations through non-target continual training on WikiText-2 rather than explicit unlearning, leaving unclear whether the observed recovery reflects restoration of private information or renewed adaptation to the association task. Furthermore, ReGap provides almost no additional recovery after NPO-based unlearning, suggesting that its effectiveness may be limited to particular forgetting mechanisms rather than general post-training privacy risks (Section 6).

  4. The paper's presentation is unnecessarily verbose, with closely related concepts frequently described using different terminology, such as behavioral accessibility, recoverability, and reawakening. This makes it difficult to follow the distinctions between the experimental setting, the proposed mechanism, and the actual privacy claims, reducing the clarity of an otherwise relatively straightforward method.

Questions for authors
  1. ReGap generates one greedy response and five sampled responses per query, then selects the 50 candidates with the lowest answer-token loss under the frozen model. Since greedy decoding already favors high-probability tokens, it is unclear how much stochastic sampling contributes to the final selected supervision. Could the authors report the proportion of selected candidates originating from greedy versus sampled responses, and evaluate whether sampling provides additional recovery benefits over greedy-only candidate generation?

  2. The main experiments focus on structured person–email associations and use non-target continual training to reduce direct recoverability, while the reported opaque-binding and NPO-unlearning results show substantially weaker recovery. These findings raise questions about the generalizability of ReGap across different private information structures and forgetting mechanisms. Could the authors evaluate more diverse private association tasks and additional unlearning methods to clarify the conditions under which ReGap remains effective and whether the observed resistance to recovery after NPO generalizes?

1 comment
AuthorsOpenReview ↗

Response to Reviewer h5Yv

We appreciate your thorough review and helpful suggestions on clarity and controls.

Q1. Greedy vs. sampling provenance and selection's independent value. We corrected Appendix B.2 to state Base settings accurately: GPT-2 used 1 greedy + 5 sampled completions (64 tokens), whereas OPT/Qwen3 used 5 sampled only (20 tokens). For five models, 15 pools, 7,361 scores, and 30 Top-50 orderings were reproduced; for Qwen3-4B (original generation script unarchived), selections were mapped to a same-family reconstruction matching the saved pools identically. These proportions reflect validated reproductions, as original per-call logs were not retained (Original-Loss, 3-seed means; "Both" = identical output from both decoders):

ModelGreedy OnlySampling OnlyBoth
GPT-2 Small11.33%62.67%26.00%
GPT-2 Medium9.33%47.33%43.33%
GPT-2 Large6.00%58.67%35.33%
OPT / Qwen1.7 / Qwen40.00%100.00%0.00%

In a matched-call GPT-2 Medium ablation using extracted, deduplicated emails and Random selection (K=50), first-sample supervision beats greedy-only by +4.86 pp in strict accuracy (+5.66 pp after common replay exclusion):

SourceSeeds 42 / 43 / 44 (/96)Strict Accuracy
Pre-adaptation25 / 25 / 2526.04%
Greedy-only39 / 41 / 3439.58%
First-sample43 / 42 / 4344.44%

Independent value of selection: To evaluate selection without placeholder padding or substring artifacts, we added the unified Extract-Deduplicate (Content-NLL) protocol (1 greedy + 5 sampled, 64 tokens across all 6 models; extracting the first valid email, deduplicating exact prompt-answer pairs, and training with aligned tokenization + EOS; strict first-email accuracy %, common 96 targets, three-seed means):

ModelBaselineBase Rand.Orig.-LossED Rand.ED Content-NLL
GPT-2 Small9.3820.8318.7524.6525.00
GPT-2 Medium26.0435.0732.9943.7545.83
GPT-2 Large4.172.080.000.351.04
OPT-1.3B3.139.031.0440.9740.97
Qwen3-1.7B20.8323.9619.4436.8150.35
Qwen3-4B35.4248.2648.9654.8659.38
Large (Bal. EOS)4.17--7.6415.63

Extract-Deduplicate gives positive mean selection gains on 5/6 models (tied on OPT-1.3B), largest on Qwen3-1.7B (+13.54 pp) and Qwen3-4B (+4.51 pp). On GPT-2 Large, standard token-mean loss rarely emits EOS (4% active EOS in the ordinary Content-NLL arm), accompanied by malformed continuations; as a post-hoc mechanistic control (not part of the unified objective), equally weighting content-mean and EOS losses raises strict accuracy to 15.63% (vs. 7.64% Random) while lowering substring recovery (71.00% -> 62.33%). On Qwen3-1.7B, a length- and weight-matched Random control leaves Content-NLL ahead by +11.46 pp (50.35% vs. 38.89%; conditional paired 95% CI [6.25, 17.01] pp). Further, as shown in General Response Section 3, adaptation beats same-pool reranking on 4/6 models and recovers targets absent even as substrings from all six candidate generations of the query. We frame selection as a model- and objective-dependent amplifier rather than a universal advantage.

Q2. Less structured associations and unlearning. In Setting A (General Response), Content-NLL reaches 73.00% vs. 54.50% for Random (+18.63 pp post-exclusion), contrasting with the 12-char opaque null (95/100 -> 1/100 baseline). In the 3,000-item four-model numeric benchmark (Table 42; unexposed controls at 0.00%), results separate content from full-email recovery: Content-NLL improves six-digit accuracy across models (e.g., Qwen3-1.7B reaches 73.53% vs. 62.00% baseline), but does not improve strict complete-email accuracy everywhere (Qwen3-1.7B yields 0.00% vs. 0.20% baseline, 0.33% Random), with selector margins varying by architecture. We also tested GradDiff on GPT-2 Medium (12-char bindings): GradDiff reduced normalized-substring matches (M0) from 95/100 to 71/100 (Retain 276/300 -> 270/300). None of the 24 lost substring hits was restored across 12 adaptation runs (Overlap/Disjoint queries x Content-NLL/Random x 3 seeds), despite 94-99/100 output changes; strict full-email matches remained zero throughout. Together with NPO, these negative results identify failure cases under the evaluated configurations.

Presentation. We revised the paper for clarity: (i) defining reawakening strictly as behavioral re-readout; (ii) standardizing terminology (Hidden, Base Original-Loss, Extract-Deduplicate Content-NLL) and adding a master protocol table (Table 10); (iii) separating historical diagnostics from current-protocol results; and (iv) framing gradient alignment as an empirically tested premise.

Reviewer_dUw2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether fine-tuning can restore the behavioral accessibility of private associations that a language model encountered during earlier training but can no longer directly produce after subsequent training. The paper introduces ReGap, a Generate–Select–Adapt pipeline in which the post-training model generates candidate person–email associations, a frozen copy of the model ranks these candidates using answer-token likelihood, and a LoRA adapter is trained on the selected pseudo-labels. The main claim is that private information can become recoverable again without using externally supplied genuine private associations.

Critical strengths
  • Unlike attacks that fine-tune directly on known private associations, ReGap uses model-generated candidate associations. The unseen-identity experiments are particularly useful because they attempt to exclude exact target-identity and target-answer replay from the adaptation data.

  • The paper appropriately emphasizes that later customization may change what information becomes accessible, making post-deployment adaptation a relevant privacy threat surface.

Critical weaknesses
  • The paper’s broad "reawakening private memories" interpretation is not fully supported. The experiments establish increased behavioral recoverability after adaptation, but they do not isolate whether this comes from item-specific retained information, improved task-format competence, email-pattern completion, domain-frequency learning, or changes in output calibration. The matched prior-exposure experiment is suggestive but does not exclude these competing explanations.

  • The theoretical analysis is conditional and insufficiently connected to the final algorithm. The main result assumes that higher-support candidates have better expected target-gradient alignment. This is close to assuming the relationship needed for the selection rule to work. In addition, the empirical gradient diagnostics appear to use a historical compatibility score or ReGap-Abs variant rather than the final Original-Loss selector used for the headline experiments.

Questions for authors
  • How do the authors distinguish item-specific recovery from name-to-email reconstruction? Since the identities are derived from the email local parts, many answers can be partially constructed from the queries. Could the authors report results stratified by token overlap, edit distance, and the predictability of the email from the person’s name? In particular, does ReGap provide a statistically significant improvement for targets whose local parts cannot be generated by ordinary name-normalization rules?

  • What experiment uniquely supports the word "reawakens"? The current results are also compatible with generic task adaptation, output calibration, and movement of near-threshold answers into the greedy decoding path. Can the authors design a control that distinguishes these mechanisms from retrieval of historically exposed item-specific associations?

  • The current metric emphasizes whether the correct answer appears, but a practical attacker must distinguish correct private information from hallucinated alternatives. How many incorrect email addresses are generated per successful recovery, and what external verification effort would be required? Specifically, what is the false-positive rate of the extraction procedure?

  • Does the final Original-Loss score predict gradient alignment and recovery? The present gradient analysis appears to involve historical compatibility measures or ReGap-Abs. Please compute the analysis using the exact selector employed in the headline experiments and report the relationship across all model families.

2 comments
AuthorsOpenReview ↗

Response to Reviewer dUw2

Thank you for the detailed review; we address each question below.

Q1. Name-based rules vs. item-specific recovery. Standard name heuristics cover 17/100 local parts (first rule), 23/100 (8-rule union), and 89/100 with single-component copying, but supply no domains. Of 11 uncovered targets, 4 exceed the 64-char RFC limit (unsupported by strict M1, not model failures). On the 7 supported targets, Qwen3-1.7B Content-NLL recovers 4/7 first emails across all three seeds (vs. 2/7 baseline, 3/2/1 Random; Table 49, Appendix G). For Graniere (reference [email protected]), baseline outputs [email protected]; reranking and Random fail all seeds, while Content-NLL recovers the exact address across all three seeds with zero query or answer replay in selected supervision. Removing spelling cues in Setting A (General Response), Content-NLL outperforms Random by +18.50 pp (+18.63 pp post-exclusion).

Q2. Evidence supporting "reawakening". Our primary evidence combines per-target restoration and randomized exposure:

Per-target restoration (Setting A): Among non-lexical historical failures (Memory correct -> Baseline wrong), Content-NLL restores 50.97% (48.14% post-exclusion) vs. 17.02% (15.82%) for Random. Opposite-branch cross-bindings occur in only 1/200 queries in Rep. 3 Branch A (present at baseline) and 0/200 elsewhere. Randomized exposure under identical supervision (Setting B): On the jointly baseline-wrong subset, exposed vs. unexposed post-adaptation hits are:

Rep.Baseline-WrongExposed CorrectUnexposed Correct
186340
2117460
3133390

On this jointly baseline-wrong subset, exposed accuracy averages 36.06% (vs. 0% unexposed). In Repetition 1, 33 of 34 newly correct items had correct memorization predictions. Together with cross-checkpoint transfer (42% -> 63% exposed vs. 13% -> 13% unexposed), this supports a contribution of prior exposure under identical auxiliary supervision.

Q3. Incorrect candidates and verification burden. We extract and deduplicate valid emails per response across the 96 parser-supported Enron targets (4 references exceed the 64-char RFC limit). Verification burden is total reference-unmatched candidates divided by queries with at least one matching email (pooled across three seeds). Under initial Base (Original-Loss), unmatched/success ratios are:

ModelRandomOriginal-Loss
GPT-2 Small1.8671.889
GPT-2 Medium0.7920.874
GPT-2 Large6.000NA (83 unmatched, 0 successes)
OPT-1.3B2.88510.667
Qwen3-1.7B3.0864.313
Qwen3-4B0.7770.844

The Base table above evaluates all deduplicated valid candidates per response, whereas Extract-Deduplicate (Content-NLL) (Table 18, Appendix C.4) evaluates only the first strictly extracted email per query. Under this first-email metric, ratios are 1.708 (Small), 0.674 (Medium), 1.347 (OPT-1.3B), 0.986 (Qwen3-1.7B), and 0.591 (Qwen3-4B); Large is 15.000 without EOS weighting. These distinct aggregation scopes must not be viewed as a cross-protocol decrease. In Setting A (known synthetic mapping), first-email Content-NLL raises successes across 1,200 queries from 551 (Pre-adaptation) and 654 (Random) to 876, while wrong candidates rise from 100 and 118 to 153 (0.175 vs. 0.181 Baseline and 0.180 Random per success). Unmatched Enron candidates are not verified false addresses, and no inspection cost was measured.

Q4. Gradient diagnostics with the final Original-Loss selector. We measured post-hoc gradient alignment on exact checkpoints, pools, scores, and K=50 selections from Original-Loss runs across all three model families (with per-matrix ED and Setting A diagnostics in Tables 26-27):

ModelSpearman (loss, cos)Set Cos > RandomStrict Acc. (Delta vs. Random)
GPT-2 Medium-0.706 / -0.722 / -0.6973/3-2.08 pp
OPT-1.3B-0.235 / -0.176 / -0.1852/3-7.99 pp
Qwen3-1.7B-0.265 / -0.199 / -0.1982/3-4.51 pp

Historical loss negatively correlates with single-example target-gradient cosine: lower loss tends toward higher cosine, without guaranteeing set-level geometry or recovery. Effective-weight gradient alignment does not prove an optimizer mechanism or ensure recovery: Qwen3-1.7B trails Random in dot products and target projections across all seeds, and GPT-2 Medium gains +10.0 pp substring despite -2.08 pp strict accuracy. In GPT-2 Base pools, Random picks 4/11/8 (Small), 16/21/19 (Medium), and 13/23/21 (Large) fallbacks vs. 0 for Original-Loss; removing fallbacks weakens Medium's correlation to -0.08 to -0.27 and narrows Medium/Large gaps vs. Random to +5.33/+3.00 pp substring (-2.78/-0.69 pp strict). Assumption 2 is thus a model-dependent premise, not a guarantee.

While I appreciate the author's effort in addressing my concerns, I do not find it convincing. I will therefore maintain my score

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their constructive feedback. Our revision adds controlled experiments demonstrating that adaptation on self-generated supervision can restore behavioral access to previously learned associations—reflecting history-specific recovery rather than generic task adaptation alone—with likelihood-based selection providing a complementary, setting-dependent amplifier.

  1. Permuted Six-Digit Random Bindings (Setting A). To remove spelling cues, we built three independent GPT-2 Medium repetitions with paired A/B branches. Each branch memorized 350 associations (16 epochs, then 800 non-target steps): 200 evaluation, 100 calibration, and 50 shared associations. The 200 evaluation names and random six-digit email values (e.g., [email protected]) were identical across branches under permuted one-to-one assignments. From each branch's fixed pool of raw continuations generated for 100 separately constructed auxiliary identities (disjoint from memorization identities, with no reference answers), we selected 50 examples for LoRA via uniform Random, lowest mean non-EOS content NLL (Content-NLL), or lowest NLL with EOS (Content+EOS), using consistent tokenization.
MethodRep. 1Rep. 2Rep. 3Mean
Pre-adaptation41.25%52.00%44.50%45.92%
Random49.00%62.50%52.00%54.50%
Content-NLL67.75%77.00%74.25%73.00%
Content+EOS70.25%78.00%74.25%74.17%

Values report strict full-email accuracy averaged over A/B branches. Both selectors beat Random in 6/6 branches (+18.50/+19.67 pp). Jointly excluding A/B swap blocks with post-selection replay retains 164/162/158 IDs per branch and preserves +18.63/+19.95 pp gains. On per-target trajectories (Memory correct -> Baseline wrong -> Adapted correct), Content-NLL restores 105/231, 101/190, and 120/221 historical failures (50.97% equal-weighted mean; 48.14% post-exclusion) vs. 31/231, 44/190, and 32/221 for Random (17.02%; 15.82% post-exclusion).

  1. Randomized Exposure Control (Setting B). Across three more GPT-2 Medium repetitions, 200 evaluation IDs were randomly assigned to exposure only in A (100) or B (100), alongside 200 shared pairs (50 calibration, 150 evaluation) and 50 branch-calibration pairs. After 16 memorization epochs and identical 800-step non-target training, both branches trained independent LoRA adapters on the exact same 50 experimenter-constructed auxiliary pairs (disjoint from evaluation IDs/answers). Exposed accuracy rose from 114/200 to 144/200, 83/200 to 129/200, and 67/200 to 106/200 (+19.17 pp mean); unexposed stayed at 0/200. Shared 150 pairs gained +20.89 pp; 50 jointly unexposed pairs yielded zero hits. Task adaptation and history-specific readout thus act together.

  2. Adaptation vs. Same-Pool Reranking (Extract-Deduplicate). To test whether fine-tuning merely picks answers already surfaced during sampling, we evaluated an inference-only baseline taking the lowest Content-NLL extracted email from the same 6-completion pool per query (Table 23, Appendix C.6; 96 common targets, 3-seed means; unified token-mean loss for Large):

ModelRerank6-Sample Coverage*ED Content-NLLAdapt - Rerank
GPT-2 Small26.04%28.82%25.00%-1.04 pp
GPT-2 Medium42.71%51.04%45.83%+3.13 pp
GPT-2 Large18.06%18.06%1.04%-17.01 pp
OPT-1.3B17.36%19.44%40.97%+23.61 pp
Qwen3-1.7B34.38%39.93%50.35%+15.97 pp
Qwen3-4B47.22%52.08%59.38%+12.15 pp

*Coverage checks whether the reference is among the first valid emails extracted from six completions; invalid outputs and duplicates may leave fewer candidates (requiring ground truth; not an executable selection rule). Pool generation used 64 max tokens and repetition penalty 1.05, versus 48 tokens and 1.0 for post-adaptation greedy evaluation.

Adaptation beats reranking on four of six models. On OPT-1.3B, Qwen3-1.7B, and Qwen3-4B, it also exceeds six-completion candidate coverage by 21.53, 10.42, and 7.29 pp, respectively. Targets absent even as substrings from all six candidate generations of that query are still strictly recovered after adaptation (19/15/17 across seeds on OPT-1.3B, 7/3/6 on Qwen3-1.7B, 7/7/5 on Qwen3-4B). Thus, fine-tuning expands the recoverable answer set beyond per-query sampling. On OPT-1.3B, the tie with ED Random (40.97%) reflects adaptation's effect, whereas selection's gain is isolated by Content-NLL vs. Random (+13.54 pp on Qwen3-1.7B, +4.51 pp on Qwen3-4B).

  1. Paper Updates. We add strict first-email matching and Extract-Deduplicate (Content-NLL) alongside initial Base (Original-Loss, noting it trails Random on five models under strict matching), define reawakening behaviorally, retain NPO results, and add GradDiff unlearning tests, framing selection and mechanism diagnostics as complementary analyses.
AuthorsOpenReview ↗

Additional Evidence: Recovery After Removing Target Answers from Adaptation Data

To evaluate whether adaptation gains require replaying memorized answers, we conducted a ground-truth-aware diagnostic on GPT-2 Small's synthetic six-digit task. Supervision used 100 auxiliary identities absent from controlled memorization. Before adaptation, candidate units containing reference emails were filtered from the pool (329 reduced to 292). Both arms trained on 50 examples (3 epochs, 21 updates) across three adaptation seeds sharing one history. Strict accuracy on 500 exposed targets rose from 65.40% to 73.67% under Content-NLL and 73.27% under Random; 500 unexposed controls yielded zero hits. A post-training audit confirmed zero complete target six-digit strings throughout training inputs. Direct replay of complete target answers is therefore not necessary for these adaptation gains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.