acceptodds
Under review as a conference paper at ICLR 2027

Analyzing LLM Unlearning: Token-Triggered Forgetting Mechanism and Entity-Weighted Remedy for Trigger-Free Queries

Abstract

Large language model (LLM) unlearning aims to suppress targeted training knowledge while preserving unrelated capabilities. We identify a *token-triggered forgetting* mechanism associated with favorable forget–retain trade-offs in several gradient-based objectives and low-rank updates: forget-target tokens attract attention, accompanied by selective collapse of the forget representation subspace. Controlled trigger injection and activation patching provide functional evidence for this pathway. The mechanism is vulnerable on *trigger-free* queries, where the target knowledge is inferred rather than explicitly named. We demonstrate this failure on TOFU and reproduce its pattern on RWKU. To mitigate it, we propose EE-TNPO, which uses early-exit distributions to up-weight knowledge-bearing answer tokens during unlearning. By emphasizing these tokens, which include names and factual attributes, EE-TNPO reduces trigger-free leakage while preserving general utility on TOFU and RWKU. Moreover, WMDP and MUSE results support its application to corpus-level unlearning without entity-defined forget targets.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Unlearning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_iNLQOpenReview ↗
Rating
2(+1)
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies a token-triggered forgetting pattern in LLM unlearning. It observes that forget-target names attract attention and that forget representations lose effective rank. Models showing this pattern often leak information when the target name is absent from the query. The proposed remedy, EE-TNPO, uses disagreement between intermediate and final-layer predictions to reweight token-level unlearning losses. Experiments cover TOFU, RWKU, WMDP, and MUSE.

Critical strengths
  1. Evaluating queries without explicit target names is useful. Strong forgetting on name-cued questions does not necessarily imply that the same information cannot be recovered through descriptions or reverse queries. The paper makes this distinction explicit and supplements standard forgetting scores with name-leakage measurements, which helps expose failures that aggregate metrics may miss.
  2. EE-TNPO is a simple modification of an existing objective and shows meaningful improvements on the main TOFU setting. For example, under LoRA, name leakage falls from 65.0% with TNPO to 27.5%, while retain ES remains similar. The paper also includes controlled trigger injection, activation patching, and results over four independent TOFU runs. These provide useful evidence, although they do not fully establish the proposed mechanism.
Critical weaknesses
  1. The model coverage is outdated, especially for the central mechanistic analysis. The main TOFU experiments rely on Llama-2-7B-Chat from 2023, there is no comparable analysis on recent Llama, Qwen, or Gemma models. More importantly, an additional performance table does not establish that the same attention and rank-collapse mechanism appears in another model. Since the paper makes claims about how unlearning works, reproducing the mechanism across backbones matters more than simply adding another benchmark score.
  2. The baseline set is too limited for a paper proposing token reweighting. The experiments mainly compare against NPO, WGA, and TNPO. SatImp is discussed but not evaluated. The paper should also compare with Selective Unlearning from Not Every Token Needs Forgetting: Selective Unlearning to Limit Change in Utility in Large Language Model Unlearning (Wan et al., 2025) and ATWU from Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importance (Yüce et al., 2026). These methods directly address token-level selection or weighting and are needed to establish the value of the early-exit signal. The comparison should cover trigger-free forgetting at matched retention levels, rather than only the selected configurations of three baselines.
  3. The novelty is limited, and the mechanistic evidence does not establish the full proposed explanation. Ren et al. (Keeping an eye on LLM unlearning: The hidden risk and remedy) already studied prompt tokens acting as unlearning signals. Examining what happens when those tokens are absent is a useful extension, but does not by itself establish a new mechanism. Appendix D.2 patches entire attention or MLP outputs. This shows that late-layer modules affect forgetting; it does not isolate whether attention to trigger tokens causes rank collapse, or whether that collapse causes forgetting. On the method side, EE-TNPO combines a DoLa-inspired layer-disagreement signal with existing TNPO. Without a more targeted causal test, the work remains an incremental combination of known observations and techniques
  4. The weighting rule does not distinguish factual importance from forget-specific importance. A large early-exit discrepancy indicates that predictions change across layers, but does not establish that the corresponding token should be forgotten rather than retained. The name-detection results in Table 4 do not resolve this distinction. There is also no comparison against simple entity-span weighting, shuffled weights, or another inexpensive token score to establish why JSD is needed. The MUSE-Books result exposes this limitation: EE-TNPO and TNPO both achieve an ES-Forget score of 0.79, but EE-TNPO reduces retain ROUGE-L from 65.97 to 57.19. The authors acknowledge the shared-entity issue, but it directly limits the proposed scoring rule and its claimed generality.
Questions for authors
  1. Does the proposed attention–rank-collapse pattern reproduce on newer backbones? Can the authors show both the mechanism and the effectiveness of EE-TNPO on recent Llama, Qwen, or Gemma models?
  2. Does EE-TNPO outperform SatImp, Selective Unlearning, and ATWU under comparable tuning budgets and matched retention levels, particularly on trigger-free queries?
  3. Can the authors intervene specifically on attention to trigger tokens, rather than replacing an entire module’s output, and measure the resulting changes in both effective rank and forgetting?
  4. Does early-exit weighting improve over simple entity-span weighting or shuffled weights under the same loss normalization and optimization settings? What evidence shows that the score identifies forget-specific information rather than broadly important factual tokens?
10 comments
AuthorsOpenReview ↗

Response to Reviewer iNLQ (1/7)

We greatly appreciate the detailed and critical review. We address each concern below. W/Q numbers follow the order of the review, and the revisions we mention will be updated in the Camera-ready PDF.

W1 & Q1 — Mechanism and effectiveness on newer backbones

In the submission, the three analyses of Sec. 3 (effective rank in Fig. 1b, attention to names in Fig. 3, name injection in Table 1 of the paper) were run on Llama-2-7B only, and Qwen2 (App. E.2) had only a performance table, which, as the reviewer notes, does not show that the same mechanism is present. We repeated the three analyses on Llama-3.2-1B, Llama-3.2-3B and Qwen2-7B (this comment) and evaluated EE-TNPO on Llama-3.2 (next comment).

Setup. Llama-3.2-1B and 3B are fine-tuned on TOFU and unlearned on Forget10 with the paper's hyperparameters; Qwen2-7B uses the checkpoints of App. E.2 (4 seeds each; cells are seed means). The same analysis code is used for all backbones. The three blocks of Table R1 report the three analyses.

Table R1. Sec. 3 signatures on four backbones (mean of 4 seeds). (a) Forget-rank drop minus retain-rank drop vs. the original model (pp; effective rank of hidden states). For each method and backbone, both drops are averaged over the 25% of layers with the largest forget-rank drop (8 of 32, 4 of 16, 7 of 28 layers; chosen once from the mean over 4 seeds and applied to every seed), so a collapse is counted wherever it occurs. Large = only the forget subspace collapses, ≈0 (including small negative values) = both change alike; compare within a backbone. (b) Attention to forget-author names ÷ attention to retain-author names (upper half of layers); 1 = equal. (c) ROUGE-L recall (%) on unrelated questions with a forget / retain author name inserted (Table 1 probe); a low first value next to an unchanged second one = the forget name alone triggers suppression.

Llama-2-7BLlama-3.2-1BLlama-3.2-3BQwen2-7B
(a) Rank: forget − retain drop
NPO-FT5.5−0.41.21.4
NPO-LoRA17.03.411.526.6
TNPO-FT38.011.520.513.2
TNPO-LoRA46.942.821.337.2
(b) Name attention: forget ÷ retain
Original0.950.980.980.96
NPO-FT1.081.011.141.01
NPO-LoRA2.161.392.151.44
TNPO-FT1.931.461.311.32
TNPO-LoRA2.322.621.351.59
(c) Injection: forget / retain name
Original47.8 / 44.248.4 / 48.353.2 / 52.152.6 / 50.1
NPO-FT44.9 / 41.357.2 / 50.849.7 / 45.264.5 / 38.7
NPO-LoRA7.3 / 39.824.4 / 49.315.3 / 49.18.2 / 49.8
TNPO-FT11.2 / 49.828.7 / 45.712.7 / 54.939.9 / 50.3
TNPO-LoRA3.3 / 42.00.8 / 46.94.3 / 53.62.2 / 48.3

(1) NPO-FT shows no name trigger on any backbone. Its forget and retain ranks change alike (difference −0.4 to 5.5 points), forget names receive almost no extra attention (1.01–1.14, against 0.95–0.98 in the original models), and inserting a forget name does not lower the score below that for a retain name (44.9–64.5 vs. 38.7–50.8).

(2) NPO-LoRA, TNPO-FT and TNPO-LoRA are triggered by the name on every backbone. In all three, forget names receive more attention than retain names (1.31–2.62), a forget name alone lowers the score below the retain-name score in every cell (by 10–49 points), and in all but one cell the forget rank drops at least 11 points more than the retain rank. TNPO-LoRA has the strongest rank and injection signatures on every backbone (rank 21–47; forget-name score 0.8–4.3). The two deviations are in strength, not in direction: NPO-LoRA has a weaker rank signature (11.5–26.6 on three backbones, 3.4 on Llama-3.2-1B), and TNPO-FT varies with the backbone, strong on Llama-2 and Llama-3.2-3B (rank 38.0 and 20.5; forget-name score 11.2 and 12.7) and weaker on Llama-3.2-1B and Qwen2 (rank 11.5 and 13.2; forget-name score 28.7 and 39.9).

In summary, the pattern of Sec. 3 holds on all four backbones: NPO under FT does not rely on the name, whereas TNPO and LoRA updates do. What varies with the backbone is the strength of the signatures under FT and the rank signature of NPO-LoRA.

AuthorsOpenReview ↗

Response to Reviewer iNLQ (2/7)

W1 & Q1 (continued) — Effectiveness of EE-TNPO on Llama-3.2

Setup. Same Llama-3.2-1B and 3B models as in the previous comment, unlearned with TNPO and EE-TNPO under identical settings (TOFU Forget10, 4 seeds).

Table R2. TNPO vs. EE-TNPO on Llama-3.2 (mean over 4 seeds; %). ES-F / ES-R = extraction strength on forget / retain QA. MU = model utility. NRR-F = share of the 20 name-seeking (trigger-free) forget questions answered with the author's name. NRR-Gap = the same rate on retain questions minus NRR-F. NS ES-Sim = extraction strength on the name-seeking questions.

SettingMethodES-F ↓ES-R ↑MU ↑NRR-F ↓NRR-Gap ↑NS ES-Sim ↓
1B FTTNPO4.3866.4857.7917.5054.8027.2
EE-TNPO4.6468.7559.783.7560.8213.5
1B LoRATNPO3.2877.7159.2025.0053.4239.0
EE-TNPO3.2677.5760.893.7573.4118.0
3B FTTNPO4.4275.0265.1523.7556.8328.2
EE-TNPO4.3076.2365.6911.2570.7616.7
3B LoRATNPO3.4389.5866.1830.0059.2126.0
EE-TNPO3.5090.0466.4920.0070.1123.7

(3) The trigger-free failure reproduces on Llama-3.2. TNPO brings ES-F to 3.3–4.4 (originally 70.6 / 89.0) yet still names the forget author in 17.5–30% of name-seeking questions.

(4) EE-TNPO reduces trigger-free leakage in all four settings, at no cost in retention. It lowers NRR-F and NS ES-Sim and has the larger NRR-Gap in every setting. Standard forgetting is unchanged (ES-F within 0.3 points of TNPO), MU is higher in all four settings, and ES-R is higher under FT and within 0.5 points under LoRA.

(5) Gemma was not run. It requires TOFU fine-tuning, the unlearning runs and all analyses from scratch, which did not fit the rebuttal period. The three added backbones are 2024 models.

We will add Tables R1 and R2 to the paper and state for which backbones and update types each signature holds.

AuthorsOpenReview ↗

Response to Reviewer iNLQ (3/7)

W2 & Q2 — Token-level baselines: search grids and tuning budgets

The submission compared EE-TNPO with NPO, WGA and TNPO at one configuration each; SatImp was discussed but not run, and, as the reviewer notes, the methods were not compared at matched retention or under comparable budgets. We added SatImp, Selective Unlearning and ATWU, tuned every method under the same rule, and compared them on trigger-free queries at matched retention. This comment gives the search grids, the selection rule and the budgets; the next one gives the results.

Setup. TOFU Forget10, Llama-2-7B-Chat. All methods except ATWU use LoRA with the same learning rate (5e-4), 10 epochs and batch size 32. Every run is evaluated at epochs 4, 6, 8 and 10, with 2 seeds.

Methods and search grids. Every grid contains the method's published or official setting.

  • NPO, WGA, TNPO, EE-TNPO (the methods of the paper). NPO: β∈0.05,0.1,0.5β ∈ {0.05, 0.1, 0.5}. WGA: α∈3,7,10α ∈ {3, 7, 10}. TNPO and EE-TNPO: β∈2,5,8β ∈ {2, 5, 8}. The paper used β=0.1β = 0.1, α=7α = 7 and β=5β = 5. For EE-TNPO we search only ββ and keep the early-exit layer at the paper's value.
  • SatImp (Yang et al., 2025) weights each token loss by pβ1⋅(1−p)β2p^{β_1}·(1−p)^{β_2}, where p is the current probability of the token. We use the official code and six settings of (β1β_1, β2β_2, retain coefficient): (5, 1, 1), (3, 1, 1), (5, 0.5, 1), (5, 0.1, 1), (5, 1, 0.1) and (5, 0.1, 0.1). They cover the setting of its paper (β1=5β_1 = 5, β2=1β_2 = 1), the β2=0.1β_2 = 0.1 recommended in its README and the retain coefficient 0.1 of its code.
  • Selective Unlearning (SU) (Wan et al., 2025) unlearns only the tokens on which a model that knows the forget data and a model that does not disagree by more than γ. As these two assistant models we use the original model and the TOFU Retain90 model, which was never trained on the forget authors. Such an oracle does not exist in practice, so this choice favours SU. With γ ∈ {0.5, 0.8, 0.9}, its original gradient-ascent form never reached the retention level required below (best ES-R 88.3). We therefore report SU-TNPO: SU's token mask inside the EE-TNPO loss, which differs from EE-TNPO only in how the tokens are selected, with γ∈0.5,0.8,0.9×β∈2,5,8γ ∈ {0.5, 0.8, 0.9} × β ∈ {2, 5, 8}.
  • ATWU (Yüce et al., 2026) learns a token-importance scorer during unlearning. Its official code supports FT only, so it runs under its authors' protocol, and we search its budget parameter ρ∈0.1,0.2,0.3ρ ∈ {0.1, 0.2, 0.3}.

Selection rule (matched retention). The rule was fixed before the runs: for each method, among all (setting, epoch) checkpoints with extraction strength on retain QA (ES-R) ≥ 95, we take the one with the lowest NRR-F (share of the 20 name-seeking forget questions whose generated answer contains the author's name), averaged over the 2 seeds. The selected settings of EE-TNPO, TNPO, SatImp, SU-TNPO and ATWU were then retrained with 2 further seeds that played no part in the selection.

Table R3. Search budget and selected setting per method. Checkpoints compared = settings × 4 evaluated epochs. GPU-h = training time of all LoRA runs of the method (ATWU runs under FT and is not comparable).

EE-TNPOTNPONPOWGAATWUSatImpSU-TNPO
Settings searched3333369
Checkpoints compared12121212122436
Selected settingββ 2, ep 10ββ 2, ep 8ββ 0.1, ep 10αα 3, ep 10ρρ 0.3, ep 10(5, 0.1, 0.1), ep 6γγ 0.5, β 2, ep 6
GPU-h0.780.750.720.47–1.092.08

(1) No baseline had a smaller search than EE-TNPO. EE-TNPO, TNPO, NPO, WGA and ATWU have 3 settings each. SatImp and SU-TNPO have 6 and 9, because we added settings to both for fairness: with their first three settings neither was competitive (numbers in the next comment).

(2) EE-TNPO uses no explicit knowledge about the dataset. Its weights are the disagreement between the model's own early-exit and final predictions, computed in the same forward pass. It needs no second model, no oracle and no entity list, whereas SU needs the Retain90 oracle (whose training cost is not counted above). The extra cost is one projection of a hidden state onto the vocabulary: +3.7% training time over TNPO (LoRA, median of 8 runs each), with no additional forward or backward pass.

Reviewer_ghc6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why some LLM unlearning methods achieve strong forgetting while preserving unrelated knowledge, and whether that apparent success survives changes in how questions are phrased.

The authors find that TNPO, WGA, and NPO-LoRA can rely on target names to trigger forgetting. When a question describes the target without mentioning its name, the model may still reveal the information. To address this, they propose EE-TNPO, which compares predictions from intermediate and final layers to give key factual tokens more weight during unlearning.

Experiments show improved forgetting–retention trade-offs across several benchmarks. Although some leakage and unintended knowledge loss remain, EE-TNPO offers a meaningful approach to more reliable LLM unlearning.

Critical strengths
  1. The paper identifies trigger-free leakage and supports it with trigger injection and activation patching, exposing a weakness in existing unlearning evaluations.

  2. EE-TNPO substantially reduces name leakage while maintaining comparable aggregate utility, giving the mechanistic findings practical value.

Critical weaknesses
  1. Shared entities can cause substantial retention loss, as shown on MUSE-Books. Aggregate utility scores do not rule out damage to other named-entity QA tasks.

  2. Activation patching supports the proposed pathway, but does not establish whether rank collapse causes forgetting or whether the mechanism generalizes broadly.

Questions for authors
  1. Retention when entities overlap.
    The retention drop on MUSE-Books suggests that suppressing shared entities may interfere with knowledge that should be preserved. Could you provide a controlled evaluation of this interference when forget and retain facts share entities or name tokens, including other named-entity tasks such as scientific QA?

  2. Causal role of rank collapse.
    Activation patching supports the involvement of deep-layer attention and MLP outputs, but does not isolate the contribution of effective-rank collapse. Could targeted interventions clarify whether rank collapse itself causes forgetting under both FT and LoRA?

  3. Forgetting beyond names.
    Appendix D.3 evaluates description-based queries, but a finer-grained analysis would clarify whether the improvements extend to different factual attributes. Could you quantify leakage of non-name facts, such as birth dates or affiliations, under trigger-free queries that do not require generating the target’s name?

7 comments
AuthorsOpenReview ↗

Response to Reviewer ghc6 (1/5)

We greatly appreciate the constructive feedback. We address each concern below. W/Q numbers follow the order of the review, and the revisions we mention will be updated in the Camera-ready PDF.

W1 & Q1 — Retention when forget and retain facts share entities

The submission showed this interference only on MUSE-Books (App. G.3), an end-to-end result that does not isolate entity sharing, and it reported only aggregate utility on TOFU. We therefore additionally ran (1) a controlled evaluation in which only entity sharing differs, (2) the same evaluation for shared name tokens, and (3) evaluations on scientific and named-entity QA outside TOFU (next comment).

(1) Controlled evaluation for shared entities.

Setup. Llama-2-7B-Chat, TOFU Forget10 (20 forget authors; the other 180 authors are retain data), 4 seeds. We first compute the EE weights of the original model on all forget answers and collect the capitalized entity words among the top 20% of weights, excluding numbers and all forget-author names. This gives 241 word types such as "Award", "Prize", "German" and "Seoul"; these words carry factual content about the authors (awards, nationalities, places). We then build two sets of retain QA: a shared set of 500 QA whose gold answer contains one of these words, and a matched set of 500 QA that contain none. The matched set has the same distribution of answer lengths, since longer answers are more likely to contain these words and all three metrics depend on answer length. Both sets were fixed before any unlearned model was evaluated. For each set we measure the drop from the original to the unlearned model and report DiD = drop(shared) − drop(matched). If unlearning damaged all retain facts alike, DiD would be 0; a positive DiD is the damage that comes from sharing entity words with the forget set. Uniform-FT is the EE-TNPO loss with all token weights equal; it keeps EE-TNPO's per-answer normalization but removes the EE weights.

Table R1. Extra drop on shared-entity retain facts (DiD, points; mean ± sd over 4 seeds; in brackets, the number of seeds whose 95% bootstrap CI over authors is above 0). All three metrics are computed on the retain QA of the two sets. ES-R = extraction strength (the fraction of the gold answer the model completes on its own; teacher-forced, as in the paper). Prob-R = probability of the gold answer, normalized by answer length (%; not a log-probability). ROUGE-L-R = ROUGE-L recall of the generated answer.

ModelES-RProb-RROUGE-L-R
EE-TNPO-FT+5.00 ± 1.00 (3/4)+1.06 ± 0.48 (3/4)+1.68 ± 0.69 (0/4)
Uniform-FT+3.76 ± 1.78 (1/4)+0.53 ± 0.42 (1/4)+0.81 ± 1.20 (0/4)
TNPO-FT+2.03 ± 1.14 (0/4)+0.14 ± 0.30 (0/4)−0.24 ± 1.09 (0/4)
EE-TNPO-LoRA+1.14 ± 0.72 (1/4)+0.30 ± 0.47 (1/4)+0.67 ± 0.44 (0/4)
TNPO-LoRA+1.20 ± 1.00 (2/4)+0.19 ± 0.29 (1/4)+1.28 ± 1.09 (1/4)

(a) Under FT, EE-TNPO pays a measurable but limited extra cost on shared-entity facts. EE-TNPO-FT loses 5.0 ES-R points more on shared than on matched facts, against 2.0 for TNPO-FT, which is the interference the reviewer describes and has the same direction as MUSE-Books. This cost is only partly due to the EE weights: uniform weights with the same normalization already give +3.76, so about 58% of the gap to TNPO comes from the per-answer normalization. The cost is also smaller in what the model actually generates: on generated answers the extra drop is +1.68 ROUGE-L-R points, and its 95% CI includes zero in all 4 seeds.

(b) Under LoRA, EE-TNPO has no extra cost relative to TNPO. The extra drop on shared-entity facts is small for both methods and at the same level: +1.14 vs. +1.20 in ES-R, +0.30 vs. +0.19 in Prob-R and +0.67 vs. +1.28 in ROUGE-L-R, and for neither method is the CI above zero in more than 2 of 4 seeds. The additional cost of EE-TNPO is therefore specific to full fine-tuning.

(2) Retain answers containing forget-name tokens. We repeated the comparison for retain answers that contain a word which is also part of a forget author's name (e.g. "Patrick" or "Ji"; 327 QA vs. 327 matched). All FT methods show a similar extra drop in ES-R: TNPO +4.32, uniform +3.79, EE-TNPO +4.36. Therefore, sharing name tokens with forget authors carries no cost that is specific to EE-TNPO.

AuthorsOpenReview ↗

Response to Reviewer ghc6 (2/5)

W1 & Q1 (continued)

(3) Other named-entity tasks and scientific QA. The submission reported Model Utility, a harmonic mean of nine scores on TOFU's own subsets, and no QA task outside the benchmark. As the reviewer noted, an aggregate can hide damage, so we evaluated the unlearned models on two external benchmarks and looked at the entity-related parts of the utility score separately.

Setup. Same models as above (TOFU Forget10, 4 seeds), evaluated with the lm-eval harness. SciQ (scientific QA): 1,000 multiple-choice science questions, asked closed-book (without the support passage) and 0-shot. TriviaQA (answers are mostly named entities): 2,000 questions, 5-shot.

Table R2. QA outside TOFU (%; mean ± sd over 4 seeds). acc / acc_norm = raw / length-normalized multiple-choice accuracy. EM = exact match. Alias = the answer contains a gold alias.

ModelSciQ accSciQ acc_normTriviaQA EMTriviaQA alias
Original76.7068.3048.4554.20
TNPO-FT77.90 ± 0.5470.33 ± 1.7048.89 ± 1.1754.64 ± 0.63
EE-TNPO-FT77.90 ± 0.2670.67 ± 0.9648.95 ± 0.6254.40 ± 0.76
TNPO-LoRA76.68 ± 0.3368.97 ± 0.7948.16 ± 0.6454.25 ± 0.49
EE-TNPO-LoRA77.15 ± 0.7469.92 ± 0.7147.90 ± 0.9853.76 ± 0.54

Scientific and named-entity QA are preserved. No model in the table is more than 0.6 points below the original model.

The entity-related parts of Model Utility agree. On Real Authors and World Facts, the two TOFU subsets about real-world entities that are not part of the retain loss, EE-TNPO-FT stays at TNPO-FT's level: ROUGE-L / probability / truth ratio are 91.06 / 48.80 / 63.59 vs. 92.04 / 47.91 / 62.33 on Real Authors and 88.53 / 46.80 / 59.67 vs. 88.51 / 46.36 / 58.13 on World Facts (original model: 91.68 / 46.85 / 61.20 and 90.46 / 42.39 / 55.21).

We will add both evaluations to the paper alongside the MUSE-Books result, and specify in the limitations that the cost under shared entities arises mainly under FT.

AuthorsOpenReview ↗

Response to Reviewer ghc6 (3/5)

W2 & Q2 — Causal role of rank collapse

In the submission, Fig. 1b shows that forget rank collapses in the models that forget selectively, which is a correlation, and App. D.2 patched whole module outputs and measured only extraction strength, not rank. Neither tests the collapse itself. We therefore ran two interventions on the unlearned models, under both FT and LoRA, and measured rank and knowledge on the same inputs.

Setup. Llama-2-7B-Chat, TOFU Forget10, 4 seeds. (A) Name-attention knock-out (Geva et al., 2023, [1]): at layers 29–30 we add −10910^9 to the attention logits of the author-name tokens of the question, in all heads, so that no token can attend to the name; everything else stays unlearned. The control applies the same edit to equally many adjacent non-name tokens. (B) Residual interpolation: at the output of layer 29 we replace the hidden state h, at all positions, by (1−α)⋅hunlearned+α⋅horiginal(1 − α)·h_{unlearned} + α·h_{original}.

Table R3a. Name-attention knock-out (mean over 4 seeds). ES-F = extraction strength on the 379 forget questions that contain the name (%; the fraction of the gold answer the model completes on its own, so higher = more knowledge recovered). Gold log-p = mean log-probability of the gold answer tokens (nats). Forget erank = effective rank of layer-30 hidden states on the forget set, as in Fig. 1b.

ModelES-FGold log-pForget erank
Original98.1−0.011443
TNPO-FT3.0−7.09175
+ name knock-out8.1−2.78743
+ control3.0−7.09174
TNPO-LoRA3.1−6.58122
+ name knock-out17.0−3.26434
+ control3.1−6.60121
NPO-FT6.6−2.32646
+ name knock-out7.1−1.94892

Table R3b. Residual interpolation at layer 29 (mean over 4 seeds; metrics as in Table R3a). α=0α = 0 is the unlearned model; α=1α = 1 replaces the layer-29 hidden state by that of the original model.

ModelMetricα = 00.250.50.751
TNPO-FTForget erank1752033429171443
ES-F3.03.68.846.590.4
Gold log-p−7.09−6.37−4.17−0.56−0.03
TNPO-LoRAForget erank1221472385301436
ES-F3.13.77.934.797.2
Gold log-p−6.58−5.97−4.91−2.38−0.01
  • (1) Removing the trigger restores rank and knowledge together, under both FT and LoRA. The name knock-out recovers 45% (FT) and 24% (LoRA) of the lost forget rank and 61% and 51% of the lost gold log-p, while the control changes neither (Table R3a).

  • (2) When the collapsed representation is restored gradually, knowledge returns in step with rank. In Table R3b all three quantities rise monotonically with α under both FT and LoRA. At α = 0.5 and 0.75, TNPO-LoRA has recovered 9% and 31% of its lost rank and 5% and 33% of its lost ES-F; TNPO-FT has recovered 13% and 59% of its rank and 6% and 46% of its ES-F. Gold log-p recovers somewhat earlier (25% and 64% under LoRA, 41% and 92% under FT).

(3) The two interventions give a consistent relation between rank and knowledge. They act at different points (attention at layers 29–30 vs. the residual state at layer 29), yet in both rank and gold log-p the knock-out falls between the interpolation steps α=0.5α = 0.5 and 0.750.75: under LoRA it reaches erank 434 and log-p −3.26 (238 to 530; −4.91 to −2.38), under FT erank 743 and log-p −2.78 (342 to 917; −4.17 to −0.56).

In answer to Q2: under both FT and LoRA, interventions that undo the rank collapse undo the forgetting with it, step by step. In the token-triggered models the collapse and the forgetting therefore do not merely co-occur, as in Fig. 1b: the collapse accompanies the suppression under every intervention we ran; whether it mediates the suppression or is a by-product of the same attention pathway is not separated by these experiments. This coupling is specific to these models: in NPO-FT, which does not rely on the name (next comment), the knock-out raises rank (646 → 892) with little change in gold log-p (−2.32 → −1.94). As no intervention changes the rank alone in the token-triggered models, we state the collapse as a tightly coupled part of the pathway and not as a separate cause.

We will describe rank collapse throughout the paper as tightly coupled to the name-attention pathway, not as a separate cause of forgetting, and add both interventions to the paper.

[1] Dissecting Recall of Factual Associations in Auto-Regressive Language Models, Geva et al., 2023

Reviewer_uhsaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies why several gradient-based LLM unlearning objectives achieve favorable forgetting-retention trade-offs while remaining vulnerable to queries that omit the target entity's name. Comparing NPO, WGA, and TNPO under full fine-tuning and LoRA, it associates selective forgetting with attention concentration on target-name tokens and a reduction in the effective rank of forget-set representations. Controlled name injection and module-level activation patching provide functional evidence for a trigger-dependent suppression pathway. The paper then proposes EE-TNPO, which reweights token-level NPO losses using the Jensen-Shannon divergence between intermediate-layer and final-layer predictive distributions, emphasizing answer tokens associated with factual information. Experiments report reduced trigger-free leakage on TOFU and RWKU, with additional evaluations on Qwen2, WMDP, and MUSE. The improvements are substantial in several entity-centric settings, while MUSE-Books exposes a retention cost when forget and retain knowledge share entities.

Critical strengths
  1. The paper identifies a concrete failure mode that aggregate forgetting scores can obscure. The NC/NS analysis in Tables 3 and 15 shows that low extraction strength on name-cued queries can coexist with substantial leakage on name-seeking queries. The 800 context-matched name-injection probes provide useful evidence of trigger sensitivity, and the description-based evaluation in Appendix D.3 strengthens the connection to queries that omit the target name. This is a consequential finding because successful suppression under one prompt format is insufficient evidence of robust forgetting.

  2. EE-TNPO provides a simple, well-motivated remedy for the identified failure mode. Experiments on TOFU and RWKU show improved forgetting on trigger-free queries while generally preserving utility. The combination of standard evaluation metrics, targeted trigger-free probes, and manual name-leakage analysis strengthens the empirical evidence (Tables 5-6 and Appendix F). Evaluations across model families and update strategies further support the method's usefulness, while the reported retention loss on MUSE-Books helps establish its limitations.

Critical weaknesses
  1. The activation-patching results in Appendix D.2 provide causal evidence that deep attention and MLP outputs influence target-answer suppression, but do not isolate the proposed pathway from target-token attention concentration to representation collapse and suppression. Replacing an entire module output allows its effect to propagate through multiple downstream routes. Path patching or other targeted interventions with control paths would help identify which connections mediate suppression. Measuring effective rank alongside behavioral changes would further assess the proposed role of representation collapse. The current evidence supports component involvement more strongly than the specific causal chain.

  2. The specific contribution of EE weighting remains insufficiently isolated. Comparisons with normalized uniform weights, shuffled EE weights, and explicit name-span weighting on TOFU can be supportive evidence. These controls would clarify whether the gains depend on the early-exit signal's ability to identify informative tokens or can be achieved with simpler weighting schemes.

Questions for authors
  1. Clarification of the implemented objective: Are EE scores computed from the frozen pre-unlearning model or recomputed from the current model, and are they detached from the gradient? Eq. (2) normalizes EE weights within each answer, whereas Eq. (10) is written as a token sum. How are the loss scales aligned in the implementation? Please also specify the intermediate-state normalization and retain-loss coefficients for TOFU and RWKU.

  2. Could the authors further discuss the extent to which the improvements can be attributed specifically to EE weighting? If available, comparisons with simple controls—such as normalized uniform weights, shuffled EE weights, or explicit name-span weighting on TOFU—under comparable training conditions would help clarify this point. A comparison with SatImp could also provide useful context.

  3. Does intervening specifically on attention to target-name tokens change both effective rank and forgetting, relative to a matched non-target intervention? If such evidence is unavailable, can the paper distinguish the demonstrated contribution of deep attention/MLP modules from the hypothesized causal chain, and clarify whether EE-TNPO changes that chain or supplements it?

3 comments
AuthorsOpenReview ↗

Response to Reviewer uhsa (1/3)

We greatly appreciate the careful review and the concrete suggestions for controls. We address each concern below. W/Q numbers follow the order of the review, and the revisions we mention will be updated in the Camera-ready PDF.

W1 & Q3 — Targeted intervention on attention to target-name tokens

As the reviewer points out, the activation patching of App. D.2 replaces an entire module output, has no control path, and does not measure effective rank. We therefore ran the intervention the reviewer described in Q3: we block only the attention to the target-name tokens, compare it with a matched non-target intervention, and measure effective rank and forgetting on the same inputs.

Setup. We take the unlearned models (Llama-2-7B-Chat, TOFU Forget10, 4 seeds) and apply attention knockout (Geva et al., 2023, [1]), which blocks the information flow along chosen attention edges by masking their pre-softmax scores. At layers 29–30, where App. D.2 located the effect, we add −10910^9 to the attention logits of the positions that hold the author's name in the question, in all heads. No token can then attend to the name at these two layers, while all weights, value vectors, MLPs and other layers remain those of the unlearned model. The matched control applies the same edit to equally many non-name question tokens next to the name.

Table R1. Change caused by the name knock-out relative to the unedited unlearned model (matched control in brackets); mean ± sd over 4 seeds. ES-F = extraction strength on the 379 forget questions that contain the name (%; the fraction of the gold answer the model completes on its own, so higher = more knowledge recovered). Gold log-p = mean log-probability of the gold answer tokens (nats). Injection = ROUGE-L recall on unrelated general-knowledge questions with a forget-author name inserted, i.e. the probe of Table 1 of the paper (low when the name triggers suppression). Forget erank = effective rank of layer-30 hidden states on the forget set, as in Fig. 1b. Bold = the name knock-out exceeds the control in all 4 seeds.

ModelΔES-FΔ gold log-pΔ injectionΔ forget erank
TNPO-FT+5.05 ± 0.99 (+0.01)+4.31 ± 0.09 (+0.00)+32.9 ± 3.4 (−0.5)+568 ± 124 (−1)
TNPO-LoRA+13.91 ± 2.18 (−0.03)+3.32 ± 0.72 (−0.02)+26.5 ± 3.0 (−0.2)+312 ± 204 (−1)
EE-TNPO-FT+2.04 ± 0.68 (+0.01)+3.16 ± 0.28 (−0.00)+23.7 ± 6.6 (−0.4)+584 ± 46 (−1)
EE-TNPO-LoRA+1.36 ± 1.58 (−0.00)+0.91 ± 0.69 (−0.00)+4.9 ± 3.7 (+0.1)+36 ± 27 (−1)

(1) Intervening on name-token attention changes both effective rank and forgetting; the matched control changes neither. In TNPO-FT, blocking the name recovers 61% of the lost gold log-probability (−7.09 → −2.78; original model −0.01), 92% of the injection score (11.0 → 43.9; original 46.9) and 45% of the forget rank (175 → 743; original 1443). In TNPO-LoRA the recovery is 51% / 60% / 24%. ES-F rises from 3.0 to 8.1 (FT) and from 3.1 to 17.0 (LoRA). These results show that this pathway carries a substantial part of both the suppression and the forget-rank collapse.

(2) EE-TNPO supplements the chain rather than replacing it. Under FT, EE-TNPO still uses the name pathway (rank +584 vs. +568 for TNPO) but relies on it less for suppression (ES-F +2.04 vs. +5.05, log p +3.16 vs. +4.31 nats); under LoRA its reliance is far weaker (ES-F +1.36 vs. +13.91, log p +0.91 vs. +3.32 nats). With the name blocked, EE-TNPO still suppresses more than TNPO: after the knock-out, ES-F is 4.9 vs. 8.1 (FT) and 4.0 vs. 17.0 (LoRA). EE-TNPO therefore adds suppression that does not depend on the name in the prompt.

We will add this intervention to the paper as the causal evidence for the name-attention pathway and state the mechanism at the level this evidence supports.

[1] Dissecting Recall of Factual Associations in Auto-Regressive Language Models, Geva et al., 2023

AuthorsOpenReview ↗

Response to Reviewer uhsa (2/3)

W2 & Q2 — Contribution of the EE weighting

In Table 5 of the paper, EE-TNPO and its closest baseline TNPO differ in two ways at once: the token weights and the normalization. EE-TNPO normalizes the weights within each answer and then averages over answers (Eq. 2), whereas TNPO averages its token losses over all tokens of the batch. That table therefore cannot tell how much of the gain comes from the EE weights. We ran the three controls the reviewer suggested, changing only the token weights.

Setup. Full FT on TOFU Forget10 (Llama-2-7B-Chat), 4 seeds per arm. All arms use the EE-TNPO loss with its per-answer normalization and the same optimizer, learning rate, epochs and seeds. They differ only in the weight of each answer token before normalization:

  • Uniform: 1 on every token, which isolates the effect of normalization.
  • Shuffled EE: the EE weights of each answer, permuted across its token positions (same values, wrong tokens).
  • Name-span: 0.9 on the tokens of the forget author's name (gold list of full, first and last names), 0.1 elsewhere.
  • EE: the proposed weights.

TNPO (token mean over the batch, no per-answer normalization) is shown for reference. The TNPO and EE rows use the paper's settings (Table 5) and are evaluated with the same code as the controls, so their values differ slightly from Table 5.

Table R2. Uniform vs. EE (FT; mean over 4 seeds; %). NRR-F = share of the 20 name-seeking (trigger-free) forget questions whose generated answer contains the author's name. NS ES / NS ES-Sim = extraction strength on these questions, exact / with the 0.8-similarity criterion of App. C.3. Desc ES = extraction strength on forget questions whose name is replaced by a description (App. D.3). NRR-Gap = name recall on retain name-seeking questions minus NRR-F. ES-F / ES-R = extraction strength on forget / retain QA. MU = model utility. EE share = fraction of the TNPO → EE change that comes from the uniform → EE step.

WeightingNRR-F ↓NS ES ↓NS ES-Sim ↓Desc ES ↓NRR-Gap ↑ES-F ↓ES-R ↑MU ↑
TNPO30.007.2738.374.7046.983.2575.6662.01
Uniform20.004.6030.774.3953.743.0376.4362.37
EE10.004.0116.253.8465.182.9277.0762.84
EE share50%18%66%64%63%–––

Table R3. Shuffled and name-span controls (same setting and metrics; bold = column best).

WeightingNRR-F ↓NS ES-Sim ↓Desc ES ↓NRR-Gap ↑ES-R ↑MU ↑
Shuffled EE32.5032.245.3743.9474.0661.38
Name-span0.0014.236.4871.5875.1660.44
EE10.0016.253.8465.1877.0762.84

(1) Normalization and the EE weights both contribute. The step from TNPO to uniform is the effect of normalization, and the step from uniform to EE is the effect of the EE weights. The EE step accounts for 50–66% of the total gain on four of the five trigger-free metrics (see EE share of Table R2). EE improves on uniform weights on all five, with no loss in ES-F, ES-R or MU.

(2) Shuffled EE eliminates the gain. When the same weight values are put on the wrong tokens (see Shuffled EE of Table R3), the gain over uniform weights disappears: NRR-F is 32.5 and NS ES-Sim 32.2, against 20.0 and 30.8 for uniform weights and 10.0 and 16.3 for EE. The improvement therefore comes from the tokens that the EE signal selects.

(3) Explicit name-span weighting trades name leakage for other leakage. Oracle name weights remove name leakage (NRR-F 0.0), but they give the highest leakage on description queries and the lowest utility (Desc ES 6.48 vs. 3.84; MU 60.44 vs. 62.84). EE keeps both kinds of trigger-free leakage low while using no explicit knowledge about the dataset.

(4) Against SatImp, EE-TNPO has lower name and description leakage at matched retention. SatImp weights each token loss by pβ1⋅(1−p)β2p^{β_1}·(1−p)^{β_2}, where p is the current probability of the token. Under LoRA, we take each method at its checkpoint with the lowest NRR-F among those with ES-R ≥ 95 and evaluate it on 4 seeds. Compared to SatImp, EE-TNPO has lower Desc ES (3.15 vs. 4.48) and name leakage (NRR-F 7.5 vs. 16.3) at similar ES-R (98.1 vs. 97.4) and MU (62.7 vs. 63.5). The protocol and the full baseline table are in our response to Reviewer iNLQ (W2 & Q2).

We will add the uniform, shuffled and name-span controls to the paper as an ablation of EE-TNPO, describe EE-TNPO as per-answer normalization combined with an EE weight, and add SatImp as a baseline.

AuthorsOpenReview ↗

Response to Reviewer uhsa (3/3)

Q1 — Implementation of the objective

The submission did not state these details. Below, we clarify the implementation and training settings used in the runs reported in the paper.

(1) EE scores are recomputed from the current model. They are computed at every step, in the same forward pass as the loss (under LoRA with the adapters active). The frozen pre-unlearning model only supplies pθ0p_{θ_0} in the NPO term.

(2) They are detached from the gradient. Everything up to the JSD runs under torch.no_grad() with a detached LM head, so gradients flow only through log pθ(yi)log \ p_{\theta}(y_i).

(3) Loss scales of Eq. (2) and Eq. (10) are not aligned by a constant; the two losses are normalized differently. EE-TNPO averages within each answer and then over answers. Our TNPO takes one mean over all answer tokens of the micro-batch (a token mean, not the token sum written in Eq. (10)), with ϵ=10−5\epsilon=10^{-5} in the weight of Eq. (11). With uniform weights, the two gradients are identical when all answers have the same length and differ otherwise. On Forget10, answer lengths range from 9 to 96 tokens, so each answer is effectively reweighted by a factor of 0.46–4.94 relative to TNPO. The TNPO vs. EE-TNPO comparison in Table 5 of the paper therefore also contains a normalization difference, which the uniform-weight control in W2 & Q2 separates.

(4) Intermediate-state normalization. The hidden state of layer LeL_e is passed through the LM head. In the TOFU and RWKU runs of the paper (Llama-2 and Llama-3-8B-Instruct; LeL_e = 31 of 32 in both) it is the raw residual, without the final RMSNorm or a temperature. On TOFU, the raw projection separates author-name tokens from other answer tokens better than the normed one (token-level AUROC on the 20 name-seeking forget answers: 0.86 vs. 0.78). On Qwen2 and Llama-3.2 the raw projection saturates: the divergence is close to its maximum (ln 2) at almost every token, so the normalized weights are nearly uniform. Our Qwen2 runs and new Llama-3.2 runs therefore apply the final norm before the LM head (results in our response to Reviewer iNLQ, W1 & Q1). The weight is JSD / ln 2, normalized per answer as in Eq. (2).

(5) Retain-loss coefficients. We use token-mean NLL on the retain data with λ\lambda = 1 and forget coefficient 1 on both TOFU and RWKU. On TOFU this holds in every run configuration of the paper. On RWKU it holds for EE-TNPO and the three baselines (NPO, WGA, TNPO); the retain texts are those of one fixed, unrelated retain celebrity per target (App. C.2).

We will state all five points in the paper: Eq. (10)/(11) will be written as a token mean with ϵ\epsilon, and the projection in (4) will be given per backbone. Two details of C.1 will also be corrected: the LoRA dropout is 0.1 (not 0.05), and the TOFU FT runs used no gradient clipping (the same for all methods).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised manuscript uploaded: summary of changes

We thank the reviewers for the discussion. In the uploaded revision, all added or changed content is marked in blue, including corrected values; in newly added tables only the caption is colored, and in Table 8 the changed entries. Updated references, minor wording fixes, and renumbered cross-references are not colored. Deletions can be seen in OpenReview's comparison with the submitted version. Section and table numbers refer to the revised PDF; appendix table numbers have shifted because of the added tables.

1. Changes that reflect our responses

ContentRaised byLocation in the revision
Targeted attention knock-out on the author-name tokens, with a matched controluhsa (W1, Q3), ghc6 (W2, Q2), iNLQ (W3, Q3)Abstract; Sec. 3.3; App. D.4, Tables 13–14
Residual interpolationghc6 (W2, Q2), iNLQ (W3, Q3)App. D.2, Table 11
Revised description of rank collapse: tightly coupled to the name-attention pathway, not a separate cause of forgettinguhsa (W1), ghc6 (W2, Q2), iNLQ (follow-up)Introduction; Sec. 3.3; App. D.4
Mechanism analysis on Llama-3.2-1B, Llama-3.2-3B and Qwen2-7Bghc6 (W2), iNLQ (W1, Q1)Sec. 3.3; App. E.2.1, Table 19
TNPO and EE-TNPO on Llama-3.2-1B and 3BiNLQ (W1, Q1)App. E.2.2, Tables 21–22
Weighting controls under FT (uniform, shuffled EE, oracle name-span), separating the normalization from the EE-weightsuhsa (W2, Q2), iNLQ (W4, Q4)Sec. 6.1; App. E.6, Table 27
EE-TNPO described as per-answer normalization combined with EE-weights on knowledge-bearing answer tokensuhsa (W2), iNLQ (W4, Q4)Sec. 5; App. B.2
Implementation of the objective: EE-weights recomputed at every step and detached, early-exit projection used for each model, loss coefficientsuhsa (Q1)Sec. 5; App. B.2; App. C.1
Retention when retain facts share entities with forget facts, and QA outside TOFU (SciQ, TriviaQA)ghc6 (W1, Q1), iNLQ (W4, Q4)App. E.7, Tables 28–29; App. G.3; Limitations
Leakage of non-name facts by attribute type, the birth-fact questions, and the RWKU probesghc6 (Q3)Sec. 6.1; App. D.5, Tables 15–16
Training-time overhead under LoRA from the controlled measurementiNLQ (W2, Q2)App. C.5, Table 8

Three items differ from what our responses announced. (i) The shared-entity and external-QA evaluations are in App. E.7 with the other TOFU results rather than next to the MUSE-Books result; App. G.3 refers to them. (ii) Eq. (10) and (11) keep their original form, and the token mean and ε are stated in the text below them (App. B.2). (iii) In Table 8, only the LoRA column is replaced by the controlled measurement; the FT column keeps the original measurement, because FT training time depends on factors beyond the method (App. C.5).

2. Corrections

  • The controlled probe has 720 instances (18 base sentences × 40 names), not 800 (Sec. 3.2, App. H.2). Only the stated count was wrong; the reported values are unchanged.
  • App. C.1: the LoRA dropout is 0.1 (not 0.05), and the TOFU FT runs use no gradient clipping (the same for all methods).

3. Other changes

  • Ethics Statement. We added two paragraphs: Data, Models, and Licenses and Dual Use and Limits of Unlearning.
  • References. Entries were updated to their published versions, and references for the added analyses were added.

4. Not yet included

Two sets of results from our responses are not in this revision.

  • Comparison with additional token-level baselines at matched retention: SatImp and Selective Unlearning under LoRA, and ATWU under FT, with the search grids, tuning budgets, selection rule and trade-off curves (Tables R3 and R4 of our response to Reviewer iNLQ; point (4) of our response to Reviewer uhsa on W2 & Q2).
  • Weighting control under LoRA: the tuned uniform-weight baseline (Table R7 of our follow-up to Reviewer iNLQ). The weighting controls under FT are included (App. E.6).

Reason. To finish within the discussion period, we trained the new Llama-2-7B LoRA models of the rebuttal with a one-GPU launcher, whereas the LoRA rows of Table 5 were trained on two GPUs. As explained in our follow-up to Reviewer iNLQ, the two launchers give different absolute values for the same hyperparameters. The two sets above compare unlearning methods in the setting of Table 5, on the same metrics and with hyperparameters re-tuned under a retention constraint, so they would report a second set of values for the methods in Table 5. The conclusions of our responses rest on comparisons within the one-GPU setting and are unaffected. In the final version, we will report both sets together with the LoRA rows of Table 5 under one training setting, with per-seed values and the full configuration.

All other analyses of our responses are included as listed above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.