- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes attention-only unlearning for LLMs. They train an "Anchored Terms Identifier" (ATI) (a small MLP classifier trained on features of entity-sensitive attention states) in order to put more weights on subject/entity tokens in forget inputs. Subsequent unlearning defines loss over model internal states instead of final output, which is different from many existing unlearning algorithms.
Critical strengths
-
Validated interpretability-grounded design. The core design choice of targeting ATI-selected anchored tokens is useful for interpretability and validated with a downstream selection ablation where ATI beats frequency/random/whole-input targeting in 5 out of 6 model–dataset pairs (Table 4). This directly supports the knowledge-entanglement motivation.
-
Large efficiency and retention gains. Updating only K/V in five layers yields the best retention scores on SUITE across all models (Table 2) at a fraction of the compute (Sec. 3.5).
Critical weaknesses
-
The headline win depends on the authors' custom aggregate, and per-metric forgetting is weak. On SUITE, "Ours" is not best at forgetting (F=0.68 vs NPO 0.71 on Llama, Table 2); on TOFU it leads no individual metric (Table 3). If we change the weights of different individual metrics, the overall ranking might be different. More concerning, EM score on SUITE remains 2–5× higher than NPO (e.g., 0.346 vs 0.073 on Phi-4, Table 7), and INT4 quantization partially reverses forgetting on TOFU (Fig. 3). This makes it unsuitable for the authors' critique of existing works in related work: "suppressing the expression of knowledge does not necessarily eliminate its underlying influence." In addition, no relearning attack is run to verify whether information is actually eliminated or not.
-
The paper is structured and written in a somewhat confusing way for general readers, leaving some design choices unexplained.
- Eq (1), what's x? And how does it relate to the right hand?
- Line 198: what's k? Please define the ML task more clearly, otherwise it is difficult to understand these hyperparams and the numbers in in Table 1.
- How to read figure 2? what does a bar mean?
- Eq (3): why iterating over R_x? And please explain this loss: why is it valid by minimizing the attention score?
- Line 300: why exclude identified anchored spans?
Questions for authors
In addition to questions in the weakness:
- Why is EM on SUITE much higher than NPO (Table 7)? How does the method behave under benign relearning or stronger extraction attacks — does knowledge remain recoverable?
- For SUITE, are the steering reference tokens still TOFU author names?
- Line 431: why Qwen 2.5 on SUITE is the exception?
3 comments
Rebuttal part 2
PART 2:
- Questions:
-
"Why is EM on SUITE much higher than NPO (Table 7)? How does the method behave under benign relearning or stronger extraction attacks — does knowledge remain recoverable?"
Table 7 is now table 9, EM is a metric that we expect to underperform compared to other methods due to our loss objective. Methods like NPO that train directly on forget data labels will push the log-probability of label tokens down. This decrease in log-prob of label token is what EM measures.
On the other hand, our loss objective mainly concerns representations of identified subject tokens, which does not directly change log-prob. However, this is not necessary a downside, as we see that despite the higher EM, Forget ROUGE remains low, so no sensitive information is actually leaked.
We have included a relearning attack influenced by https://doi.org/10.48550/arXiv.2406.13356](https://doi.org/10.48550/arXiv.2406.13356). The attack relies on benign samples that are relevant to the unlearn data without the explicit knowledge expression.
For SUITE, we selected a subset of 100 samples each from Syntactic and Lexical in Retain Eval split. Syntactic includes samples that share the same question structure as forget data, while Lexical includes samples that share the same term (term is the unlearning subject in question) as forget data but different context.
For TOFU, it is more difficult to find a dataset with the same forget authors set but different context. So we opted for the Real Authors dataset included in TOFU.
The attack finetunes the selected dataset in 5 epochs with a fixed learning rate of 1e-05. We perform the attack on both our own method and baseline methods. Due to the large comparison, we have composed the results in a graph to be included in the Appendix A.7.
To summarize, our method generally maintain unlearning effectiveness across models and datasets as reported by the 3 unlearning metrics, the only exception is Qwen2.5 on SUITE dataset where a slight increase to ES (0.55 --> 0.65) and EM (0.2-->0.4) are observed while ROUGE metric maintain low after 5 epochs. We also want to note that Qwen2.5 SUITE has only 3 layers targeted by our unlearning engine. It is possible that the increase in EM and ES are due to this smaller number of modified layers, leading to other retrieval pathway in Attention. However, such increases in EM and ES do not directly yield the actual leakage of sensitive information itself, as seen by ROUGE metric.
-
"For SUITE, are the steering reference tokens still TOFU author names?"
Yes, steering for SUITE also uses the TOFU fictitious author names. The reason why we keep it this way is due to the fact that these are fictitious author names, they do not exist in real life and have never been seen by the models.
So theoretically, these fictitious names are treated as "unknown" combination of tokens with little to no significant semantic.
For example: a subject in SUITE might be "Space Shuttle", which is an object rather than someone's name, steering it towards "Siriporn Nakharin" (fictitious name) might carry over unwanted semantic. But since the name itself is never seen by models, from the model's perspective (theoretically), they could be anything.
-
Summary of changes to the document
- Table 2, 3 updated with new 3 seeds run and tuning results. descriptions in Table 2 and 3's captions are removed. Section 3.4.1 is modified to reflect the new results.
- Tables 9, 10, 11, 12 in Appendix A.6 are updated with the new seeded runs and tuning results.
- Table 4 updated with the new tuned result on first seed only. We also added comparison with anchored selected by spaCy NER.
- Figure 4 updated with total wall-clock and per-optimizer-step time. Reporting only relative speedup compared to mean baseline's speed. Relevant sections are also updated to reflect new results from Figure 4 (Section 3.5, Section 4, Section 1 - last paragraph).
- Update Equation 1 and Section 3.2 (first paragraph) to not explicitly determine a subject token selection threshold
- Update Appendix A.5 to include detailed description and analysis of the tuning process as well as documented configurations.
- Add Appendix A.7 Relearning attack that shows details about the relearning attack.
- Add Appendix A.8 Unlearning objective ablation. Here, we ablate different components of out unlearn loss objective (Equation 2)
Rebuttal
We thank the reviewer for their insightful reviews.
We would like to address your concerns and questions in 2 parts
PART 1:
x here in Equation 1 is the input. An input x is made up of many tokens t. Equation 1 essentially reads "For all non-padded token t in input x, select all t with sigmoid score pt greater than or equal to a threshold τ". Where sigmoid score pt is assigned to tokens by our ATI module.
k in line 198 is further described in Table 1's caption: "@k denotes the output cutoff". To evaluate ATI's performance, we select only top k scored tokens by ATI for evaluation.
Figure 2 shows the causal probing results of ATI versus random subject tokens (left) and versus random non-subject tokens (right). X axis shows Contrastive Impact, a metric defined above (line 208-210) that essentially show how much corrupting certain tokens weakens the model's preference for the correct factual answer over an incorrect counterfactual answer.
For example: Input - What is the capital of France ? Factual answer: Paris Counterfactual answer: London Under a clean run, assuming the model has the proper knowledge to answer the question, log-prob of Paris should be higher than London. Under a corrupted run that corrupts different tokens, we show that out ATI identifies tokens that results in larger decrease to factual answer (Paris). So causally, ATI's identified tokens are what yield the correct answer, while corrupting them does the opposite.
Not only that, since a term like "France" could be tokenized as 2 sub-word tokens "Fr" and "ance" (hypothetically), which of the 2 tokens are more important to the factual recall process ? Left figure shows that our ATI can also identify the more important sub-word tokens that are essential to factual recall.
Comparison in Figure 2 take the differences between CI produced by ATI's tokens and CI produced by random tokens. A positive number means ATI has better CI than random's.
Iteration over Rx here represents our objective of disrupting attention to subject tokens from non-subject tokens (which is Rx) instead of from subject tokens with each other.
For example: Input - "What is the capital of France and China ?", with "Fr" "ance" and "Chi" "na" being the subject tokens (again, assuming a sub-word tokenization).
When processing token like "capital" (in Rx), it might assign high attention to tokens in "France" and "China". This unlearning objective reduce this attention value.
We purposely exclude subject tokens' attention to themselves on purpose, under the intuition that doing so might change model's knowledge about the subjects themselves, instead of a fact (capital) regarding the subjects.
In line 300, we mention the exclusion of identified anchored spans in our Retain-projection preservation. This is due to the fact that forget data and retain data often contain the same subjects, but different facts. For example: SUITE dataset's forget data contains input querying "The 1986 Space Shuttle Challenger disaster", while retain data contains "The rest of Space Shuttle history". Same subject "Space Shuttle", but different contexts. If we include the identified anchored spans during the retention loss, we argue that it will unwind forget loss imposed on the identified anchored during forget loss.