- Rating
- 3(+1)
- Confidence
- 1(−1)
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies probability redistribution in suppression-based LLM unlearning, arguing that suppressing target tokens can increase the probability of alternative tokens that still reveal the knowledge to be forgotten. It proposes Safe-Anchored Relocation (SAR), which dynamically constructs reference-only and current-model-only token sets and optimizes a contrastive probability-mass objective to redirect probability toward reference anchors. The paper provides a local theoretical analysis and evaluates SAR on TOFU and MUSE, reporting improved forgetting with competitive utility compared with several unlearning baselines.
Critical strengths
Insightful problem analysis. The paper highlights how probability redistribution in suppression-based unlearning can lead to residual knowledge leakage and provides a systematic analysis of this behavior in existing methods.
Well-developed theoretical analysis. The paper derives the local logit-update behavior of SAR and establishes conditions under which probability mass shifts toward reference anchors, providing a clear mathematical foundation for the proposed objective.
Critical weaknesses
-
From the perspective of novelty: The method treats candidate tokens unique to the reference branch as safe anchors, but there is currently insufficient evidence showing that these candidates will not retain or leak the target knowledge. The existing theoretical analysis characterizes the direction of probability redistribution, but does not establish a connection between the reference anchors and semantic safety. Therefore, the core claim of "safe relocation" still needs further support.
-
From the perspective of writing: The introduction emphasizes that the same token can play different semantic roles in different contexts, but the subsequent introduction of SAR does not clearly explain how the reference-guided dynamic sets address this issue. Although the set construction depends on the current context, it still lacks an explanation of why the prediction difference between the reference and the current model can distinguish knowledge leakage from harmless use. It is suggested to clarify the correspondence between the motivation and the method design, and to add a comparative analysis of the same token in target and non-target contexts.
-
From the perspective of layout: Table 1 is placed near the method section, causing the experimental results to appear before the corresponding experimental setup and interrupting the reading of the method section.
Questions for authors
How do the authors establish that reference-only top-k tokens are safe anchors? Can the authors provide direct evidence of anchor safety and analyze SAR’s behavior when the reference branch retains the target knowledge? How does SAR’s context-dependent token-set construction address the concern raised in the introduction that the same token may be harmful in one context but harmless in another? Can the authors provide a targeted evaluation of this distinction?
6 comments
(Continuation of W1/Q1)
| Setting | Reference | Candidate set | Intervention segment leakage (%) | Leakage after intervention (%) |
|---|---|---|---|---|
| Single-token | Instruct | 4.00 | 96.00 | |
| 20.00 | 80.00 | |||
| 16.06 | 83.94 | |||
| NPO Ref | 1.75 | 98.25 | ||
| 24.56 | 75.44 | |||
| 17.74 | 82.26 | |||
| Self-prompt | 9.52 | 90.48 | ||
| 21.43 | 78.57 | |||
| 14.74 | 85.26 | |||
| Three-token | Instruct | 16.67 | 83.33 | |
| 48.89 | 51.11 | |||
| 51.43 | 48.57 | |||
| NPO Ref | 8.33 | 91.67 | ||
| 47.37 | 52.63 | |||
| 48.65 | 51.35 | |||
| Self-prompt | 22.22 | 77.78 | ||
| 51.85 | 48.15 | |||
| 47.87 | 52.13 |
These results support two observations. 1. Across different reference models and intervention positions, our anchor set generally reduces continuation leakage compared with the control sets. 2. Attribution analysis shows that most remaining leaks emerge during subsequent generation by the original, non-unlearned model. This is consistent with the model retaining the target knowledge and reintroducing it during continuation. Together, these results provide empirical support for the relative safety of our anchor set.
2. [W2/Q2] Same token, different token sets
We first wish to clarify that the token set selection in SAR is dynamic and empirical. We use the prediction disagreement between the reference model and the current model as the signal for selection. The intervention experiments above have already shown that the receiving sets under all three reference settings produce lower target leakage rates than the two control sets. At the same time, this set construction is dynamic and context-dependent. Let denote the current question and answer prefix, and denote the top- token sets of the reference model and the current model under this context, respectively, and denote the true next token. SAR constructs at each prediction position
\qquad S_{\mathrm{cur},t}(h)=(C(h)\setminus R(h))\cup\{y\}.$$ Therefore, SAR does not fix in advance which tokens are safe or harmful, but rather re-determines set membership based on the predictions of the two models in the specific context. In addition, by definition, $$S_{\mathrm{ref}}(h)\cap S_{\mathrm{cur},t}(h)=\varnothing.$$ This ensures that the same token will not simultaneously receive two opposite optimization requirements, acceptance and suppression, at the same prediction position. A practical example is given below. The common candidate token is `village`, and square brackets mark this token. | Set | Question | Prefix and candidate token | | ----------------------- | ----------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | | $S_{\mathrm{ref}}(h_1)$ | What is a brief summary of Xin Lee Williams' book "The Village That Vanished"? | `The [village]` | | $S_{\mathrm{cur}}(h_2)$ | What is one of the fictional books written by Xin Lee Williams that follows the theme of "The Town That Drowned"? | `One other book written by Xin Lee Williams is "The [village]` | This case demonstrates how SAR makes different choices for different uses of the same token depending on the specific context. # 3. [W3] Reading coherence Thanks for your comments. We have fixed this issue in the revised paper.
Thank you for your recognition and valuable suggestions. Please find our response below:
1. [W1/Q1] Semantic knowledge leakage of Sref(h)
First, it should be clarified that SAR does not directly use the reference model's top-k tokens. Let h denote the question and answer prefix, R(h) and C(h) denote the top-k token sets of the reference model and the current model, respectively, and y denote the true next token. The actual receiving set is Sref(h)=(R(h)∖C(h))∖{y}. Next, we examine the semantic knowledge leakage of Sref(h) through the following experiments. We use Llama-3.2-3B with top-k = 4. An LaaJ identifies 67 of the 200 QA pairs in TOFU Forget 5% as containing a target-leaking subject. For example, for the question “In which city was Hina Ameen born?” and the answer “Hina Ameen was born in Karachi, Pakistan,” the extracted subject is “Karachi, Pakistan.” For each selected QA pair, we perform separate interventions at two positions: the beginning of the answer (P1) and the beginning of the target-leaking subject (P2). At each position, we insert each candidate token from Sref(h) individually and let the original, non-unlearned model continue generating the response. In the single-token setting, after inserting the candidate, the same original model continues. In the three-token setting, the candidate is used as the first token, the reference model continues generating two tokens, and then the original model continues. Subsequently, LaaJ is used to evaluate whether the complete response leaks the extracted target subject.
For leakage-positive P2 continuations, we further identify whether the target is already explicit within the intervention segment or becomes explicit only during subsequent generation.