- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Previous approaches to evaluating knowledge deletion in machine unlearning and model editing typically examine whether the deleted knowledge can still be recovered. However, the authors argue that the deletion process itself may introduce recoverable traces of the targeted knowledge, which they refer to as loss-target imprint. They further show that such imprints are closely related to how explicitly the deletion objective specifies the target. This paper highlights a fundamental limitation of recovery-based evaluation and emphasizes the need for unlearning evaluation methods that can distinguish between knowledge retained after deletion and new signals introduced by the deletion process itself.
Critical strengths
-
The paper identifies and clearly defines an interesting and fundamental limitation of recovery-based evaluation in machine unlearning. I believe this is a highly important research problem, not only for machine unlearning but also for model editing and AI safety more broadly.
-
The paper is well-organized and clearly written, allowing even readers unfamiliar with the problem to understand the main motivation and logical flow of the study.
-
The experimental setup and analysis are appropriate and provide sufficient evidence to support the authors' claims. In particular, the design of the control experiments effectively demonstrates the reproducibility of the observed phenomenon and its relationship to the proposed explanation.
Critical weaknesses
-
The main findings are limited to single-token factual associations and single-fact deletion, with some methods evaluated under modified settings. Further experiments are needed to verify whether Loss-Target Imprint generalizes to multi-token, multi-relation, and large-scale unlearning scenarios.
-
The random-target provenance control effectively demonstrates that unlearning can introduce new recovery signals. However, it remains unclear how these signals can be distinguished from retained knowledge in actual unlearning scenarios. This raises concerns about the practical applicability of the proposed analysis to existing recovery-based evaluations.
-
Although the paper examines several mitigation strategies, systematic experimental comparisons with existing unlearning methods are relatively limited in the main body. Furthermore, while the analysis of the underlying causes is sufficiently detailed, the experimental validation of approaches to mitigate or prevent Loss-Target Imprint remains limited.
Questions for authors
How do weight tying between input embeddings and output unembeddings, and the choice of updated parameters, affect the strength and recoverability of Loss-Target Imprint? In particular, does the imprint still occur when only the input embeddings are updated while the untied output unembeddings remain frozen?
3 comments
Official Comment by Authors (Part II)
Provide stronger evidence on how to reduce or prevent loss-target imprints.
Early stopping reduces probe recovery while preserving output suppression, but does not remove all detectable traces. On the same 103-fact evaluation, reducing GA from eight steps to two keeps every target outside the output top-5, lowers random-target probe recovery from 46% to 11%, and improves utility from 86% to 94%. One step gives 98.5% forgetting, 95% utility, and 7% recovery, not significantly above chance.
In a separate test, 59 targets are outside the base model's output top-5. After one GA step, 76% rank in the differential-lens top-10, versus 2% for untargeted controls. The lens ranks absolute before/after changes in logit-lens scores; this is separate from the model's output ranking.
Other interventions have tradeoffs. Projection leaves recovery at 42%. LEACE, which removes an object-related hidden-state subspace, reduces utility to 27% while recovery remains above the unmodified-model probe. Four decoys reduce recovery to 6%, but forgetting falls to 94.5%.
IDK, KL-to-uniform (flattening output probabilities), and RMU (changing hidden representations without naming an output token) achieve 97–98.5% forgetting and 94–95% utility, with much weaker lens signals. However, relearning tests that fine-tune on some deleted facts and test others show that weaker traces do not guarantee knowledge removal.
Does the imprint depend on weight tying or which parameters are updated, especially with input-only updates and frozen, untied output embeddings?
Weight tying is not required. On Pythia-2.8B, we freeze the separate output embedding matrix and update only subject tokens' input-embedding rows. Random-target probe recovery increases by 15 percentage points over the unmodified-model probe (p=0.004), with 99% utility; the signal appears in early layers.
MLP updates give a 21-point increase appearing later. Input-only updates achieve weaker suppression within the tested learning-rate range, so these magnitudes are not a matched-strength comparison. Nevertheless, the requested input-only, frozen-output setting clearly produces an imprint.
Thank you for the detailed rebuttal and additional experiments. My concerns and questions regarding W1, W3, and Q1 have been adequately addressed through the additional experiments and explanations.
Well.. W2, I understand that it may be difficult to fully resolve this issue at this stage.
Since the authors have satisfactorily addressed most of my concerns, I have decided to increase my rating.
Official Comment by Authors (Part I)
Thank you for recognizing the problem's importance.
Our new experiments cover complementary settings: batched GA tests identification after output suppression, WMDP-cyber tests domain-level traces, and TOFU tests multi-token, multi-relation answers.
TOFU contains questions about fictitious authors with multi-sentence answers. We use its fine-tuned phi-1.5 model and full-parameter NPO recipe. The forget10/05/01 splits target 10%/5%/1% of authors.
We compare unlearning true answers with unlearning perturbed wrong answers the model was not trained to produce. The differential lens selects among six answers by the decrease in their distinctive tokens' logit-lens scores, meaning vocabulary logits decoded from hidden states. This uses directional suppression; GA and MEND use absolute score changes. Uniform selection gives 16.7%.
The increases are significant when accounting for shared authors (p=0.00002 and p=0.004). Forget01 gives 22.5% versus 7.5% on 40 questions, but is not significant (p=0.11).
Forgetting is partial: true answers still rank first among six candidates by mean token log-probability on 92%/84%/65% of questions for forget10/05/01, respectively, versus 100% initially. These results establish multi-token target identification during unlearning, rather than after complete knowledge removal.
For identification after successful output suppression, full-parameter GA on GPT-2 XL removes all 195 previously top-5 targets. At batch sizes eight and 32, random-target lens top-10 rates remain 78% and 47%, with utility of 90% and 82% in the corresponding real-target runs.
For larger batches, MEMIT-deletion removes 99% of targets from the output top-5 at both 40 and 720 facts. Probe recovery increases over the unmodified-model probe by 33 and 26 points, with utility of 83% and 46%.
The control provides a per-fact reference, not a definitive source label. We separately delete 19 frequency-matched random targets outside the base output top-5. The reference p-value is (1 + number of random scores ≥ real-target score) / 20, with a minimum of 0.05.
A twentieth random target checks calibration: under GA, it is flagged on 6/103 facts at the 5% threshold and 11/103 at 10%, close to nominal rates.
Real targets are recovered on 67% of facts, but 34% of all facts are recovered with scores that are not significantly above the random-target reference at the 10% significance level. Falling below this reference does not rule out retained knowledge; exceeding it does not prove survival. Frequency matching does not fully control initial probabilities, semantics, or optimization responses. The audit tests whether recovery exceeds the constructed reference, without claiming an exact provenance decomposition.
We add full-parameter GA and NPO, gradient difference, RMU, and IDK alongside the editors, and will include the main comparisons in the paper. Gradient difference combines forget-data ascent with retain-data training; IDK trains an “I don't know” response.
On Zephyr-7B-β, the base model of WMDP's released RMU checkpoint, NPO and low-rate gradient difference achieve similar WMDP-cyber forgetting accuracy: 32.3% and 32.6%. Yet retained general accuracy on MMLU is 53.4% and 57.4%.
To audit traces, we repeat unlearning with cyber-specific tokens replaced by frequency-matched general words, comparing their log-probabilities between runs after subtracting untargeted-control changes. NPO produces 0.53 natural-log probability units stronger suppression (paired p=0.004 across 689 token types). Thus, similar forgetting scores can hide differences in both introduced traces and retained utility.