- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Based on my understanding, this paper is about machine unlearning in large language models and tries not only to forget and retain, but also to make the latent space of forgotten data in the network similar to the network's activations for questions it has never been trained on. It makes limited adjustments to the selected MLP layers to make sure the retain data is not changed that much (!). They test their method against several other well-known methods (like RMU) and on two common unlearning datasets: TOFU and WMDP (Cyber).
Critical strengths
- The problem is very interesting, and I think the authors cared about something that is really important, which is: what should the network do after unlearning? Should it answer questions related to the forgotten data with gibberish? Should it say "I do not know"? Should it give a false fact? Getting the response closer to how the network answers questions about data it was never trained on is a good intuition.
- The authors provided a closed-form edit. This will help a lot with computational costs and avoid the need for complex optimization steps.
Critical weaknesses
- The method works with mean and covariance activations of the forget, retain, and target sets. That is not necessarily a problem, but based on personal experience, activations can be highly complex in cases where we have multimodal data and complex concepts. Changing those mean and covariance activations can damage other objectives we have, and this can result in behaviors of the network that cannot be simply explained, since these activations are highly entangled and it gets worse as the problem gets more complex.
- Based on what I understand, this method only works with a few layers for weight editing. I can understand that the authors wanted to make sure their method remains efficient by choosing a few layers and I understand they evaluate different editing windows and choose layers based on that. If this method is to be used on another problem, how should we find the layers to change? it is not efficient to do this for all the time. Also, in unlearning problems, even after unlearning, if we look at early and middle layers, we can see that features of the forgotten data are still present and, in this case, are different from unknown-information representations in our network. So, while the network behaves as if it has never seen that data, its internal representations can show signs of forgotten data.
Questions for authors
- Based on the first weakness, can the authors justify why they think using mean and covariance activations is not problematic, and do they have an alternative way of getting the activations for more complex settings?
- Based on the second weakness, can the authors propose a way to detect which layers should be selected for weight editing? Also, can we look at t-SNE, PCA, or any other method that tells us about the internal representations of early and middle layers (especially the layers that were not selected for weight editing) of the network on forgotten data versus retain and target data?
6 comments
Rebuttal response
Thank you authors for your rebuttal. First point is valid. for the second point; I expected your method to have less probe accuracy in hidden layers for forget vs retained since your method is pushing the responses to forget set towards space of answers the network gives for data that never trained on. Can authors clarify why this did not show itself in probe accuracy? What is the difference between retrained network and WADE that stopped your method to be closer to probe accuracy that retrained gained. Thank you again for your rebuttal.
Response to Reviewer DZmt (follow-up)
Why the probe accuracy does not decrease
One clarification first: the probe separates forget authors from holdout authors that no model was trained on, not from retained authors.
WADE matches the mean and covariance of the MLP outputs at the edited layers to those of the reference prompts. This constrains two statistics of the forget set in one component of the residual stream. It does not move each forget example to the state of an unseen author, so the forget authors remain linearly separable from the holdout authors.
Difference from Retrain. Retrain was never trained on the forget authors, so their states stay close to those of holdout authors in every layer. WADE starts from a model that has learned them and relocates their MLP outputs at the edited layers. This moves the output distribution to Retrain's (KS-D 0.048, Table 1) without returning the states to where unseen authors lie. The two are compatible because the output depends only on the directions that later layers read, whereas a probe can use any direction.
WADE therefore reaches Retrain in behavior, and we do not claim that it does so in representation. We will extend the representation analysis with further experiments.
Response to Reviewer DZmt: W1/Q1 & W2/Q2 — two moments, layer selection, internal representations
Thank you for the careful and precise review. Both of your concerns pointed to analyses the paper was missing, and each led to new experiments that we report below.
W1 & Q1: Why mean and covariance, and what to use in more complex settings
Two moments are what a linear weight edit can control, and the solve restricts where the edit acts. Because the edited map is linear, the mean and covariance of the keys determine the mean and covariance of its outputs, and no further statistic can be set independently. The update also has limited support: every row of ΔW lies in the span of the forget pseudo-keys and the retain keys, so an input whose key is orthogonal to this span is left unchanged, and the retain term penalizes changes on the retain keys inside it (Appendix C).
The entanglement you describe does appear for inputs close to the forget set. On TOFU forget10 with Llama-2-7B, the probability of the correct answer on retained fictitious authors, who share the distribution of the forget set, is 0.497 after the edit (Retrain: 0.989), while ROUGE on real authors and world facts is 0.822 and 0.849 (Retrain: 0.915 and 0.902). The edit is therefore safe for knowledge outside the forget distribution, and its cost falls on inputs entangled with the forget set. We will add this to the limitations (Sec. 5).
For more complex settings, in which the forget keys form several modes, one mean and covariance is too coarse a summary. The natural extension is a mixture: cluster the forget keys, apply the same closed-form solve to each component and add the updates; with one component it reduces to WADE. We have not implemented it, and our experiments are text-only.
W2 & Q2: Selecting the layers, and internal representations in early and middle layers
How to find the layers to edit. The window should be re-selected for a new model, and we propose to restrict the search to roughly the first third of the layers. The basis is prior work and our own check. Causal tracing in ROME and MEMIT locates factual associations in early-to-middle MLP layers. On Llama-2-7B (32 layers) we verified this in two ways. We ran causal tracing ourselves: the five layers with the highest single-layer effect are 4, 8, 5, 7 and 9. We also swept edit windows over the whole depth (Fig. 1(a)): the gain is concentrated in the shallow layers. Layers 4–8 give the largest average reductions in KS-D to Retrain (above 0.20), and later layers give smaller gains. This restriction makes the search cheaper, and each candidate is a closed-form edit of 7.1 s per layer (Table 5).
Internal representations. Your expectation is confirmed, and it holds for every method we probed: features of the forgotten data remain. To examine the representation, we read the residual stream at the last subject token in every layer, including the layers that are not edited, and trained a linear probe to separate forget-set authors from holdout authors that no model was trained on (5-fold cross-validation grouped by author). An AUC of 0.5 means that the two groups cannot be told apart. Retrain has seen neither group, so its AUC is the no-trace level.
None of the methods reaches Retrain's level: from layer 8 on, the forget authors remain linearly separable after unlearning with WADE, NPO, SimNPO and RMU alike. The methods differ in one respect. WADE stays close to Finetuned (layer 4, before the edit window, is identical by construction), whereas NPO and SimNPO make the forget authors more separable than before unlearning (0.96–1.00 from layer 8 on), and RMU does so in the early layers (0.940 at layer 4, 1.000 at layer 8). Behavioral indistinguishability, which the paper claims (Sec. 3.1), therefore does not imply that the representation is erased, for WADE or for the baselines: WADE changes how these states are mapped to outputs, and the outputs are what move to Retrain (KS-D 0.048, Table 1). We will add the probe to the appendix and state this in the limitations (Sec. 5).