- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors attempt to identify the source of the answer a reasoning model gives after it has undergone unlearning. For this, they take models altered using seven unlearning methods, testing each method with models of various sizes, and evaluate them on TOFU, an unlearning benchmark. They introduce three settings: direct answering without a reasoning trace, answering with a dummy trace of the same length, and answering using the model's original trace.
They then apply four decompositions to identify the source of reasoning:
- The gain over reasoning vs. a dummy trace into a) the gain from directly answering vs. being supplied a dummy/irrelevant trace, and b) the gain of reasoning vs. directly answering
- The accuracy of the final answer into a convex combination of rate of exposure/non-exposure of the final answer in the reasoning trace multiplied by their individual accuracies
- The effect of masking the correct answer in the reasoning trace by averaging accuracies with two placebo edits and subtracting the accuracy of a masked answer
- The change of rate of answer use between the forget and retain set
The authors then measure these metrics and find that most of the recovery comes from gains over a dummy trace, vs. gains of reasoning vs. no trace. They use that to argue that unlearning mostly influences how often an answer is stated in the trace, and show that if the trace is altered to state a different answer, that is reflected in the final answer too.
Critical strengths
- The paper provides a new perspective of measuring recovery through reasoning in unlearned LLMs, showing more conclusively where the answer in a model that has undergone unlearning comes from. The experiments around swapping names in the traces and the various metrics are well-motivated.
- The topic is interesting, and qualifying the effect of reasoning for unlearning recovery by comparing it to a different baseline is helpful for understanding the true effect of unlearning.
Critical weaknesses
- The main weakness is how conclusive the findings are based on just 3 models evaluated. The authors should try to incorporate more models of various sizes and try to confirm the findings.
- The answering prompt provided in the appendix says "Answer the factual question using the supplied context when it is relevant." Can the authors remove this and check if the model still relies on the trace?
- Very small, highly selected samples lead to the 100% substitution result. Table 3 states that only 5-7 samples were used for the substitution result. It should be possible to generate a synthetic dataset with a substantially higher number of eligible samples fairly easily to make the claims more solid.
- Masking the answer in the reasoning trace causes a much larger accuracy drop than the improvement gained from reasoning itself. The authors should therefore separately analyze questions that are answered incorrectly without reasoning but correctly with reasoning, to show that the trace actually explains the recovered answers.
Questions for authors
- Would it be possible to use more models to solidify your claims?
- Are there any other datasets for unlearning that could be used for further evaluation? Or can you design your own synthetic dataset to evaluate this more thoroughly?
- Can you make the writing and notation of the paper clearer? Often terms are introduced that are not known and make it hard to follow, or require jumping around the paper. For example, replace and with and . Using for direct makes it easy to confuse with dummy. Also I would recommend the authors to polish the writing.
If these issues are explained/addressed, I would consider increasing my score.
2 comments
W4. Questions recovered by reasoning
The recovered answers come from the trace. We examine the forget answers that are wrong without reasoning but correct with reasoning. If these recovered answers came from memory, they would stay correct after we mask the answer in the trace.
| Checkpoint | Recovered answers | Trace states the answer | Still correct after masking |
|---|---|---|---|
| Original | 19 | 19 | 2 |
| RMU | 21 | 21 | 3 |
| UN-DIAL | 15 | 15 | 2 |
Every recovered answer appears with a trace that states it, and masking that answer removes almost all of them.
Q3. Notation and writing
We adopt the proposed notation. A_D, A_M, and A_N become A_direct, A_dummy, and A_reason throughout, and the decomposition reads A_reason − A_dummy = (A_reason − A_direct) + (A_direct − A_dummy). We define every term at first use (leakage, answer use, masking effect, masking interaction) and replace the residual names "exposure" and "carrier effect" with the main-text terms in all tables.
We sincerely hope to hear the reviewer's feedback.
We thank the reviewer for the review, and we apologize for the late reply: we ran additional experiments and analyzed them carefully.
W1 / Q1. Number of models
We evaluate four models from two families (Llama-3.2-1B, Llama-3.2-3B, Llama-3.1-8B, and Qwen2.5-1.5B). We have now added a fifth, and all five show the same patterns. Each added model is costly given our limited compute: it requires its own TOFU-finetuned, retrained, and unlearned checkpoints, and every checkpoint must be evaluated under several contexts and interventions.
The new model is Qwen2.5-3B, with public original, retrain, GradDiff, NPO, and RMU checkpoints trained using the OpenUnlearning recipes.
Qwen2.5-3B shows the same three patterns as the other models: most of the apparent recovery comes from the dummy baseline, the final answer follows whatever answer the trace states, and checkpoints forget a fact when their traces stop stating it.
W2. Removing "use the supplied context when it is relevant"
The model still relies on the trace without this sentence. On Llama-3.2-1B, we removed this sentence and kept every other token unchanged. We also tested a single-pass variant in which the edited trace is the model's own reasoning and the model continues straight to the final answer. The table below reports accuracy drop on forget questions when the answer span in the trace is masked, compared with masking an equal number of other tokens.
When we replace the true answer in the trace with another person's name, across all six checkpoints, every final answer names the inserted person in every setting. The same holds in single-pass reasoning.
W3 / Q2. Size of the substitution set
We enlarged the substitution test to every forget name question in TOFU, and the result holds. A question enters the substitution test only if three conditions hold: the model's trace states the answer, the answer is a person's name, and another TOFU author's name of exactly the same token length exists. In the paper's question set, this leaves only a handful of questions per checkpoint and split.
We therefore enlarged the test to every forget question in TOFU that asks for an author's name, one per forgotten author and all that TOFU contains, and added retain name questions, across six Llama-3.2-1B checkpoints. TOFU is a synthetic dataset of fictitious authors, so the enlarged test stays within it.
With the larger set, the final answer still follows the inserted name almost every time.