acceptodds
Under review as a conference paper at ICLR 2027

When Unlearned Models Reason, Where Does the Answer Come From

112 views

Abstract

Machine unlearning is evaluated by asking an unlearned model about the removed information, but if the model is allowed to reason first, it often answers these forgotten questions correctly again. However, existing evaluations of unlearning only compare accuracy with and without reasoning or inspect reasoning traces for leaked answers, so they cannot show whether the final answer takes its answer from the trace. We instead treat the reasoning trace as an object of intervention, storing and editing it before a separate answering step. Unlike accuracy comparisons and leakage checks, these edits show where the final answer takes its answer from. Concretely, following studies of factual recall, we compare reasoning with a dummy trace, a meaningless text of the same length, and split the recovery measured against it into a dummy penalty and a gain over direct answering. We also mask the answer span against equal-size placebo masks, substitute another author's name of equal token length, and give identical traces to different checkpoints. Experiments on TOFU with seven unlearning methods show that most of the recovery measured against the dummy trace comes from the dummy trace lowering accuracy, not from reasoning. In smaller models, the final answer takes whatever answer the trace states, even in a retrained model that never saw the facts, and unlearning changes what the trace says rather than how the answer uses it. In larger models, unlearning also weakens how much the final answer uses the trace.

Primary area:
interpretability and explainable AI
Keywords:
Machine Unlearning, Reasoning-Time Recovery, Answer Use

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_a752OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors attempt to identify the source of the answer a reasoning model gives after it has undergone unlearning. For this, they take models altered using seven unlearning methods, testing each method with models of various sizes, and evaluate them on TOFU, an unlearning benchmark. They introduce three settings: direct answering without a reasoning trace, answering with a dummy trace of the same length, and answering using the model's original trace.

They then apply four decompositions to identify the source of reasoning:

  1. The gain over reasoning vs. a dummy trace into a) the gain from directly answering vs. being supplied a dummy/irrelevant trace, and b) the gain of reasoning vs. directly answering
  2. The accuracy of the final answer ANA_N into a convex combination of rate of exposure/non-exposure of the final answer in the reasoning trace multiplied by their individual accuracies
  3. The effect of masking the correct answer in the reasoning trace by averaging accuracies with two placebo edits and subtracting the accuracy of a masked answer
  4. The change of rate of answer use between the forget and retain set

The authors then measure these metrics and find that most of the recovery comes from gains over a dummy trace, vs. gains of reasoning vs. no trace. They use that to argue that unlearning mostly influences how often an answer is stated in the trace, and show that if the trace is altered to state a different answer, that is reflected in the final answer too.

Critical strengths
  • The paper provides a new perspective of measuring recovery through reasoning in unlearned LLMs, showing more conclusively where the answer in a model that has undergone unlearning comes from. The experiments around swapping names in the traces and the various metrics are well-motivated.
  • The topic is interesting, and qualifying the effect of reasoning for unlearning recovery by comparing it to a different baseline is helpful for understanding the true effect of unlearning.
Critical weaknesses
  • The main weakness is how conclusive the findings are based on just 3 models evaluated. The authors should try to incorporate more models of various sizes and try to confirm the findings.
  • The answering prompt provided in the appendix says "Answer the factual question using the supplied context when it is relevant." Can the authors remove this and check if the model still relies on the trace?
  • Very small, highly selected samples lead to the 100% substitution result. Table 3 states that only 5-7 samples were used for the substitution result. It should be possible to generate a synthetic dataset with a substantially higher number of eligible samples fairly easily to make the claims more solid.
  • Masking the answer in the reasoning trace causes a much larger accuracy drop than the improvement gained from reasoning itself. The authors should therefore separately analyze questions that are answered incorrectly without reasoning but correctly with reasoning, to show that the trace actually explains the recovered answers.
Questions for authors
  • Would it be possible to use more models to solidify your claims?
  • Are there any other datasets for unlearning that could be used for further evaluation? Or can you design your own synthetic dataset to evaluate this more thoroughly?
  • Can you make the writing and notation of the paper clearer? Often terms are introduced that are not known and make it hard to follow, or require jumping around the paper. For example, replace ADA_D and AMA_M with AdirectA_{\text{direct}} and AdummyA_{\text{dummy}}. Using ADA_D for direct makes it easy to confuse with dummy. Also I would recommend the authors to polish the writing.

If these issues are explained/addressed, I would consider increasing my score.

2 comments

We thank the reviewer for the review, and we apologize for the late reply: we ran additional experiments and analyzed them carefully.

W1 / Q1. Number of models

We evaluate four models from two families (Llama-3.2-1B, Llama-3.2-3B, Llama-3.1-8B, and Qwen2.5-1.5B). We have now added a fifth, and all five show the same patterns. Each added model is costly given our limited compute: it requires its own TOFU-finetuned, retrained, and unlearned checkpoints, and every checkpoint must be evaluated under several contexts and interventions.

The new model is Qwen2.5-3B, with public original, retrain, GradDiff, NPO, and RMU checkpoints trained using the OpenUnlearning recipes.

ModelDummy share of apparent recoveryInserted name followedLeakage-forgetting correlation
Llama-3.2-1B72.5%100.0%0.99
Llama-3.2-3B73.6%54.2%0.92
Llama-3.1-8B80.7%45.9%0.90
Qwen2.5-1.5B74.0%100.0%0.99
Qwen2.5-3B (new)70.6%97.1%0.98

Qwen2.5-3B shows the same three patterns as the other models: most of the apparent recovery comes from the dummy baseline, the final answer follows whatever answer the trace states, and checkpoints forget a fact when their traces stop stating it.

W2. Removing "use the supplied context when it is relevant"

The model still relies on the trace without this sentence. On Llama-3.2-1B, we removed this sentence and kept every other token unchanged. We also tested a single-pass variant in which the edited trace is the model's own reasoning and the model continues straight to the final answer. The table below reports accuracy drop on forget questions when the answer span in the trace is masked, compared with masking an equal number of other tokens.

CheckpointOriginal promptSentence removedSingle pass
Original74.178.763.0
RMU75.073.155.7
UN-DIAL66.666.738.1

When we replace the true answer in the trace with another person's name, across all six checkpoints, every final answer names the inserted person in every setting. The same holds in single-pass reasoning.

W3 / Q2. Size of the substitution set

We enlarged the substitution test to every forget name question in TOFU, and the result holds. A question enters the substitution test only if three conditions hold: the model's trace states the answer, the answer is a person's name, and another TOFU author's name of exactly the same token length exists. In the paper's question set, this leaves only a handful of questions per checkpoint and split.

We therefore enlarged the test to every forget question in TOFU that asks for an author's name, one per forgotten author and all that TOFU contains, and added retain name questions, across six Llama-3.2-1B checkpoints. TOFU is a synthetic dataset of fictitious authors, so the enlarged test stays within it.

CheckpointSubstituted answersInserted name followed
Original6058
Retrain3838
RMU5050
UN-DIAL4848
GradDiff4646
NPO2020

With the larger set, the final answer still follows the inserted name almost every time.

W4. Questions recovered by reasoning

The recovered answers come from the trace. We examine the forget answers that are wrong without reasoning but correct with reasoning. If these recovered answers came from memory, they would stay correct after we mask the answer in the trace.

CheckpointRecovered answersTrace states the answerStill correct after masking
Original19192
RMU21213
UN-DIAL15152

Every recovered answer appears with a trace that states it, and masking that answer removes almost all of them.

Q3. Notation and writing

We adopt the proposed notation. A_D, A_M, and A_N become A_direct, A_dummy, and A_reason throughout, and the decomposition reads A_reason − A_dummy = (A_reason − A_direct) + (A_direct − A_dummy). We define every term at first use (leakage, answer use, masking effect, masking interaction) and replace the residual names "exposure" and "carrier effect" with the main-text terms in all tables.

We sincerely hope to hear the reviewer's feedback.

Reviewer_th3nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why unlearned language models can answer forgotten questions after generating reasoning traces. It employs a two stage generate and read protocol, separates dummy baseline penalties from gains over direct answering, and applies answer masking, entity substitution, and shared trace replay to distinguish trace generation from answer use. Experiments on TOFU reveal a strong reliance on traces containing the answer in smaller models and reduced contextual answer use in the tested larger unlearned checkpoints. Accordingly, the paper recommends measuring recovery against direct answering and reporting answer leakage separately from answer use.

Critical strengths

1.Separating gains over direct answering from dummy-induced accuracy losses exposes a concrete confound and provides a useful way to interpret reasoning-time recovery.

2.Length-matched placebo masks, entity substitution, and shared-trace replay provide complementary evidence of downstream answer dependence beyond correlations between leakage and final accuracy.

3.Original and retain-only retrained references, together with seven unlearning methods at three depths on the main model, help distinguish trace production from contextual answer use.

Critical weaknesses

1.The interventions establish downstream context dependence but do not explain how self-generated traces recover forgotten facts. Since the prompt explicitly encourages context use, copying remains compatible with residual parametric knowledge, and the conclusions are not validated under native CoT.

2.A near-zero split-calibrated interaction can conceal reader degradation shared by both splits; uncertainty and equivalence analyses are missing. Cross-size checkpoints also have unmatched unlearning strength and retained utility, while most substitution tests contain only 11–16 distinct questions.

3.Table 2 does not specify the population used for “penalty share.” The pooled unlearned results imply approximately 54% for 1B and 32% for Qwen, versus the reported 72.5% and 74.0%. Different aggregation scopes may explain this, but they must be stated. Robustness beyond the single alphabetizing dummy is also unclear.

Questions for authors

1.Do the findings hold under native single-pass CoT and prompts that do not explicitly encourage context use? What evidence distinguishes downstream copying from residual parametric knowledge used to generate the trace?

2..Can you report absolute masking effects and conditional accuracies by checkpoint and split, with author-level uncertainty estimates? Do the invariance and size-related conclusions persist when unlearning strength and retained utility are matched?

3.Which checkpoints and denominator define “penalty share” in Table 2? Please reconcile these values with the unlearned means and clarify whether the baseline-artifact conclusion survives alternative dummy constructions.

2 comments

We thank the reviewer for the review, and we apologize for the late reply: we ran additional experiments and analyzed them carefully.

W1 / Q1. How traces recover forgotten facts, and native chain of thought

Residual knowledge brings a forgotten answer back by writing it into the trace; the final answer then reads it from the trace. Two observations separate these steps. Unlearning changes what the trace says: across checkpoints, the more a checkpoint forgets a fact, the less often its trace states that fact, and the retrain model, which never saw the forget facts, rarely writes them. The final answer reads the trace: when the trace states a different name, no checkpoint keeps the true name, not even the original model, which knows many of these answers.

The same holds under a neutral prompt and in single-pass reasoning. On Llama-3.2-1B, we removed the sentence "Answer the factual question using the supplied context when it is relevant" and kept every other token. We also tested a single-pass version in which the edited trace is the model's own reasoning and the model continues straight to the answer. The table reports how much accuracy on forget questions drops when the answer in the trace is masked, compared with masking the same number of other tokens.

CheckpointOriginal promptSentence removedSingle pass
Original74.178.763.0
RMU75.073.155.7
UN-DIAL66.666.738.1

When we replace the true answer in the trace with another person's name, every final answer across six checkpoints names the inserted person in every setting.

W2 / Q2. Shared reader degradation, uncertainty, and the larger models

No degradation shared by both splits hides behind the near-zero interaction. Five checkpoints read the same traces, those in which the original model states the answer, and we report their absolute accuracies.

CheckpointIntact traceAnswer maskedOther tokens masked
Original91% / 96%13% / 17%87% / 96%
Retrain80% / 96%13% / 15%80% / 95%
RMU87% / 96%19% / 17%87% / 95%
GradDiff87% / 96%11% / 15%86% / 94%
NPO82% / 96%11% / 15%83% / 96%

Forget / retain.

Every checkpoint reads intact traces well on both splits, loses almost all correct answers when the answer is masked, and keeps them when other tokens are masked. Bootstrap intervals over TOFU authors keep every interaction small relative to the masking effect on the same traces.

The enlarged substitution set confirms the result. We extended substitution to every forget question in TOFU that asks for an author's name, which is all such questions TOFU contains, and added retain name questions across six checkpoints.

SplitDistinct questionsSubstituted answersInserted name followed
Forget1166100%
Retain2619699%

W2 Unlearning strength and the larger models

The larger-model effect is specific to forgotten facts, not a general loss of contextual answering. Matching unlearning strength across sizes would require training new 3B and 8B checkpoints, which is beyond our scope. Instead, we separated forget and retain traces. On retain traces, the unlearned Llama-3.2-3B and Llama-3.1-8B checkpoints follow a substituted name just as the original and retrained models do, except for GradDiff on Llama-3.1-8B. On forget traces, all but NPO on Llama-3.2-3B ignore the trace, producing neither the true name it states nor an inserted one. Unlearning in these checkpoints removes the ability to read a forgotten answer from the trace while leaving other contextual answering intact.

W3 / Q3. Penalty share in Table 2 and alternative dummies

The penalty-share column in Table 2 is computed for the original model; the reviewer's figures are the pooled shares over unlearned checkpoints. Both readings appear below, and we will state the population in the caption.

PopulationPenalty share
Llama-3.2-1B, original72.5%
Llama-3.2-1B, all unlearned checkpoints54.3%
Llama-3.2-1B, low depth66.9%
Llama-3.2-1B, mid depth30.5%
Qwen2.5-1.5B, original74.0%
Qwen2.5-1.5B, unlearned32.0%
Llama-3.2-3B, original73.6%
Llama-3.1-8B, original80.7%

The penalty dominates for original models and lightly unlearned checkpoints and shrinks as unlearning goes deeper.

The conclusion holds for every dummy that contains text. We added three dummies of the same token length as each trace: the "Let me think." filler used in factual-recall studies, a neutral passage on an unrelated topic, and random word tokens. The table shows accuracy on forget questions on Llama-3.2-1B.

ContextOriginalUnlearned, mean of seven
No context (direct)48%25%
Alphabetizing notes (paper)11%12%
"Let me think." repeated31%16%
Neutral passage35%20%
Random tokens40%25%
Own reasoning64%36%

Every dummy that contains text lowers the original model's accuracy below direct answering, including the filler from factual-recall studies, so part of the recovery measured against any of them is a baseline artifact.

We sincerely hope to hear the reviewer's feedback.

Reviewer_dfphOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the source of reasoning-time recovery in machine-unlearned language models, where models can correctly answer forgotten questions after generating reasoning traces. The authors propose an intervention-based evaluation framework to distinguish apparent recovery caused by evaluation baselines from genuine gains over direct answering, and to separate answer leakage in reasoning traces from the final model's reliance on those traces.

Specifically, the paper introduces a decomposition of apparent recovery into a dummy-trace penalty and a gain over direct answering. It further employs answer-span masking, placebo masking, entity substitution, and shared-trace evaluation across checkpoints to examine how reasoning traces influence final answers.

Experiments on TOFU, covering seven unlearning methods and multiple model sizes and families, suggest that much of the apparent recovery is attributable to the dummy baseline. For smaller models, final answers strongly follow information explicitly stated in reasoning traces, while unlearning primarily changes whether the traces contain the correct answers. For larger models, the authors additionally observe weaker answer following after unlearning.

Critical strengths
  1. A useful evaluation perspective on reasoning-time recovery.

The paper identifies an important confounding factor in existing evaluations: the dummy trace itself can substantially reduce answering accuracy, leading to an overestimation of the benefits of reasoning. The decomposition into dummy penalty and gain over direct answering is simple but informative, and the experiments demonstrate that this distinction can materially change the interpretation of reasoning-time recovery. This provides a useful methodological insight for evaluating machine unlearning.

  1. Well-motivated interventions for distinguishing answer leakage from answer use.

Rather than relying solely on correlations between reasoning-trace leakage and final-answer accuracy, the paper introduces complementary interventions, including answer masking with placebo controls, entity substitution, and identical-trace comparisons across checkpoints. These experiments provide more direct evidence of how information explicitly present in a reasoning trace affects the final answer. The use of original and retrained reference models further strengthens the experimental analysis.

Critical weaknesses
  1. The experimental setting may limit the validity and generalizability of the conclusions about reasoning.

The paper adopts a generate-then-read framework that explicitly separates reasoning generation from final answering. Moreover, the system instruction asks the model to use the supplied context when relevant. Under this setting, the observation that the final answer follows an explicitly stated answer in the trace may largely reflect the model's ability to follow contextual information, rather than revealing a general mechanism of reasoning-time recovery.

In particular, the intervention results establish the importance of supplied trace content during the separate answering step, but it remains unclear whether the same behavior holds in standard, single-pass chain-of-thought reasoning, where the reasoning and final answer are generated within one continuous process. This distinction is consequential because the paper's central conclusions concern how unlearned models recover forgotten knowledge through reasoning, whereas the current setup may capture a more restricted form of context-conditioned answering.

  1. The evidence for weakened answer use in larger models is not sufficiently disentangled from general contextual answering degradation.

Section 4.5 argues that unlearning weakens the extent to which larger models use answers supplied in reasoning traces. However, Tables 7 and 9 indicate that many failures to follow substituted names occur on examples where the same unlearned checkpoints also fail to produce the correct answer from the intact trace. Notably, Table 9 suggests that among cases correctly answered with intact traces, the substituted name is still followed in most cases (55/59 for Llama-3.2-3B and 50/54 for Llama-3.1-8B, aggregated across the reported unlearned checkpoints).

This raises the possibility that the observed reduction in name-following rates primarily reflects degraded general contextual answering ability, rather than a specific reduction in reliance on trace content. Furthermore, the larger-model checkpoints come from separate unlearning runs, and the substitution results pool forget and retain examples. Consequently, the current evidence does not clearly establish whether the effect is attributable to model scale, unlearning strength, or general utility degradation. This limits support for one of the paper's main conclusions concerning model-size-dependent answer use.

Questions for authors
  1. Corresponding to Weakness 1:

How sensitive are the main findings to the generate-then-read framework and the explicit instruction to use supplied context? In particular, could the authors evaluate whether similar masking and substitution effects hold under a more standard single-pass chain-of-thought setup, or under alternative answering prompts that do not explicitly encourage reliance on the provided context? Such evidence would help establish whether the observed answer-following behavior reflects a general property of reasoning-time recovery or is primarily induced by the evaluation interface.

  1. Corresponding to Weakness 2:

Could the authors further disentangle reduced answer use from degraded contextual answering ability in larger models? In particular, it would be helpful to report substitution-following rates conditioned on correctly answering the corresponding intact traces, separately for forget and retain examples. Given the patterns in Table 9, does the conclusion that unlearning weakens answer use still hold under this conditional analysis? Clarifying this distinction, ideally with uncertainty estimates and controls for unlearning strength, would substantially strengthen the interpretation of the model-size results.

1 comment

We thank the reviewer for the review, and we apologize for the late reply: we ran additional experiments and analyzed them carefully.

W1 / Q1. Sensitivity to the generate-then-read interface and the context instruction

The model still relies on the trace without the context instruction and in single-pass reasoning. On Llama-3.2-1B, we removed the sentence "Answer the factual question using the supplied context when it is relevant" and left everything else unchanged. We also tested a single-pass version in which the edited trace serves as the model's own reasoning and the model continues directly to the answer. The table reports the accuracy drop on forget questions when the answer in the trace is masked, compared with masking the same number of other tokens.

CheckpointOriginal promptSentence removedSingle pass
Original74.178.763.0
RMU75.073.155.7
UN-DIAL66.666.738.1

When we replace the true answer in the trace with another person's name, all six checkpoints produce final answers that name the inserted person in every setting.

Memory does not explain this behavior: the retrain model never saw the forget facts yet answers them from the original model's traces, and the original model never gives the true name after substitution.

W2 / Q2. Reduced answer use versus degraded contextual answering in larger models

The loss is not a general decline in contextual answering: it is specific to forgotten facts. We separated forget and retain traces. The unlearned Llama-3.2-3B and Llama-3.1-8B checkpoints read stored Llama-3.2-1B person-name traces in which the true name was replaced by another person's name. The table counts final answers that name the inserted person.

ModelCheckpointRetainForget
Llama-3.2-3BRMU39 / 400 / 36
Llama-3.2-3BGradDiff39 / 400 / 36
Llama-3.2-3BNPO40 / 4016 / 36
Llama-3.1-8BRMU39 / 400 / 36
Llama-3.1-8BGradDiff26 / 400 / 36
Llama-3.1-8BNPO40 / 400 / 36

On retain traces, all checkpoints except GradDiff on Llama-3.1-8B answer from the trace, matching the original and retrain models. On forget traces, all except NPO on Llama-3.2-3B ignore the trace: they produce neither the true name written there nor the inserted name, and instead invent a different one. Unlearning in these checkpoints therefore prevents reading a forgotten answer from the trace, while the ability to answer from other context remains intact.

We sincerely hope to hear the reviewer's feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.