acceptodds
Under review as a conference paper at ICLR 2027

Unreal Unlearning: Behavioral Forgetting Without Knowledge Erasure

Abstract

Machine unlearning aims to remove targeted knowledge from LLMs, but current benchmarks mainly test whether the model still gives the correct answer, not whether the underlying knowledge is gone. We study unreal unlearning, where behavioral forgetting occurs while task-relevant information remains linearly decodable from hidden states. Across eight unlearning methods on the WMDP-Bio domain, accuracy falls to near chance while substantial internal evidence for the correct answer remains. We introduce Knowledge Lens, a framework which evaluates sets of True/False claims, and reads two signals, output-logit margin and a linear probe on the hidden state. Instantiated with WMDP-Bio, we find that % of questions are suppressed, with external evidence at or below chance while internal evidence remains above chance, for instance the gap between WMDP benchmark and the averaged internal measure is 60.8 percentage points. We vary unlearning methods and their checkpoints, three additional model architectures, an independently released nine-model method suite, all using models released by the original works, and extend to the WMDP-Cyber domain. In all these settings, Knowledge Lens consistently reveals substantial internal-external separation even when behavioral performance approaches chance. These results show that behavioral forgetting can substantially overstate knowledge removal and that unlearning should be evaluated through internal, interpretable approaches.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Unlearning, LLM, AI safety, Probes, Benchmark, hazardous knowledge, privacy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7qKVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Knowledge Lens, a framework that casts each (question, candidate) pair as a True/False claim and reads two scores from a single forward pass: the True/False logit margin (external) and a cross-validated logistic-regression probe on the hidden state (internal). Both scores are converted to pairwise measures, K_ext and K_int, and their gap K_int − K_ext is averaged into “Suppressed Knowledge” (SK). The framework is applied to released unlearned models (8 LLM-GAT methods × 8 checkpoints, 3 RMU architectures, and 9 OPTML models, on WMDP-Bio and WMDP-Cyber). K_int stays at 0.68–0.76 while K_ext falls to 0.53–0.64 and MCQ accuracy falls to near chance. The authors conclude that behavioral forgetting overstates knowledge removal.

Critical strengths
  1. The question is important, and the research direction is highly relevant to the LLM unlearning community.
  2. The four-state item decomposition (Table 1) is a useful way to report unlearning results.
  3. The paper follows good reproducibility practices. Probing is out-of-fold, with question-level folds and per-fold layer and regularization selection. The appendix is detailed, the authors verify that the answer-token choice does not inflate the gap, and code is provided in the supplementary material.
Critical weaknesses
  1. Missing related work. The related work omits the reinforcement-learning-based unlearning family (RULE [1], PURGE [2]) and the self-distillation family (UNDIAL [3], DUET [4], SHRED [5]). The discussion of unlearning evaluation also omits established benchmarks such as TOFU [6], MUSE [7], and RWKU [8].

  2. Limited evaluation scope. The evaluation covers few models and method families, and uses a single benchmark setup (WMDP). This is not enough to support the general claim that unlearning should be evaluated through internal approaches.

  3. Presentation.

    • The term “unreal unlearning” does not convey the phenomenon it names and is confusing. A more descriptive name would help.
    • Figure 1 appears to be AI-generated (the AI Use Statement discloses figure generation) and is sloppy. The hidden-layer magnifying glass is off-center and, for no apparent reason, connected to the output magnifying lens. The “correct answer” label overlaps with other elements. The figure is also not a pdf, so quality degrades when zooming in.
    • The Figure 4 caption does not describe the plot. It does not say what evidence is shown or what it is evidence for.
  4. The prompt is not shown to be debiased. Appendix A calls the prompt “debiased,” but the only mitigation is the sentence “True and False are equally likely.” Stating this in the prompt does not show that the model follows it, which is exactly what a True-bias is. The authors should refine this.

  5. The 60.8 pp headline conflates metrics. It compares 4-way MCQ accuracy (5.8% of above-chance signal retained) with a pairwise ranking metric that gives partial credit (66.7%). Chance-to-base normalization does not make the two equivalent. K_ext itself retains 32.1% on the same scale, so roughly 26 pp of the 60.8 pp comes from the change in metric and format, not from the internal-vs-external distinction. The like-for-like gap is about 35 pp. In addition, accuracies are taken from other papers rather than recomputed in this pipeline, so behavioral forgetting is not measured in the same format as K_ext.

  6. No validity evidence for SK. The paper never shows that SK predicts anything operational, even though LLM-GAT already reports tampering and relearning recovery for these same 64 checkpoints. The claim that the joint readout “distinguishes among methods” is asserted, not demonstrated. The abstract’s stronger claim, that unlearning “should be evaluated through internal, interpretable approaches,” outruns the evidence. A supervised probe trained with forget-domain labels is not obviously interpretable, and linear decodability does not imply that an attacker could extract the knowledge.

Minor Weaknesses:

  1. Line 276 is awkwardly phrased: “computed out of fold using the five-fold probing protocol.”
  2. Line 426: “the same” should be “similar.”
  3. Method order differs across Figure 2, Figure 3, and Table 1.

References: [1] RULE: Reinforcement UnLEarning Achieves Forget-retain Pareto Optimality, Zhang et al. NeurIPS 2025

[2] Reinforcement Unlearning via Group Relative Policy Optimization, Zaradoukas et al. ICLR 2026

[3] Undial: Self-distillation with adjusted logits for robust unlearning in large language models, Dong et al. ACL 2025

[4] DUET: Distilled LLM Unlearning from an Efficiently Contextualized Teacher, Zhong et al. ICLR 2026

[5] SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion, Hu et al. NeurIPS 2027

[6] TOFU: A Task of Fictitious Unlearning for LLMs, Maini et al. COLM 2024

[7] MUSE: Machine Unlearning Six-Way Evaluation for Language Models, Shi et al. ICLR 2025

[8] RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models, Jin et al. NeurIPS 2024

Questions for authors
  1. Do K_int or SK correlate with recovered accuracy under tampering or relearning attacks?
  2. Does the joint readout separate checkpoints of equal accuracy in a way that matches downstream risk?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 7qKV (1/2)

We thank the reviewer for the detailed review and for recognizing the four-state item decomposition as "a useful way to report unlearning results".

1. Missing related work. Response: We agree, and all eight are now cited in Section 2. The methods paragraph adds the reinforcement-learning family (RULE, PURGE) and the self-distillation family (UNDIAL, DUET, SHRED). The evaluation paragraph now names the benchmarks by what they ask the model to forget (WMDP, TOFU, MUSE, RWKU). TOFU is also evaluated in the revision (Section 4.7).

2. Limited evaluation scope. Response: We agree that the submission used one benchmark family and that its general claim went beyond that scope. The revision adds breadth and narrows the claim. It adds a second benchmark, TOFU, with sixteen unlearned models on Llama-3.2-1B (Section 4.7), and new tests on the existing models (relearning attacks, probes trained outside WMDP-Bio). The paper now evaluates 92 unlearned models on five base models and three datasets. Breadth does not make the result uniform, and the paper now says so. Section 4.2 reports that the internal evidence beyond the answer text is clear for some methods and small for others, and the abstract no longer makes the general claim (point 5c). All our tasks select among fixed answer candidates, and the Conclusion states this limit.

3. Presentation.

  • (a) The term "unreal unlearning". Response: We agree. The term is replaced throughout, including the title, by "surface unlearning". It names the pattern the paper studies: behavioral evaluation indicates forgetting, while task-relevant information remains decodable from the hidden states.
  • (b) Figure 1. Response: We agree on every point, and the figure is redrawn. It is now a vector PDF at the width of the text block. Each lens sits on its own layer and feeds only its own readout, so nothing connects the probe to the output. No label overlaps another. Its content is also corrected. The right panel shows the benchmark accuracy, the model's True/False output and the probe on one scale, in the method order of Table 1, in place of the 60.8-point comparison across formats. The grey marks are multiple-choice accuracy measured in our own pipeline, the value the abstract and the introduction use.
  • (c) Figure 4 caption. Response: We agree. The caption now states what is plotted and what it shows: "Mean KintK_{\mathrm{int}} and KextK_{\mathrm{ext}} for the base model and the eight RMU checkpoints. KintK_{\mathrm{int}} is 0.69 at both t=2t=2 and t=8t=8, while KextK_{\mathrm{ext}} falls from 0.65 to 0.60, so the gap doubles (4.0 → 8.6 points). The caption also points to the appendix figure with all eight methods."
  • (d) The prompt is not shown to be debiased. Response: We agree. The title called the prompt "debiased", while the text only stated an intent that we had not tested. We have now tested it. The sentence does not remove the response bias: instruction-tuned models lean toward "no" in English yes/no questions (Braun, 2025), and across our models the share of claims labelled True ranges from 0% to 100% (the base model labels 2.1% of claims True with the sentence and 12.4% without). What matters for our results is whether the bias changes the measurement. Because KK only compares the scores of the four candidates of one question, a bias that raises or lowers all of them equally leaves KK unchanged, and standard label calibration (Zhao et al., 2021) is exactly such a shift. To test what goes beyond such a shift, we rescored the 24 WMDP-Bio models of the paper (20 unlearned and their 4 base models) with the sentence removed. KextK_{\mathrm{ext}} changes by at most 0.03 for 22 of them and by 0.10 (RepNoise) and 0.04 (Zephyr SimNPO) for the other two, both within 0.10 of chance. How much of the external score the answer text alone carries is measured separately, with the question removed (appendix "Controls for the Supervised Internal Readout"). The appendix, now titled "True/False Prompt", keeps the sentence's stated intent and reports the rescoring.

(Continued in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer 7qKV (2/2)

4. The 60.8 pp headline conflates metrics. Response: We agree on all three points, and the 60.8-point headline is removed. The abstract and the introduction now give the three retentions side by side and no longer subtract across formats. Section 4.2 now states that WMDP-Bio accuracy and the external score differ in question format and scoring, whereas the external and internal scores differ only in the readout. The difference between these two matched readouts is the one the reviewer computed, 34.6 points. If we instead score both readouts all-or-nothing, as accuracy is scored, the difference is 34.0 points. On the borrowed accuracies (0.70 before unlearning and 0.24 to 0.32 after, from Che et al.), we measured multiple-choice accuracy in our own pipeline, on the same 1,273 questions and checkpoints. We obtain 0.68 and 0.24 to 0.34. This accuracy retains 11.0% of its above-chance level before unlearning. The abstract and the introduction now use this value, and Section 4.2 reports both. The same output scored as True/False claims in the same all-or-nothing way retains 28.7%, and partial credit adds 3.4 points. The step the reviewer attributes to metric and format is therefore mostly the question format. The new appendix "Matched Comparison of the Two Readouts" reports these values.

5. No validity evidence for SK.

  • (a) Predictive validity, and Q1. Response: We agree. The submitted version has no such test, and "distinguishes among methods" was asserted. The revision adds a new appendix, "Suppressed Knowledge and Recovery", and a short Section 4.8. At the level of methods, none of SK, the internal score or the external score correlates with LLM-GAT's published tampering or input-space attacks (eight methods, all |ρ| ≤ 0.52), and we report this as a null result. This test is weak: it has 8 methods, and after their strongest tampering attack 7 of them return to an accuracy of 0.57 to 0.71 (TAR stays at 0.36). We therefore test single questions. For each model, we take the questions the unlearned model answers wrongly and its base model answered correctly, and correlate each question's internal score with whether an attack recovers it. We ran two attacks with their authors' code on 18 unlearned models. The relearning attack of Fan et al. (ICML 2025) fine-tunes on documents and never sees a benchmark question. A higher internal score predicts recovery in 13 of the 18 models. In 4 of the 5 exceptions the attack does not succeed (accuracy changes by at most 0.05). The Retraining on T (RTT) attack of Deeb & Roger (2024) raises the accuracy of all 18 models. A higher internal score predicts recovery in 17 of them (Spearman ρ 0.17 to 0.40).
  • (b) Q2: equal-accuracy checkpoints and downstream risk. Response: The same appendix tests this. Within a model the readout does track risk, as in (a): it tells which questions an attack recovers. Across models, our data do not allow this test (paragraph "Checkpoint level"). The 18 models differ in their external score (0.46 to 0.64), so groups with a similar score and the same base model are small, and the largest has seven models, nearly the same set as the eight methods in (a), where we find no correlation.
  • (c) The abstract's claim. Response: We agree, and the sentence is rewritten. The probe is supervised and we do not interpret its weights, so "interpretable" is removed. "Should be evaluated through internal approaches" is replaced by what the evidence supports: internal evidence complements behavioral evaluation. On extraction, point (a) above gives the test we added, with two relearning attacks. In the abstract, the two sentences before the rewritten one no longer say "consistently" and "substantial".

6. Minor points.

  • (a) Line 276. Response: We agree. The sentence now reads "All internal scores are out-of-fold (Section 3.1): no question is used to fit, or to select the layer and regularization of, the probe that scores it."
  • (b) Line 426. Response: We agree. The sentence now reads "A similar internal-external separation appears on WMDP-Cyber." We made the same correction where Sections 4.5 and 4.6 summarize these results.
  • (c) Method order. Response: We agree. Figures 2 and 3 now list the methods in the order of Table 1 and Section 4.1 (GradDiff, RMU, RMU-LAT, RepNoise, ELM, RR, TAR, PB&J), as do all other figures and tables.

We thank the reviewer again for the questions, which led to the new controls and analyses above. We believe these revisions address the concerns raised. We hope the reviewer will consider strengthening their support in light of them.

Thank you for the clarifications (although at the end of the rebuttal period). I lowered the rating to 1, because you had not responded to any of the reviews, in order to help the AC. Overall, I think the paper is rushed and needs more work, so I will maintain my original rating of 2.

Thank you for reading our responses and for restoring the original score. We submitted our responses late in the discussion period because we addressed the review with new experiments, and performing and adding them to the paper took time. If any points from your review still require further work after the revision, we would be grateful to know which ones. If the main concerns are resolved, we hope you will consider strengthening your support for the paper.

Reviewer_Q7cBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Knowledge Lens, a framework for examining the gap between external behavior and internal knowledge representations in LLMs after unlearning. It converts multiple-choice questions into True/False tasks, computes internal and external scores, and measures the gap between them. Experiments cover several unlearning methods, training checkpoints, and model architectures. The results show that answer-related information remains in the model's internal representations even when its accuracy drops to near chance.

Critical strengths
  1. The paper examines the difference between behavioral forgetting and internal knowledge erasure, offering a useful perspective on evaluating the effectiveness of machine unlearning.
  2. The authors evaluate multiple unlearning methods, training checkpoints, and model architectures, and extend their experiments to WMDP-Cyber.
Critical weaknesses
  1. The comparison between internal and external evidence may be influenced by the different readout methods. KintK_{\mathrm{int}} is obtained using a supervised probe, whereas KextK_{\mathrm{ext}} comes directly from the model's own output head. It is therefore unclear how much of the observed gap reflects knowledge suppression and how much is due to the additional supervised training.
  2. The difference from previous probing results is not directly validated. Section 4.5 notes that the probes used by Li et al. (2024) failed to recover some unlearned knowledge. The authors mainly attribute the different findings to differences in evaluation protocols, but do not provide a direct comparison under the same conditions. It is therefore difficult to determine which factors account for the difference.
  3. The generalizability of the method remains unclear. Although the authors evaluate multiple unlearning methods and model architectures across WMDP-Bio and WMDP-Cyber, all experiments use four-choice questions. There is little evidence that the method would generalize to other question-answering tasks.
Questions for authors

Table 1 would benefit from including the corresponding results for the base model, allowing a more direct comparison. The paper analyzes changes in internal and external scores across checkpoints from different unlearning methods. However, the paper provides limited information about how these checkpoints were trained and selected. More details would help readers understand the unlearning process and assess the comparability of results across methods.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Q7cB (1/2)

We thank the reviewer for the review and for seeing in the paper "a useful perspective on evaluating the effectiveness of machine unlearning".

1. The gap may reflect the different readouts. Response: We agree that part of the gap comes from the readout, and the revision now measures it in two ways: a probe that sees only the answer, applied to both readouts, and the WMDP-Bio Robust subset, which excludes questions answerable from the answer choices. The answer-only probe reaches 0.66–0.69 in every model, including the base model, and it scores higher than the output on the same claims. This part of the gap is not necessarily suppression, and Section 4.2 now says so. To measure what remains, we subtract the answer-only score from both readouts and compare the remainder with the base model's. The probe keeps 27–69% for GradDiff, TAR and ELM, more than the output does, and at most 17% for the other five methods, on both question sets. The difference between the two readouts is significant for all three, on both question sets. The same holds on Zephyr-7B: eight of the nine OPTML models, all GradDiff- or NPO-type, keep significantly more in the probe (72–113%) than in the output (−19% to 81%), while for the WMDP RMU models the probe keeps at most 12%. The new appendix "Controls for the Supervised Internal Readout" reports these controls for every model, and also shows that the same probing procedure, trained with random labels, stays at chance, so the probe's score is not an artifact of the procedure.

2. No comparison with Li et al. (2024) under the same conditions. Response: We agree. Section 4.5 attributed the difference to the evaluation protocol without testing it. We have now tested it on the two checkpoints Li et al.'s statement covers, Zephyr-7B and Yi-34B, changing one factor at a time on the same questions and splits. A four-way probe trained with their protocol on the multiple-choice prompt agrees with their result, retaining 18.7% (Zephyr) and 21.3% (Yi) of the base model's above-chance accuracy. Changing the probe's target raises this to 30.0% and 25.9%, and presenting one candidate at a time as a claim raises it to 56.3% and 46.8%. Adding our chat template and system message changes this only to 58.6% and 47.9%. The claim format is therefore the main factor. On these two checkpoints, the evidence our probe finds comes mostly from the answer choices (point 1). Section 4.5 now reports the comparison, and a new appendix, "Comparison with the WMDP Linear Probe", gives the factor-by-factor table.

3. Generalization beyond four-choice questions. Response: We agree that the submitted version evaluated only four-choice WMDP questions. A new section, "Extension to Another Benchmark", and its appendix, "Knowledge Lens on TOFU", apply the lens unchanged to TOFU: questions about fictitious authors, each with one correct answer and five distractors, on Llama-3.2-1B models released by OpenUnlearning. Every candidate is scored as an independent claim, so only the chance level changes. Three unlearned checkpoints, obtained with GradDiff, NPO and RMU, lower the external score substantially. On the questions where training on the authors measurably helps, they show internal-external gaps of 0.22 to 0.36, while the thirteen other unlearned checkpoints we evaluated stay between 0.04 and 0.14. TOFU's synthetic candidates also let a model never trained on the authors rank the correct answer first in 42% of questions, against 17% chance, so the appendix states that TOFU establishes the gap but not how much of the internal evidence is retained knowledge. Open-ended generation, with no candidates to score, remains outside the framework and is named in the conclusion.

(Continued in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer Q7cB (2/2)

4. Questions.

  • (a) The base model in Table 1. Response: We agree. Table 1 now has the base model as its first column. After unlearning, the forgotten share (10.8–15.2%) stays close to the base model's 11.3%, while the suppressed share rises from 8.9% to 21.8–31.3%. Per question, 21.1–32.8% of the questions known to the base model become suppressed and only 5.4–12.0% become forgotten. Section 4.4 now reports these comparisons, so suppression is stated as a rise from the base model's level rather than as the post-unlearning share alone.
  • (b) How the checkpoints were trained and selected. Response: We agree. We train no models: every unlearned model is a public release used as fixed weights. A new appendix, "Unlearned Models: Training and Selection", now summarizes, from the LLM-GAT paper (Che et al., 2025), how the eight main methods were trained and checkpointed, and describes the other suites, including how we chose the TOFU configurations. Since training length is published only for TAR and PB&J, a checkpoint index cannot be assumed to mark the same training budget across methods, so we compare methods only at their final checkpoint. Section 4.1 now states this. We also corrected Section 4.2, which called Che et al.'s reported accuracies "final", although they are for each method's best-scoring checkpoint.

We thank the reviewer again for the questions, which led to the answer-only control, the comparison with Li et al. under the same conditions and the extension to TOFU. We believe these revisions address the initial concerns, and we hope the reviewer considers strengthening their support to reflect them.

Reviewer_DUBmOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a framework (Knowledge Lens) to study how deeply targeted knowledge is removed from an unlearned model. Focusing on multiple-choice unlearning benchmarks (WMDP), the authors convert each question-option pair into a True/False verification claim and evaluate whether the correct answer can still be predicted from the model's internal hidden states at the final token. They compare this internal linear probe score against the external True/False logit margin by computing how often the correct answer outranks the incorrect distractors. They find that deep unlearning—where the model has no internal bias toward the correct answer over incorrect distractors—is not satisfied across existing unlearning methods, even when external metrics (such as final-layer logits or benchmark accuracy) suggest the knowledge is gone

Critical strengths

The paper is well-written, legitimate, and very easy to follow and digest.The experimental details are clear—specifically, the matched definitions of the pairwise scores KintK_{\text{int}} and KextK_{\text{ext}} are clean and well-motivated. The methodology is solid, and the empirical study is extensive, covering multiple unlearning methods, trajectories, model architectures, and domains using publicly released checkpoints. The trajectory analysis showing that early unlearning checkpoints account for most of the drop in internal knowledge (KintK_{\text{int}}), while later steps mainly suppress external outputs (KextK_{\text{ext}}), is an interesting empirical observation

Critical weaknesses

The methodological and conceptual novelty of this paper is limited.The core takeaway—that current unlearning methods only achieve shallow behavioral suppression rather than deep knowledge erasure—is already well-established in the prior unlearning literature, making the overall findings expected rather than surprising. The technical approach of comparing output logits against linear probes trained on final-token internal activations is standard and has been widely used in LLM interpretability, truthfulness probing, and hallucination literature (including the exact pairwise formulation adapted from Gekhman et al., 2025).

Questions for authors

Quick question: I know the WMDP questions are different from each other, but could there be some bias since we train the internal classifier on WMDP questions? It doesn't have to be trained on them, right? I mean, it could be trained on any True/False dataset.

Which layers are actually driving KintK_{\text{int}}, and could early layers just be capturing shallow associations? The paper selects the single best layer l∗l^*. Because the prompt feeds question in a simple True/False verification format, couldn't early layers—where unlearning interventions might not have acted and where complex patterns between input tokens is not yet extracted—still separate the correct answer from distractors simply via basic lexical/co-occurrence features?

What is the practical implication if the knowledge is only decodable in early layers but erased/scrambled in later layers? Later layers are what ultimately govern final token decoding. If the target knowledge is successfully destroyed in the middle/later layers before reaching the output logits, does having a high linear probe score in an early layer actually pose a practical risk (beyond white-box activation probing), and how does KintK_{\text{int}} behave in the final layers right before the output head?

1 comment
AuthorsOpenReview ↗

Response to Reviewer DUBm

We thank the reviewer for the review and for finding the paper "very easy to follow" and the empirical study "extensive".

Is the finding new, given that shallow behavioral suppression is known and comparing logits with probes is standard? Response: We agree that the method builds on standard tools. Recovery of unlearned knowledge is well documented, and linear probing is a standard tool. The related work now states what we take from Gekhman et al. (2025): the pairwise score and both readouts. They study this gap outside unlearning, and in their main setup the probe and the output read different prompts. The finding is also not uniformly expected. The WMDP authors report that linear probes cannot recover the unlearned knowledge from their RMU checkpoints. On the same checkpoints, our readout finds about half of it (Section 4.5). A new appendix, "Comparison with the WMDP Linear Probe", changes their protocol into ours, one factor at a time, to explore this difference. The revision also adds a test we did not find in prior work: in most unlearned models, the internal score predicts which questions two relearning attacks recover, as the practical risk answer below describes.

The probe is trained on WMDP questions. Why not a generic True/False dataset? Response: We ran the suggested test, and it is in a new appendix, "Probes Trained Outside WMDP-Bio". The probe is trained on a dataset with no WMDP-Bio questions (non-biology MMLU questions, biology MMLU questions, or Geometry of Truth statements) and applied unchanged to WMDP-Bio. In the base model, all three probes score 0.77 to 0.80, close to the probe trained on WMDP-Bio (0.81), so before unlearning, the WMDP-Bio labels add little. After unlearning, the result depends on the method: a probe trained without WMDP-Bio labels scores significantly above the output for two of the eight methods, and for 10 of the 12 other unlearned models. On the bias itself, the appendix "Controls for the Supervised Internal Readout" measures what the WMDP-trained probe gains from the answer text alone, and Section 4.2 reports both checks.

Which layers drive the internal score, and could early layers capture shallow features? Response: We agree that the paper did not report the selected layers, and a new appendix, "Layer Profile of the Internal Readout", now lists them for every model. The reviewer's concern about early layers is correct. In layers 1 to 8 of Llama-3-8B-Instruct, the probe scores about the same with and without the question (a difference of at most 0.03), so its score there can be reached from the answer text alone. This matters where the probe selects those layers: for RMU, RMU-LAT, RepNoise and PB&J (layers 2 to 7) and for the three WMDP RMU models. GradDiff, TAR, and ELM select layers 13 to 21, and the nine OPTML models layers 15 to 27, where the question still adds to the score up to the last layer. The difference between the internal and external scores does not come from selecting an early layer: fixed at any layer from 9 to 32, the probe still scores higher than the output for every method.

If the knowledge is decodable only in early layers, does it matter, what is the practical risk, and what happens right before the output head? Response: The previous point answers the first part. On practical risk, a new appendix, "Suppressed Knowledge and Recovery", applies two relearning attacks with their authors' code. They recover answers in the model's output, without reading its hidden states. Within most unlearned models, the questions with a higher internal score are the ones the attacks are more likely to recover (13 of 18 models under the attack of Fan et al., 17 of 18 under that of Deeb & Roger. Two models are too large for these attacks). Right before the output head, at the last layer, the question adds nothing to the probe's score for 8 of the 20 unlearned models, as the reviewer describes. Unlike them, for the other 12, it still adds to the score.

We thank the reviewer again for the questions, which led to the probes trained outside WMDP-Bio, the layer profile and the recovery tests above. We believe these revisions address the concerns raised, and we hope the reviewer will consider strengthening their support in light of them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.