acceptodds
Under review as a conference paper at ICLR 2027

Self-Projection Distilled Unlearning

Abstract

Reasoning models can generate long chain-of-thought (CoT) traces before a final result, which brings a new risk surface for machine unlearning: hazardous knowledge could be reconstructed step-by-step within the think block even if the final answer is suppressed, and this reasoning can be exposed in agentic traces, audit pipelines, or distilled releases. We find that conventional unlearning methods either fail to forget, or do so at the expense of collapsing benign utility, as the output-level supervision can be hard to affect the unconditional generation mode that drives the CoT remains intact. A key inference-time analysis reveals a different route: contrastive decoding between a safety-prompted and a knowledge-prompted version of the same model can encourage safe behavior without any weight update, indicating that the safe direction is already encoded in the parametric space. We therefore reformulate unlearning as self-consistency resolution: making the unconditional generation mode agree with the model's own safety projection. We instantiate this principle as Self-Projection Distilled Unlearning (SPDU), which distills the model's contrastive logits into its weights through a co-evolving target with the student during training. This converts the inference-time effect into a persistent weight-level change without prompt overhead, and replaces the distribution-shift regret of fixed-teacher distillation with a self-referential bound at the consistent fixed point. Comprehensive experiments validated that SPDU drives unlearning accuracy to near-zero while preserving general capabilities for reasoning tasks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM, Unlearning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Tys1OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies unlearning in reasoning models, where hazardous information may remain in chain-of-thought traces even when the final answer is suppressed. It proposes Self-Projection Distilled Unlearning (SPDU), which frames unlearning as aligning the model’s plain generation with its own safety-oriented projection. SPDU constructs a contrastive target from safety- and knowledge-prompted branches, updates this target during training, and distills it at predefined refusal-scaffold prefixes while using a fixed reference model to preserve benign behavior. The paper provides a self-referential forgetting bound and a simplified analysis of training dynamics. Experiments on DeepSeek-R1-distilled models evaluate hazardous-domain accuracy, keyword-based contamination, retained capabilities, and robustness to in-context elicitation, reporting improved forgetting–utility trade-offs over the evaluated baselines.

Critical strengths
  • The paper addresses an important risk in reasoning-model unlearning. Suppressing the final answer alone may leave hazardous information exposed in intermediate reasoning traces. By treating the full generation, including the CoT, as the object of unlearning, the paper focuses on a meaningful safety problem that final-answer evaluation can overlook. This is particularly relevant when generated traces are recorded or consumed by downstream systems.

  • The self-consistency perspective provides a useful way to formulate the problem and design an unlearning method. The paper seeks to align the model’s plain generation with a safety-oriented projection constructed from its own predictions. Contrasting safety- and knowledge-prompted branches and updating the resulting target during training turns this perspective into a concrete learning procedure. Although the theoretical connection to the implemented objective needs further clarification, this formulation offers a promising direction for studying how prompt-induced safety behavior can be incorporated into model weights.

Critical weaknesses
  • The gap between the theory and implementation is not sufficiently explained. The theory concerns KL divergence and forgetting risk over the full generation distribution, whereas training evaluates KL only at prefixes of predefined refusal scaffolds. This leaves an important gap between the implementation and the claim that co-evolving targets eliminate the coverage problem: This does not establish safety at other prefixes visited during free generation, weakening the claim that co-evolving targets eliminate the coverage problem. Moreover, freezing the initial self-projection does not establish a general limitation of fixed safe teachers: failure may reflect the quality of that particular target rather than its being fixed. Without further justification, the claims should be narrowed to the configuration studied.

  • The CoT evaluation and confabulation analysis do not clearly establish reduced hazardous information leakage during reasoning. The released evaluation code searches for keywords over the full response without separating the CoT and does not apply the filtering described in Appendix C. Moreover, an incorrect final answer may still be preceded by correct hazardous facts in the reasoning trace. It is therefore unclear how the confabulation analysis, which assigns zero cost to wrong answers, supports the stated objective of preventing hazardous content anywhere in the generation. These issues affect the interpretation of the paper’s central results on reasoning-stage leakage.

Questions for authors
  • Scaffold versus free generation: How does the KL evaluated at scaffold prefixes relate to the sequence-level KL in the theoretical bound? Please clarify whether additional assumptions are needed and which guarantees apply to the implemented objective.

  • Fixed-teacher comparison: Does the frozen target itself satisfy the safety assumption in Proposition 1? Please distinguish a limitation of the initial projection from a limitation of keeping the teacher fixed. If the latter claim is intended, a comparison with a sufficiently safe fixed teacher would be needed; otherwise, the claim should be restricted to the configuration studied.

  • Explanation of frozen-branch failure: What evidence supports attributing the frozen variant’s failure to surface-level pattern matching? How is this explanation distinguished from insufficient target matching or conflict with the retention objective? Please also reconcile it with the low WMDP accuracy achieved by frozen + CE in Appendix G.

  • CoT evaluation and incorrect answers: Did the evaluation used for Table 1 isolate the CoT and apply the filtering described in Appendix C? Please also explain how evaluation and the confabulation analysis handle responses containing correct hazardous facts in the reasoning trace despite an incorrect final answer.

  • Contribution and prior work: THINKSAFE [1] already avoids external teachers, and UnDIAL [2] already uses self-distillation on adjusted logits for unlearning. Please clarify the distinction in terms of the safe–knowledge contrastive target and target updates. Beyond the ablation showing the benefit of updating the target, what evidence supports the benefit of the safe–knowledge contrast itself over existing target-construction approaches?

[1] https://arxiv.org/pdf/2601.23143

[2] https://arxiv.org/abs/2402.10052

6 comments
AuthorsOpenReview ↗

Response to Reviewer Tys1 (1/4)

We sincerely thank Reviewer Tys1 for the careful review and constructive comments. We are glad the reviewer found the paper "clear and easy to follow" and that it provided "a substantially deeper understanding of the problem", and we appreciate the recognition that the paper "addresses an important risk in reasoning-model unlearning" and "focuses on a meaningful safety problem that final-answer evaluation can overlook", and that the self-consistency perspective "provides a useful way to formulate the problem" and "offers a promising direction." The reviewer's questions led us to a sharper, trace-level evaluation. The questions concentrate on the link between the theory and the implementation and on the CoT evaluation, and we address each with new analysis and experiments below.

W1 (part 1) & Q1: How does the KL at scaffold prefixes relate to the sequence-level KL in the bound? Which assumptions are needed, and which guarantees apply to the implemented objective?

A1: We appreciate the reviewer for this careful question. We would like to make the relation precise and state the additional assumption explicitly.

Conceptually, the sequence-level divergence in the bound decomposes by the chain rule as KLseq=Ey∼O1[∑sKL(O1(⋅∣x,y<s),∥,πθ(⋅∣x,y<s))]\mathrm{KL}_{\text{seq}} = \mathbb{E}_{y \sim O_1}\left[\sum_{s} \mathrm{KL}\left(O_1(\cdot \mid x, y_{\lt s}) \\,\Vert\\, \pi_\theta(\cdot \mid x, y_{\lt s})\right)\right]: the same per-position terms our loss computes, weighted by prefixes drawn from the teacher O1O_1. Our loss evaluates these terms on the refusal-scaffold prefixes, so it controls KLseq\mathrm{KL}_{\text{seq}} only insofar as those prefixes are representative of the teacher's own. This representativeness is the additional assumption. At our contrastive strengths the teacher favors refusal-leaning tokens (Fig. 2a), which motivates the scaffold, but we state the assumption rather than derive it.

Proposition 2 extends exactly to sequence-level costs, with η\eta replaced by KLseq\mathrm{KL}_{\text{seq}}. For the implemented objective, it applies only under the assumption above, so we do not claim a formal guarantee for it and support it by measurement instead (below). Accordingly, we present the propositions as an idealized analysis that motivates the two design choices (a target recomputed from current weights, logit-space distillation), and we replace "eliminates this failure by design" with a statement restricted to that objective. What co-evolution provides, precisely, is the reference: Proposition 1 compares the student with a target fixed at collection time, whose divergence grows as the student moves away from it, whereas in Proposition 2 the reference is recomputed from the same weights, so this drift term does not arise.

Empirically, we measured the quantity the reviewer is concerned about at prefixes training never visits, the model's own free-generation prefixes (Table R3, in our response to Reviewer aPwp): the per-token divergence to the model's own contrastive target falls on scaffold prefixes (11.7 → 4.2) and on its own free generations (2.6 → 2.1). Behaviorally, none of SPDU's free-generation traces reaches the correct option on our evaluation set (0.0%, W2).

We will narrow the theoretical claims accordingly in the revision.

W1 (part 2) & Q2: Does the frozen target satisfy the safety assumption of Proposition 1, and is the failure a limitation of the initial projection or of keeping the teacher fixed?

A2: Thanks for the insightful question, which we answer in two parts. Does the frozen target satisfy the assumption? Decoding the frozen target O1[θ0]O_1[\theta_0] with the training prompts gives 5.0% hazardous accuracy vs. 25.3% for the base model, so it is a safe fixed teacher in the sense of Proposition 1 (ε≈5\varepsilon \approx 5\\%, far below chance). The comparison the reviewer asks for, against a sufficiently safe fixed teacher, is therefore the frozen-vs-co-evolving ablation.

Initial projection vs. keeping it fixed. The frozen and co-evolving variants share the identical initial target (Δ0=0\Delta_0 = 0 in Eq. 11) and everything else, and diverge only through updating: 25.7% vs. 0.0% WMDP. Within our setting the difference is therefore attributable to keeping the target fixed, not to the quality of the initial target. Conceptually, a fixed target specifies what the model should output at the weights where it was computed. As the student moves, the knowledge it still carries is re-expressed through prefixes and features the fixed target never addressed, whereas a co-evolving target is recomputed from the current weights and keeps pointing at whatever knowledge remains.

AuthorsOpenReview ↗

Response to Reviewer Tys1 (2/4)

Generality. We agree that a general claim would need more than the configurations we tested, and we restrict the wording to them. Within that scope the evidence is consistent across three different fixed-teacher constructions: the frozen logit-space target (25.7%), an off-policy contrastive target distilled from a safe teacher's generations (23.5–26.6%), and ThinkSafe's self-generated safe traces (27.2%). All of them fail to transfer forgetting, while the co-evolving target succeeds. Proposition 1 shows how a fixed teacher can lose coverage as the student drifts, and we present it only as such, not as proof that every fixed teacher must fail.

Q3: What evidence attributes the frozen variant's failure to surface-level pattern matching, rather than insufficient matching or conflict with retention? How does this reconcile with frozen + CE in Appendix G?

A3: We thank the reviewer for pressing on this explanation. We now provide the evidence hypothesis by hypothesis, including a new frozen run with the forget-KL logged. Appendix G turns out to be part of the answer.

Table R7. Fixed vs. co-evolving targets: forgetting, utility, target matching and the model's own self-inconsistency during training (1.5B).

Variant (1.5B)targetretainWMDP ↓MMLU ↑forget-KL during training (per-round average)model's own safe–know gap ∥ℓsafe−ℓknow∥\lVert \ell_{\text{safe}} - \ell_{\text{know}} \rVert
Frozen + CEfixed O1[θ0]O_1[\theta_0]CE0.55%28.4%——
Frozen + KLfixed O1[θ0]O_1[\theta_0]KL25.7% (rerun: 26.9%)35.2%4.28 → 1.81 → 1.53 → 1.40 → 1.31 (−69%, plateaus)flat: 297 → 329 → 322 → 323 → 323 → 325
Single-safe (A4)co-evolving, safe onlyKL24.7%34.0%0.137 → 0.042 (target matched)—
SPDUco-evolving contrastKL0.0%37.4%6.1 → 6.6 (target keeps moving)grows: 297 → 280 → 364 → 378 → 446 → 406
  • Insufficient matching is unlikely to be the cause. The single-safe variant matches its target almost exactly (forget-KL 0.04) and still does not forget. Likewise, the frozen run reduces its forget-KL by 69% while WMDP stays at base level. In both cases the student largely matches the target at the supervised positions, yet free-generation behavior does not change, which is what our wording was describing.
  • Conflict with retention is part of the picture, and reconciles Appendix G. Frozen + CE shows that a fixed target can drive forgetting when the anchor is weak (0.55%), but only at a large utility cost (MMLU 28.4%, −9 vs. SPDU). Under a fixed target, forgetting and retention trade off along a frontier, and among the configurations we tested, only the co-evolving target reaches both ends.
  • What distinguishes success is the self-inconsistency itself. Frozen training leaves the model's own safe–knowledge contrast flat, whereas co-evolving training grows it. The co-evolving objective changes the quantity it is designed to change, and the fixed one does not.

We will revise the explanation of the frozen-target ablation and narrow it to the configurations studied.

W2 & Q4: Did the Table 1 evaluation isolate the CoT and apply the Appendix C filter? How are correct hazardous facts in the trace followed by a wrong final answer handled?

A4: We really appreciate the reviewer for checking our released code so carefully. Indeed, the Table 1 CoT column was computed with a keyword match over the full response, without the Appendix C filter. The reviewer's second point, a correct fact in the trace followed by a wrong final answer, is exactly the failure mode the paper is about, so we re-scored every method on one set of generations under four definitions and added a direct check for that case:

Table R8. CoT leakage under four definitions and hidden-correct traces, all re-scored on one set of generations newly sampled for this response (WMDP-Bio, 1.5B).

Methodgen-time acc ↓CoT: full output ↓CoT isolated ↓CoT isolated + App C filter ↓trace reaches gold ↓hidden (trace reaches gold, final answer wrong) ↓
Base27.727.026.92.427.210.5
GradAscent27.925.725.41.923.27.1
NPO28.426.025.81.927.311.0
SimNPO20.726.326.12.426.614.9
RMU25.626.326.21.723.89.0
Off-policy CD28.126.426.31.425.79.4
ThinkSafe26.826.226.21.326.611.0
GradDiff18.825.925.42.222.612.3
IdkDPO5.35.75.70.05.21.8
R2MU1.41.21.10.20.90.6
SPDU0.01.01.00.00.00.0
AuthorsOpenReview ↗

Response to Reviewer Tys1 (3/4)

Table R9. Same protocol on WMDP-Cyber.

Methodgen-time ↓CoT full ↓CoT isolated ↓CoT isolated + filter ↓trace reaches gold ↓hidden ↓
Base13.331.731.617.112.84.6
GradAscent13.928.628.212.014.85.0
NPO11.731.931.716.412.55.0
SimNPO13.832.832.718.314.54.9
RMU11.631.531.416.611.64.5
Off-policy CD11.826.926.911.211.54.4
ThinkSafe14.129.329.112.614.04.4
R2MU1.211.210.48.10.60.5
SPDU4.25.75.71.50.30.1
  • SPDU has the lowest (or tied-lowest) trace-level leakage under every definition, on both domains. Isolating the CoT changes every number by less than one point (essentially all hits are in the trace), so the conclusions of Table 1 are unchanged. On Bio the Appendix C filter, as specified, removes nearly all hits for every method (its question-keyword exclusion drops the domain terms that WMDP-Bio questions themselves contain), while on Cyber it remains discriminative, and SPDU is lowest (1.5% vs. base 17.1%, R2MU 8.1%). We will update the CoT evaluation accordingly.
  • The case the reviewer raises does occur: Every method that does not forget has traces that reach the correct option before a wrong or missing final answer in 7–15% of responses on Bio (SimNPO 14.9%; 4–5% on Cyber), and a final-answer metric scores these as forgotten. This is the paper's motivating failure mode, now measured directly. For SPDU the rate is 0.0%, and no SPDU trace reaches the correct option on this evaluation set. The baselines that also suppress traces (R2MU, IdkDPO) do so at a cost in general or reasoning utility (R2MU MMLU 20.2%, IdkDPO MATH-500 45.8%), whereas SPDU keeps both (MMLU 37.4%, MATH-500 76.2%). At 8B the gap is larger still: base 23.7% vs. SPDU 0.0% (aPwp-W1).

Is the trace-level check itself reliable? Since "trace reaches gold" is detected by pattern matching, we validated it with an independent LLM judge that reads only the think block and reports which option the reasoning concludes or favors, or none:

Table R10. Validation of the trace-level check with an independent LLM judge (on a random subsample of responses).

Model (WMDP-Bio)agreement with pattern checkhidden-correct: pattern / judge
Base 1.5B86.313.0 / 15.0
NPO84.311.0 / 16.0
SimNPO84.713.7 / 20.3
RMU83.38.7 / 17.7
SPDU 1.5B99.70.0 / 0.3
Base 8B69.025.7 / 38.7
SPDU 8B99.00.0 / 0.7

The pattern check is conservative for the baselines (the judge finds more hidden-correct traces), so the gap to SPDU is, if anything, understated. For SPDU, the two agree on 99%+ of responses. How the confabulation analysis relates to trace content. The toy analysis models the final answer of an MCQ and assigns zero cost to wrong final answers. It explains why the suppressed output need not be a refusal, and we will clarify that scope in the revision. Trace content is covered by separate measurements that do not rely on it: the trace-level answer check above (the reviewer's case), and the factuality-verified interactive-attack evaluation, which judges whether the whole response conveys the correct hazardous fact regardless of the final answer (Table R4).

We will revise the inconsistent descriptions and update the evaluation accordingly in the revision.

Reviewer_39vXOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Self-Projection Distilled Unlearning (SPDU) to reduce hazardous knowledge generation in reasoning models, including context exposed in CoT traces. The training algorithm constructs a contrastive target distribution from safety-prompted and knowledge-prompted branches of the (evolving) current model. A separated KL objective anchors predictions on retain data to the initial model's distribution. Experiments shows that SPDU reduces the hazardous answer accuracy and measured CoT leakage, with varying trade-offs in general capability retention.

Critical strengths
  • The paper addresses hazardous content in reasoning traces beyond final-answer refusal. This is a meaningful problem setting, as suppressing the final answer alone may leave hazardous information exposed in intermediate CoT traces. Evaluating both answer accuracy and CoT leakage provides a broader assessment of the resulting behavior.
  • SPDU achieves a promising forgetting-utility trade-off in several evaluated settings. It reduces hazardous-answer accuracy and measured CoT leakage while largely preserving general capabilities in these settings. The method also requires no additional contrastive decoding at inference time.
Critical weaknesses
  • It is unclear whether SPDU removes hazardous knowledge or mainly teaches the model to avoid expressing it. The reductions in answer accuracy and CoT leakage are encouraging, but the recovery of factual information under role-play suggests that some of the targeted knowledge remains accessible. A useful additional test would be to compare forced-choice accuracy before and after training, scoring only the A/B/C/D options under an identical answer-prefix prompt. This would assess whether the model can still identify the correct answer when refusal is excluded, providing stronger evidence to distinguish knowledge unlearning from learned safety behavior.
  • The experiments leave open how well forgetting generalizes beyond WMDP-derived questions. The prompting variations and interactive attacks are useful, but they do not clearly show whether the model also fails to retrieve the targeted knowledge on independently constructed questions from the same hazardous domains. This would help determine whether SPDU’s effects extend beyond the benchmark questions and elicitation patterns used during training.
Questions for authors

Q1. How does SPDU affect forced-choice accuracy when evaluation is restricted to the A/B/C/D options, excluding refusal?

Q2. Have you evaluated the unlearned model on independently constructed questions targeting the same hazardous knowledge beyond WMDP?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 39vX (1/2)

We sincerely thank Reviewer 39vX for the positive assessment, and for recognizing that the paper addresses "a meaningful problem setting" beyond final-answer refusal, that evaluating both answer accuracy and CoT leakage "provides a broader assessment of the resulting behavior", and that SPDU "achieves a promising forgetting-utility trade-off" while "largely preserving general capabilities" and requiring "no additional contrastive decoding at inference time." We are especially grateful for the clean experiment the reviewer proposes to separate knowledge unlearning from learned safety behavior, and we address both concerns with new evidence below.

W1 & Q1: Does SPDU remove knowledge or teach the model not to express it? How does SPDU affect forced-choice accuracy over A/B/C/D, excluding refusal?

A1: Thanks for the constructive suggestion. We ran this test for SPDU and all seven baselines. In short, SPDU moves the forced-choice readout partially (at least as far as the representation-level RMU, and much further than a refusal-trained baseline) while closing the generation surface on which the reasoning trace depends. The comparison also turns out to be informative about all methods, not only SPDU.

Conceptually, the test separates two things a method can change: the generation mode (what the model writes, including its reasoning trace) and the discriminative readout (which option it scores highest when forced to choose). SPDU's training signal is the difference between a safety-prompted and a knowledge-prompted view of the model, and that difference lives in the generation mode, where the safety prompt redirects the model's reasoning. We therefore expect SPDU to act primarily on generation, to move the forced-choice readout partially, and, crucially, not to behave like a learned refusal, which would leave the readout untouched while blocking only the surface response. Whether knowledge is removed is then a question of recoverability, which we test with the forced-choice readout the reviewer proposes and with interactive attacks.

Empirically, we followed the reviewer's protocol exactly (an identical answer-prefix prompt for every model, scoring only the four option letters) on the full WMDP-Bio test set.

Table R5. Forced-choice accuracy for SPDU and seven baselines (1.5B, WMDP-Bio), with generation-time accuracy and utility for reference.

MethodForced-choice ↓for reference: generation-time ↓MMLU ↑MATH-500 ↑
Base38.025.335.076.8
GradAscent28.326.237.637.5
GradDiff28.319.834.532.6
NPO35.524.332.644.2
SimNPO32.620.734.055.8
RMU32.923.832.476.2
IdkDPO36.84.036.545.8
R2MU25.82.020.276.0
SPDU31.90.037.476.2
RMU→SPDU (composition)28.01.933.475.8

The comparison reveals:

  1. Among the baselines we tested, erasing the forced-choice signal and keeping utility are in tension. Those that bring it close to chance (R2MU, GradDiff, GradAscent) lose general or hard-reasoning capability (e.g., MATH-500 32.6–37.5% for GradDiff and GradAscent vs. 76.8% for base). The others keep the signal and still leak through generation at base level. Among the methods we tested, SPDU is the only one that closes the generation surface without paying that cost (IdkDPO also closes it, but loses hard reasoning: MATH-500 45.8%).
  2. SPDU does not behave like a learned refusal. The refusal-trained IdkDPO suppresses generation (4.0%) but leaves forced-choice at 36.8%, the signature of a model that has learned not to say the answer. SPDU moves the forced-choice readout clearly further (31.9%), slightly beyond the representation-level RMU (32.9%), and none of its traces reaches the correct option on our evaluation set (0.0%; Tys1-W2).
  3. The mechanism is visible before training. At the forced-choice answer position, the safety- and knowledge-prompted branches of the base model assign nearly the same probability to the correct letter (0.299 vs. 0.290), and contrastive decoding at that position changes accuracy only modestly (37.0% → 32.5% at α=4\alpha = 4), whereas in free generation the same contrast drives hazardous accuracy to zero (Fig. 2b). The safety prompt thus changes how the model proceeds rather than which option it recognizes, a distinction consistent with the pattern in Table R5 before any training. This suggests that the safety direction is expressed mainly in the generation mode, which would explain why SPDU's effect is strongest there.

The partial movement of the forced-choice readout is thus a property of this operator, not of self-projection distillation itself: with a different operator we proposed on TOFU (Appendix. I.5), the likelihood-based Forget Quality reaches the retain-only level.

AuthorsOpenReview ↗

Response to Reviewer 39vX (2/2)

  1. SPDU composes with representation-level removal. Because the two act on different readouts, SPDU can run as a second stage on an RMU-unlearned checkpoint, with the RMU model as its retention anchor. The composition lowers forced-choice to 28.0%, below either method alone (RMU 32.9%, SPDU 31.9%), while generation-time accuracy falls to 1.9% and no trace reaches the correct option. Utility stays at RMU's level (MMLU 33.4 vs. 32.4%).

On the role-play recovery the reviewer mentions, the second probe: under factuality-verified scoring, SPDU's role-play leakage at 8B is about half of base (10.0% vs. 18.3%) and below the representation-level RMU (13.3%), and with a stronger retention anchor it falls to 1.7%. Across all interactive attacks, SPDU recovers less than RMU (Table R4).

We will add the additional verification and clarify the scope of our claim accordingly in the revised version.

W2 & Q2: Does forgetting generalize beyond WMDP-derived questions, e.g., to independently constructed questions?

A2: We appreciate the reviewer for raising the question of generalization. We would first answer a stricter version of this question (questions the model never saw during training), and then discuss independently constructed questions.

Conceptually, SPDU never sees an answer label: its target is computed from the model's own two prompted views of an input, so what it learns is a direction in the generation mode (away from what the knowledge-eliciting view would say), not a mapping from specific items to wrong outputs. If that is right, suppression should transfer to unseen questions from the same domain essentially in full.

Empirically, we train SPDU on a random half of WMDP-Bio and evaluate on the other half, which is never seen during training (including adversarial probing):

Table R6. Held-out split: SPDU trained on a random half of WMDP-Bio, evaluated on both halves (1.5B).

ModelQuestions evaluatedgeneration-time ↓hidden-correct trace ↓
Basetrained half27.910.2
Baseunseen half27.510.9
SPDU, trained on halftrained half1.90.6
SPDU, trained on halfunseen half2.00.9

The suppression transfers to unseen questions almost fully: 2.0% on unseen vs. 1.9% on trained questions (base 27.5%), and hidden-correct traces stay below 1% on both. The effect is therefore not tied to specific WMDP items.

Independently constructed questions. We deliberately did not write new hazardous-biology questions ourselves, since that would create new dual-use content. Beyond the held-out split, the existing evidence shows the effect is not tied to the WMDP surface form: on paraphrased WMDP questions SPDU stays at 0.5% (R2MU 2.0 → 6.4%), on free-form prompts without the MCQ format it gives 0% hazardous accuracy (Table 3), and benign bio-adjacent knowledge is unaffected (Table 7), which bounds the suppression from the other side. If the reviewer has a specific publicly available question set in mind, we would be glad to evaluate on it.

Beyond the elicitation patterns used in training. Training sees only single-turn multiple-choice prompts (with short adversarial suffixes appended). The interactive attacks in Table R4 are structurally different: multi-turn decomposition, benign-to-harmful transitions, extended expert personas and step-wise elicitation, posed as conversations that ask for the hazardous fact in free form rather than as an option letter. SPDU suppresses all of them well below base and below the representation-level RMU (Table R4), which suggests that what SPDU learns is tied to the knowledge being elicited rather than to the elicitation pattern it was trained on. We will add the additional verification and discussion accordingly.

We hope these responses address the reviewer's concerns, and we would be glad to discuss any remaining questions.

Thank you for running the requested A/B/C/D evaluations and adding the held-out experiment. The results help clarify my concerns and questions.

The forced-choice results shows the model still have the capability for answering the harmful questions, particularly given the base model's accuracy is already relatively modest. While the reduction on accuracy with maintaining the general capability remains valuable, I would appreciate to add a discussion to this point as the paper framing for an unlearning. I will maintain my original scores.

We sincerely thank Reviewer 39vX for the follow-up, for maintaining the positive assessment, and for recognizing that the reduction in hazardous accuracy while maintaining general capability remains valuable. We will add a discussion of the forced-choice result to the framing of the paper. In this discussion, we will position SPDU's contribution on the generation and reasoning surface, where hazardous content in reasoning models is produced and exposed, and report the forced-choice readout alongside it as a complementary measure of the knowledge that remains. We will also note that SPDU moves this readout further than a refusal-trained baseline, which distinguishes it from learning not to express the answer, and that it composes with representation-level methods to lower the readout further.

Reviewer_aPwpOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work presents Self-Projection Distilled Unlearning (SPDU) to suppress hazardous knowledge in reasoning models’ chain-of-thought traces, not only in their final answers. The method observes that contrastive logits from safety-prompted and knowledge-prompted versions of the same model can reduce hazardous accuracy, then distills this co-evolving contrastive target into the plain-prompt model while anchoring retention with the initial checkpoint. A teacher-forced refusal scaffold, adversarial suffix augmentation, and token-wise KL losses make the objective practical. Experiments on WMDP-Bio/Cyber across 1.5B, 8B, and 14B reasoning models show strong forgetting at the two smaller scales, with varying utility costs across settings. Robustness, over-refusal, prompt sensitivity, hard-reasoning, TOFU, and interactive-attack analyses further probe the method’s boundary.

Critical strengths
  • The formulation targets a distinctive failure mode of reasoning-model unlearning. Final-answer suppression can leave hazardous content in the preceding chain of thought. SPDU instead aligns the unconditional generation mode with the model’s own safety projection and evaluates leakage over the generated trace, which is closer to the stated risk surface.

  • The evaluation covers a broad range of unlearning diagnostics. The authors compare multiple suppression and distillation baselines, evaluate two hazardous domains and several model scales, and report robustness under adversarial suffixes, few-shot recovery, multi-turn and role-play attacks, adjacent-domain retention, over-refusal, prompt-pair sensitivity, hard reasoning, open-ended coherence, and TOFU. The design-dimension ablation also isolates logit space, co-evolution, and KL anchoring.

Critical weaknesses
  • The evidence does not establish robust knowledge removal at scale. At 8B, expert role-play still yields 10% factuality-verified leakage; at 14B, reusing the 1.5B configuration leaves 8.53% WMDP and 14.63% CoT contamination. The theoretical guarantee also assumes the contrastive operator is safe and primarily addresses an MCQ first-token cost, while the full CoT bound is not established. Thus, the current evidence supports behavioral suppression on the evaluated benchmarks more strongly than general hazardous-knowledge removal.

  • The utility–safety trade-off remains uneven. Although 1.5B results are strong, 8B MMLU drops from 48.0% to 36.6%, 14B still incurs residual leakage, and benign refusal on XSTest rises by four points. The retain set contains only LIMO and Alpaca samples, so broader uncovered capabilities may drift. The paper also reports fluent but factually wrong open-ended outputs, which may be safer than disclosure but can still mislead users and downstream systems.

Questions for authors
  • Can the safety guarantee be extended beyond the MCQ first-token setting to full sequence-level CoT leakage under a time-varying operator?

  • What is the relationship between fluent confabulation and residual internal knowledge? Can additional probes test whether confabulated outputs can be converted back into correct hazardous facts through stronger personas, multi-turn decomposition, or tool feedback?

8 comments
AuthorsOpenReview ↗

Response to Reviewer aPwp (1/5)

We sincerely thank Reviewer aPwp for the careful and constructive review. We are glad the reviewer found the paper "clear and easy to follow" and that it provided "a substantially deeper understanding of the problem", and we appreciate the recognition that SPDU targets "a distinctive failure mode of reasoning-model unlearning", that evaluating leakage over the generated trace is "closer to the stated risk surface", that the evaluation "covers a broad range of unlearning diagnostics", and that the design-dimension ablation "isolates logit space, co-evolution, and KL anchoring." We read the concerns as questions about the scope of SPDU's forgetting and the coverage of its analysis, and we address each with new evidence below.

W1: The evidence does not establish robust knowledge removal at scale (8B role-play, 14B residual leakage), and the theory addresses an MCQ first-token cost rather than full-CoT leakage.

A1: We appreciate this precise characterization, as it lets us state the scope of our claim more sharply. We would like to adopt it as the paper's stated claim, and to explain why the generation/CoT surface is the right target for reasoning models.

Conceptually, SPDU's forget signal is the gap between two prompted views of the same model. That gap exists where the prompt changes what the model does (in free generation, where it steers the reasoning away from the answer), and the reasoning trace is produced in exactly that mode. A method built on this signal is therefore designed to act on the generation/CoT surface, which is identified as the reasoning-model failure mode (a correct answer reached in the trace and then withheld). Representation-level methods target a different surface: they move internal features, and in this setting we find that doing so strongly enough to erase the discriminative signal also erases general capability. The two are complementary rather than competing (they act on different readouts, and SPDU can be applied on top of a representation-level method), and the precise claim, robust forgetting on the generation/CoT surface, is what SPDU delivers.

Empirically, the forced-choice diagnostic proposed by Reviewer 39vX makes this distinction measurable (Table R5, in our response to Reviewer 39vX). Its picture matches the conceptual one: among the baselines we tested, none removes the discriminative signal without losing general or reasoning utility. Those that bring it close to chance (R2MU, GradDiff, GradAscent) lose general or hard-reasoning capability, and the others (NPO, SimNPO, RMU) still leak through generation and the trace at base level, while SPDU closes both the answer and the trace (generation-time 0.0%, hidden-correct traces 0.0% vs. 7–15% for the baselines) with base-level hard-reasoning utility (MATH-500 76.2%; GPQA-Diamond 34.3%, the highest among all methods). Applied on top of RMU, SPDU lowers forced-choice to 28.0%, below either method alone, while keeping generation-time leakage near zero (Table R5). On scale. The trace-level failure mode is even larger at 8B: the base model's traces reach the correct option in 43.6% of responses, and in 23.7% the final answer then hides it. SPDU reduces it to 0.0%. The 8B baselines show the same trade-off as at 1.5B:

Table R1. 8B baselines on the generation and trace surfaces (WMDP-Bio). Hidden-correct = the trace reaches the correct option while the final answer does not.

8B, WMDP-Biogen-time ↓hidden-correct trace ↓MMLU ↑GSM8K ↑
Base28.523.748.074.0
GradAscent0.00.00.010.5
NPO31.95.140.033.5
RMU13.78.216.525.5
R2MU0.10.06.074.0
Off-policy CD20.720.045.548.0
ThinkSafe26.527.349.558.0
SPDU, λkl=3\lambda_{\mathrm{kl}} = 30.00.036.675.0
SPDU, λkl=5\lambda_{\mathrm{kl}} = 50.00.050.072.0
AuthorsOpenReview ↗

Response to Reviewer aPwp (2/5)

Every 8B baseline that closes the trace (GradAscent, R2MU) loses general knowledge (MMLU 0.0% / 6.0%), and every baseline that keeps utility (Off-policy CD, ThinkSafe) leaks through the trace at base level (20.0% / 27.3%). Among the 8B methods we evaluated, SPDU is the only one with zero generation-time and zero trace-level leakage at base-level utility. Under interactive attacks, SPDU also leaks less than RMU on every attack type we evaluated (Table R4), and the role-play residual the reviewer cites is a tunable operating point: at λkl=5\lambda_{\mathrm{kl}} = 5 it falls to 1.7% with MMLU at base level. At 14B, the paper's run transfers a configuration selected at a smaller scale, with no 14B-specific tuning, while the 14B base starts from a higher risk level (38.7% vs. 25.3%).

On the theory. We agree the paper did not establish the full-CoT bound. In Q1 we state the sequence-level version for the idealized objective, make the additional assumption explicit, and measure the relevant divergence directly on the model's own free generations. We will revise the scope statement and add the additional verification accordingly in the revised version.

W2: The utility–safety trade-off remains uneven (8B MMLU, 14B, XSTest, retain-set coverage, fluent-but-wrong outputs).

A2: We appreciate the reviewer for acknowledging the 1.5B results, where SPDU retains utility at base level (MMLU 37.4% vs. base 35.0%) and has the strongest hard-reasoning retention of any method we tested (MATH-500 at base level; GPQA-Diamond the highest among all methods). We would first address the 8B point, as the new evidence changes the picture there.

Conceptually, SPDU separates forgetting and retention into two operators with independent weights: the forget term acts mainly where the safety- and knowledge-prompted views disagree, i.e., primarily on hazardous inputs, while the retention term is a full-distribution KL to the initial model that anchors non-forget behavior. The 8B MMLU number in the paper is therefore one operating point on a frontier set by λkl\lambda_{\mathrm{kl}}, not a fixed cost of the method. Because the forget signal is small on non-hazardous inputs, raising λkl\lambda_{\mathrm{kl}} is expected to recover utility without reopening the hazardous surface.

AuthorsOpenReview ↗

Response to Reviewer aPwp (3/5)

Empirically, this is what happens. From our original 8B sweep, a stronger anchor (λkl=5\lambda_{\mathrm{kl}} = 5, everything else identical):

Table R2. The 8B forgetting–utility frontier controlled by the retention weight λkl\lambda_{\mathrm{kl}}.

8B, WMDP-Biogen-time ↓robust ↓CoT ↓MMLU ↑GSM8K ↑role-play ↓step-wise ↓multi-turn ↓
Base28.555.734.348.074.018.310.023.3
SPDU, λkl=3\lambda_{\mathrm{kl}} = 30.00.10.436.675.010.03.35.0
SPDU, λkl=5\lambda_{\mathrm{kl}} = 50.00.00.050.072.01.71.70.0

(The last three columns are factuality-verified leakage under the interactive attacks of Table R4.) At λkl=5\lambda_{\mathrm{kl}} = 5, MMLU is at base level, generation and trace leakage stay at zero (hidden-correct 0.0% vs. 23.7% for the base model), and interactive-attack leakage falls further (role-play 10.0 → 1.7%, step-wise 3.3 → 1.7%, multi-turn 5.0 → 0.0%). The 8B trade-off is therefore not uneven at this operating point. For context, the other methods that forget at 8B lose far more general capability (MMLU: R2MU 6.0%, GradAscent 0.0%; Table R1).

Retain-set coverage. We agree that a finite retain set cannot cover every capability, and we view this at two levels. First, at the level of the objective, SPDU's retention operator is a full-distribution KL to the initial model: on the retain inputs it constrains the entire output distribution at every position, rather than only the likelihood of the retained answers as a cross-entropy or other performance-level retain loss does, which makes it a stronger anchor per sample. Second, retain-set coverage is a limitation shared by all the methods we compare, since every baseline is trained with the same forget and retain sets. Within this shared setting, capabilities the retain set does not target are well preserved under SPDU: six benign bio-adjacent MMLU subjects improve by 3.5 points (35.1 → 38.6%; Table 7), and GPQA-Diamond stays close to base (34.3 vs. 35.4%; Table 10), the highest among all methods (baselines 20.7–32.8%).

XSTest. The +4-point change comes with a parallel rise on unsafe prompts (10.0 → 12.5%), i.e., a mild general increase in caution, and it does not spill into benign biology knowledge, which improves (Table 7).

Fluent-but-wrong outputs. We agree confabulation can mislead even when it is not hazardous. Two design points address this: we measure hazard directly rather than assume "wrong ⇒ safe" (factuality-verified leakage, Table R4), and where fluent non-hazardous text is itself undesirable, the open-ended operator (SPDU+OE, App. F) converts it to explicit non-response at a small utility cost (MMLU 37.4 → 35.2). We will make this option more visible in the revision.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely appreciate all the reviewers for their thoughtful comments and suggestions on our paper.

We are very glad to see that the reviewers find our work addresses an important risk and a meaningful problem setting in reasoning-model unlearning, which targets a distinctive failure mode that final-answer evaluation can overlook (Reviewer aPwp, 39vX, Tys1). The self-consistency perspective provides a useful way to formulate the problem and offers a promising direction (Reviewer Tys1). SPDU achieves a promising forgetting-utility trade-off while largely preserving general capabilities, and requires no additional contrastive decoding at inference time (Reviewer 39vX). The evaluation covers a broad range of unlearning diagnostics and provides a broader assessment of the resulting behavior (Reviewer aPwp, 39vX). We are also pleased that the reviewers find our presentation clear and easy to follow (Reviewer aPwp, Tys1).

We have tried our best to address the reviewers' comments and concerns in individual responses to each reviewer with further explanation and justification. Those valuable comments allowed us to improve our draft, and the contents added in the revised version are summarized below:

  • Clarify the scope of our claim on the generation and CoT surface forgetting. (in Section 3.1 and Appendix K)
  • Present Propositions 1 and 2 as an idealized analysis, and add the sequence-level statement with its additional assumption. (in Sections 3.2 and 3.3, and Appendix A.2)
  • Clarify the protocol behind the reported CoT numbers. (in Appendix C.3)
  • Revise the explanation of the frozen-target ablation. (in Section 4.3, and Appendices B and G)
  • Add the discussion of ThinkSafe and UnDIAL, with a comparison of their target constructions. (in Section 2.2 and Appendix J.6)
  • Add the forced-choice evaluation, the answer-position probe, and the composition of SPDU with RMU. (in Appendix J.1)
  • Add the CoT re-scoring under four definitions, the hidden-correct trace check, and its LLM-judge validation. (in Appendix J.2)
  • Add the held-out split evaluation. (in Appendix J.3)
  • Add the measurements of the divergence on free generations and of Assumption 1 along training. (in Appendix J.4)
  • Add the comparison of fixed and co-evolving targets with forget-KL logging. (in Appendix J.5)
  • Add the 8B baselines on the generation and trace surfaces, the operating point with a stronger retention weight, and the interactive attacks against RMU. (in Appendices I.2 and J.7)
  • Discuss the confabulated outputs, the scaling behavior, and the retain-set coverage. (in Appendices E, H, I.6 and K)

We appreciate your valuable comments and time, which have strengthened the presentation of our work. We have uploaded a revised version of the paper with all changes highlighted in color. Any further comments are always welcome, and we are committed to addressing any remaining concerns. We look forward to continuing the valuable discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.