- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies unlearning in reasoning models, where hazardous information may remain in chain-of-thought traces even when the final answer is suppressed. It proposes Self-Projection Distilled Unlearning (SPDU), which frames unlearning as aligning the model’s plain generation with its own safety-oriented projection. SPDU constructs a contrastive target from safety- and knowledge-prompted branches, updates this target during training, and distills it at predefined refusal-scaffold prefixes while using a fixed reference model to preserve benign behavior. The paper provides a self-referential forgetting bound and a simplified analysis of training dynamics. Experiments on DeepSeek-R1-distilled models evaluate hazardous-domain accuracy, keyword-based contamination, retained capabilities, and robustness to in-context elicitation, reporting improved forgetting–utility trade-offs over the evaluated baselines.
Critical strengths
-
The paper addresses an important risk in reasoning-model unlearning. Suppressing the final answer alone may leave hazardous information exposed in intermediate reasoning traces. By treating the full generation, including the CoT, as the object of unlearning, the paper focuses on a meaningful safety problem that final-answer evaluation can overlook. This is particularly relevant when generated traces are recorded or consumed by downstream systems.
-
The self-consistency perspective provides a useful way to formulate the problem and design an unlearning method. The paper seeks to align the model’s plain generation with a safety-oriented projection constructed from its own predictions. Contrasting safety- and knowledge-prompted branches and updating the resulting target during training turns this perspective into a concrete learning procedure. Although the theoretical connection to the implemented objective needs further clarification, this formulation offers a promising direction for studying how prompt-induced safety behavior can be incorporated into model weights.
Critical weaknesses
-
The gap between the theory and implementation is not sufficiently explained. The theory concerns KL divergence and forgetting risk over the full generation distribution, whereas training evaluates KL only at prefixes of predefined refusal scaffolds. This leaves an important gap between the implementation and the claim that co-evolving targets eliminate the coverage problem: This does not establish safety at other prefixes visited during free generation, weakening the claim that co-evolving targets eliminate the coverage problem. Moreover, freezing the initial self-projection does not establish a general limitation of fixed safe teachers: failure may reflect the quality of that particular target rather than its being fixed. Without further justification, the claims should be narrowed to the configuration studied.
-
The CoT evaluation and confabulation analysis do not clearly establish reduced hazardous information leakage during reasoning. The released evaluation code searches for keywords over the full response without separating the CoT and does not apply the filtering described in Appendix C. Moreover, an incorrect final answer may still be preceded by correct hazardous facts in the reasoning trace. It is therefore unclear how the confabulation analysis, which assigns zero cost to wrong answers, supports the stated objective of preventing hazardous content anywhere in the generation. These issues affect the interpretation of the paper’s central results on reasoning-stage leakage.
Questions for authors
-
Scaffold versus free generation: How does the KL evaluated at scaffold prefixes relate to the sequence-level KL in the theoretical bound? Please clarify whether additional assumptions are needed and which guarantees apply to the implemented objective.
-
Fixed-teacher comparison: Does the frozen target itself satisfy the safety assumption in Proposition 1? Please distinguish a limitation of the initial projection from a limitation of keeping the teacher fixed. If the latter claim is intended, a comparison with a sufficiently safe fixed teacher would be needed; otherwise, the claim should be restricted to the configuration studied.
-
Explanation of frozen-branch failure: What evidence supports attributing the frozen variant’s failure to surface-level pattern matching? How is this explanation distinguished from insufficient target matching or conflict with the retention objective? Please also reconcile it with the low WMDP accuracy achieved by frozen + CE in Appendix G.
-
CoT evaluation and incorrect answers: Did the evaluation used for Table 1 isolate the CoT and apply the filtering described in Appendix C? Please also explain how evaluation and the confabulation analysis handle responses containing correct hazardous facts in the reasoning trace despite an incorrect final answer.
-
Contribution and prior work: THINKSAFE [1] already avoids external teachers, and UnDIAL [2] already uses self-distillation on adjusted logits for unlearning. Please clarify the distinction in terms of the safe–knowledge contrastive target and target updates. Beyond the ablation showing the benefit of updating the target, what evidence supports the benefit of the safe–knowledge contrast itself over existing target-construction approaches?
6 comments
Response to Reviewer Tys1 (2/4)
Generality. We agree that a general claim would need more than the configurations we tested, and we restrict the wording to them. Within that scope the evidence is consistent across three different fixed-teacher constructions: the frozen logit-space target (25.7%), an off-policy contrastive target distilled from a safe teacher's generations (23.5–26.6%), and ThinkSafe's self-generated safe traces (27.2%). All of them fail to transfer forgetting, while the co-evolving target succeeds. Proposition 1 shows how a fixed teacher can lose coverage as the student drifts, and we present it only as such, not as proof that every fixed teacher must fail.
Q3: What evidence attributes the frozen variant's failure to surface-level pattern matching, rather than insufficient matching or conflict with retention? How does this reconcile with frozen + CE in Appendix G?
A3: We thank the reviewer for pressing on this explanation. We now provide the evidence hypothesis by hypothesis, including a new frozen run with the forget-KL logged. Appendix G turns out to be part of the answer.
Table R7. Fixed vs. co-evolving targets: forgetting, utility, target matching and the model's own self-inconsistency during training (1.5B).
| Variant (1.5B) | target | retain | WMDP ↓ | MMLU ↑ | forget-KL during training (per-round average) | model's own safe–know gap |
|---|---|---|---|---|---|---|
| Frozen + CE | fixed | CE | 0.55% | 28.4% | — | — |
| Frozen + KL | fixed | KL | 25.7% (rerun: 26.9%) | 35.2% | 4.28 → 1.81 → 1.53 → 1.40 → 1.31 (−69%, plateaus) | flat: 297 → 329 → 322 → 323 → 323 → 325 |
| Single-safe (A4) | co-evolving, safe only | KL | 24.7% | 34.0% | 0.137 → 0.042 (target matched) | — |
| SPDU | co-evolving contrast | KL | 0.0% | 37.4% | 6.1 → 6.6 (target keeps moving) | grows: 297 → 280 → 364 → 378 → 446 → 406 |
- Insufficient matching is unlikely to be the cause. The single-safe variant matches its target almost exactly (forget-KL 0.04) and still does not forget. Likewise, the frozen run reduces its forget-KL by 69% while WMDP stays at base level. In both cases the student largely matches the target at the supervised positions, yet free-generation behavior does not change, which is what our wording was describing.
- Conflict with retention is part of the picture, and reconciles Appendix G. Frozen + CE shows that a fixed target can drive forgetting when the anchor is weak (0.55%), but only at a large utility cost (MMLU 28.4%, −9 vs. SPDU). Under a fixed target, forgetting and retention trade off along a frontier, and among the configurations we tested, only the co-evolving target reaches both ends.
- What distinguishes success is the self-inconsistency itself. Frozen training leaves the model's own safe–knowledge contrast flat, whereas co-evolving training grows it. The co-evolving objective changes the quantity it is designed to change, and the fixed one does not.
We will revise the explanation of the frozen-target ablation and narrow it to the configurations studied.
W2 & Q4: Did the Table 1 evaluation isolate the CoT and apply the Appendix C filter? How are correct hazardous facts in the trace followed by a wrong final answer handled?
A4: We really appreciate the reviewer for checking our released code so carefully. Indeed, the Table 1 CoT column was computed with a keyword match over the full response, without the Appendix C filter. The reviewer's second point, a correct fact in the trace followed by a wrong final answer, is exactly the failure mode the paper is about, so we re-scored every method on one set of generations under four definitions and added a direct check for that case:
Table R8. CoT leakage under four definitions and hidden-correct traces, all re-scored on one set of generations newly sampled for this response (WMDP-Bio, 1.5B).
| Method | gen-time acc ↓ | CoT: full output ↓ | CoT isolated ↓ | CoT isolated + App C filter ↓ | trace reaches gold ↓ | hidden (trace reaches gold, final answer wrong) ↓ |
|---|---|---|---|---|---|---|
| Base | 27.7 | 27.0 | 26.9 | 2.4 | 27.2 | 10.5 |
| GradAscent | 27.9 | 25.7 | 25.4 | 1.9 | 23.2 | 7.1 |
| NPO | 28.4 | 26.0 | 25.8 | 1.9 | 27.3 | 11.0 |
| SimNPO | 20.7 | 26.3 | 26.1 | 2.4 | 26.6 | 14.9 |
| RMU | 25.6 | 26.3 | 26.2 | 1.7 | 23.8 | 9.0 |
| Off-policy CD | 28.1 | 26.4 | 26.3 | 1.4 | 25.7 | 9.4 |
| ThinkSafe | 26.8 | 26.2 | 26.2 | 1.3 | 26.6 | 11.0 |
| GradDiff | 18.8 | 25.9 | 25.4 | 2.2 | 22.6 | 12.3 |
| IdkDPO | 5.3 | 5.7 | 5.7 | 0.0 | 5.2 | 1.8 |
| R2MU | 1.4 | 1.2 | 1.1 | 0.2 | 0.9 | 0.6 |
| SPDU | 0.0 | 1.0 | 1.0 | 0.0 | 0.0 | 0.0 |
Response to Reviewer Tys1 (3/4)
Table R9. Same protocol on WMDP-Cyber.
| Method | gen-time ↓ | CoT full ↓ | CoT isolated ↓ | CoT isolated + filter ↓ | trace reaches gold ↓ | hidden ↓ |
|---|---|---|---|---|---|---|
| Base | 13.3 | 31.7 | 31.6 | 17.1 | 12.8 | 4.6 |
| GradAscent | 13.9 | 28.6 | 28.2 | 12.0 | 14.8 | 5.0 |
| NPO | 11.7 | 31.9 | 31.7 | 16.4 | 12.5 | 5.0 |
| SimNPO | 13.8 | 32.8 | 32.7 | 18.3 | 14.5 | 4.9 |
| RMU | 11.6 | 31.5 | 31.4 | 16.6 | 11.6 | 4.5 |
| Off-policy CD | 11.8 | 26.9 | 26.9 | 11.2 | 11.5 | 4.4 |
| ThinkSafe | 14.1 | 29.3 | 29.1 | 12.6 | 14.0 | 4.4 |
| R2MU | 1.2 | 11.2 | 10.4 | 8.1 | 0.6 | 0.5 |
| SPDU | 4.2 | 5.7 | 5.7 | 1.5 | 0.3 | 0.1 |
- SPDU has the lowest (or tied-lowest) trace-level leakage under every definition, on both domains. Isolating the CoT changes every number by less than one point (essentially all hits are in the trace), so the conclusions of Table 1 are unchanged. On Bio the Appendix C filter, as specified, removes nearly all hits for every method (its question-keyword exclusion drops the domain terms that WMDP-Bio questions themselves contain), while on Cyber it remains discriminative, and SPDU is lowest (1.5% vs. base 17.1%, R2MU 8.1%). We will update the CoT evaluation accordingly.
- The case the reviewer raises does occur: Every method that does not forget has traces that reach the correct option before a wrong or missing final answer in 7–15% of responses on Bio (SimNPO 14.9%; 4–5% on Cyber), and a final-answer metric scores these as forgotten. This is the paper's motivating failure mode, now measured directly. For SPDU the rate is 0.0%, and no SPDU trace reaches the correct option on this evaluation set. The baselines that also suppress traces (R2MU, IdkDPO) do so at a cost in general or reasoning utility (R2MU MMLU 20.2%, IdkDPO MATH-500 45.8%), whereas SPDU keeps both (MMLU 37.4%, MATH-500 76.2%). At 8B the gap is larger still: base 23.7% vs. SPDU 0.0% (aPwp-W1).
Is the trace-level check itself reliable? Since "trace reaches gold" is detected by pattern matching, we validated it with an independent LLM judge that reads only the think block and reports which option the reasoning concludes or favors, or none:
Table R10. Validation of the trace-level check with an independent LLM judge (on a random subsample of responses).
| Model (WMDP-Bio) | agreement with pattern check | hidden-correct: pattern / judge |
|---|---|---|
| Base 1.5B | 86.3 | 13.0 / 15.0 |
| NPO | 84.3 | 11.0 / 16.0 |
| SimNPO | 84.7 | 13.7 / 20.3 |
| RMU | 83.3 | 8.7 / 17.7 |
| SPDU 1.5B | 99.7 | 0.0 / 0.3 |
| Base 8B | 69.0 | 25.7 / 38.7 |
| SPDU 8B | 99.0 | 0.0 / 0.7 |
The pattern check is conservative for the baselines (the judge finds more hidden-correct traces), so the gap to SPDU is, if anything, understated. For SPDU, the two agree on 99%+ of responses. How the confabulation analysis relates to trace content. The toy analysis models the final answer of an MCQ and assigns zero cost to wrong final answers. It explains why the suppressed output need not be a refusal, and we will clarify that scope in the revision. Trace content is covered by separate measurements that do not rely on it: the trace-level answer check above (the reviewer's case), and the factuality-verified interactive-attack evaluation, which judges whether the whole response conveys the correct hazardous fact regardless of the final answer (Table R4).
We will revise the inconsistent descriptions and update the evaluation accordingly in the revision.
Response to Reviewer Tys1 (1/4)
We sincerely thank Reviewer Tys1 for the careful review and constructive comments. We are glad the reviewer found the paper "clear and easy to follow" and that it provided "a substantially deeper understanding of the problem", and we appreciate the recognition that the paper "addresses an important risk in reasoning-model unlearning" and "focuses on a meaningful safety problem that final-answer evaluation can overlook", and that the self-consistency perspective "provides a useful way to formulate the problem" and "offers a promising direction." The reviewer's questions led us to a sharper, trace-level evaluation. The questions concentrate on the link between the theory and the implementation and on the CoT evaluation, and we address each with new analysis and experiments below.
A1: We appreciate the reviewer for this careful question. We would like to make the relation precise and state the additional assumption explicitly.
Conceptually, the sequence-level divergence in the bound decomposes by the chain rule as KLseq=Ey∼O1[∑sKL(O1(⋅∣x,y<s),∥,πθ(⋅∣x,y<s))]: the same per-position terms our loss computes, weighted by prefixes drawn from the teacher O1. Our loss evaluates these terms on the refusal-scaffold prefixes, so it controls KLseq only insofar as those prefixes are representative of the teacher's own. This representativeness is the additional assumption. At our contrastive strengths the teacher favors refusal-leaning tokens (Fig. 2a), which motivates the scaffold, but we state the assumption rather than derive it.
Proposition 2 extends exactly to sequence-level costs, with η replaced by KLseq. For the implemented objective, it applies only under the assumption above, so we do not claim a formal guarantee for it and support it by measurement instead (below). Accordingly, we present the propositions as an idealized analysis that motivates the two design choices (a target recomputed from current weights, logit-space distillation), and we replace "eliminates this failure by design" with a statement restricted to that objective. What co-evolution provides, precisely, is the reference: Proposition 1 compares the student with a target fixed at collection time, whose divergence grows as the student moves away from it, whereas in Proposition 2 the reference is recomputed from the same weights, so this drift term does not arise.
Empirically, we measured the quantity the reviewer is concerned about at prefixes training never visits, the model's own free-generation prefixes (Table R3, in our response to Reviewer aPwp): the per-token divergence to the model's own contrastive target falls on scaffold prefixes (11.7 → 4.2) and on its own free generations (2.6 → 2.1). Behaviorally, none of SPDU's free-generation traces reaches the correct option on our evaluation set (0.0%, W2).
We will narrow the theoretical claims accordingly in the revision.
A2: Thanks for the insightful question, which we answer in two parts. Does the frozen target satisfy the assumption? Decoding the frozen target O1[θ0] with the training prompts gives 5.0% hazardous accuracy vs. 25.3% for the base model, so it is a safe fixed teacher in the sense of Proposition 1 (ε≈5, far below chance). The comparison the reviewer asks for, against a sufficiently safe fixed teacher, is therefore the frozen-vs-co-evolving ablation.
Initial projection vs. keeping it fixed. The frozen and co-evolving variants share the identical initial target (Δ0=0 in Eq. 11) and everything else, and diverge only through updating: 25.7% vs. 0.0% WMDP. Within our setting the difference is therefore attributable to keeping the target fixed, not to the quality of the initial target. Conceptually, a fixed target specifies what the model should output at the weights where it was computed. As the student moves, the knowledge it still carries is re-expressed through prefixes and features the fixed target never addressed, whereas a co-evolving target is recomputed from the current weights and keeps pointing at whatever knowledge remains.