- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper presents an LLM unlearning method based on the technique of on-policy distillation, where the student model distills from a teacher model with extra refusal instructions called privileged context. To validate whether the teacher model with privileged context is capable of providing supervision, the authors introduce a supervision signal to quantify it. When the teacher model is not good enough, the authors apply further supervised fine-tuning to strengthen the refusal behavior.
Critical strengths
- The proposed unlearning method of utilizing on-policy distillation on the model with privileged context is interesting and novel to me. The unlearned model is therefore able to learn what it should answer given forget queries.
- The introduction of a supervision signal to quantify teacher supervision capabilities is interesting.
- The experiments cover a wide range of baseline unlearning methods and benchmark datasets.
Critical weaknesses
- While the authors criticize other training-free methods in the introduction for not being able to prove "whether the target knowledge is truly removed rather than merely suppressed at inference time", the proposed on-policy distillation-based method suffers from the same problem, in my opinion. This might be my major concern with this paper. The student is trained to reproduce the refusal behavior, but the factual content may still remain in the model and can still be recovered. The authors also did not show anywhere that the target data has been truly removed from the model.
- The authors determine that the teacher model is considered good if the supervision signal passes 1. This seems to be a bit arbitrary. Can the authors provide more support or analysis on the correlation between the supervision signal and the unlearning performance?
- While the authors claim that they seek to avoid the high sampling costs of RLVR-based methods, they have not provided any comparison of the efficiency between their proposed method and other baseline methods.
- More support/explanations are needed for the adoption of the on-policy distillation technique. For example, how does on-policy distillation compare to directly applying SFT to strengthen refusal behavior, like in teacher model lightweight SFT stage 1? How much performance gain does on-policy trajectory provide compared to offline distillation?
Questions for authors
- Can the authors add more supporting evidence for the true removal of data from the model?
- Will the proposed method lead to over-refusals on other retain/unseen queries?
- Can the authors consider adversarial settings such as relearning? How quickly does the OPD unlearned model relearn the forget responses?
6 comments
The authors have posted their responses to your comments. Could you please take a look and let us know whether your concerns have been addressed? We encourage active discussion between reviewers and authors during this period.
Thank you for your response. I appreciate the additional explanations, which should assist in the general understanding of this paper. However, my major concern remains: although the student model can learn to refuse requests concerning forget content, such refusal simply avoids providing forget content in reponses, but the forget knowledge still remains in the model. This may contradict what the authors define: "LLM unlearning aims to erase targeted knowledge from large language models". For example, the forget knowledge can be easily leaked or recovered through other means, such as membership inference or relearning attacks (as mentioned in question 3). I hope the authors can futher address my concerns.
Rebuttal to Reviewer xwkv
We thank you for your time, effort, and valuable comments; below are our responses to your comments.
Weakness 1. Clarify whether LEAF-OPD truly removes target knowledge rather than merely teaching refusal behavior, given the same concern raised about training-free methods.
We thank the reviewer for raising this important concern. Training-free methods do not update model parameters and primarily operate at inference time. In contrast, methods that update model weights, including LEAF-OPD, can modify the model’s internal knowledge representations and how that knowledge is used. Specifically, LEAF-OPD distills the teacher policy’s ability to distinguish between forget and retain data, refuse requests concerning forget content, and answer requests concerning retain content into the student policy. Thus, the student learns to selectively handle forget and retain requests, rather than simply learning to refuse. This idea is similar to RULE [2], but LEAF-OPD requires only one rollout per prompt, resulting in lower sampling costs than RLVR methods that require multiple rollouts.
Determining how and to what extent each unlearning method modifies the model’s knowledge to achieve genuine forgetting is an interesting research direction that we plan to explore in future work.
Weakness 2. Justify the SS threshold of 1 and analyze the relationship between supervision signal and unlearning performance.
Our criterion considers both the overall strength and the persistence of supervision in the second half of the trajectory, rather than only the refusal tendency at the first token. By definition, if corrective supervision is concentrated primarily at the first token, its contribution to the trajectory-averaged SS diminishes as the trajectory length increases, while the second-half SS remains close to zero. Conversely, if the teacher consistently provides strong corrective supervision throughout the trajectory, both metrics remain high. Figure 4 supports this interpretation: at both the individual-sample and dataset levels, the SS values are consistent with our characterization of good and bad teachers.
Regarding the relationship between supervision signal and unlearning performance, we examine this in Section 6 (Ablation Study). We evaluate a bad teacher under different configurations, including forward and reverse KL and first-token-only supervision, to assess the importance of a good teacher for LEAF-OPD.
Weakness 3. Compare efficiency with baseline methods to substantiate the claimed reduction in sampling costs relative to RLVR-based approaches.
Thank you for pointing this out. We will provide an efficiency comparison between LEAF-OPD and the baselines below and include it in the revised manuscript (ablation 5, section 6). Runtime comparison on Qwen3-8B.
Weakness 4. Demonstrate the benefits of on-policy distillation through comparisons with direct refusal SFT and offline distillation.
Please refer to Response to Common Concerns Raised by Reviewers - 1.The tiny SFT on stage1 - b.The effect of tiny SFT on performance. We will also add this analysis to Ablation 6 in Section 6 (Ablation Study).
Weakness 5. Provide additional evidence that target data have truly been removed from the model.
Please refer to our response to Weakness 1.
Weakness 6. Assess whether LEAF-OPD causes over-refusal on retain and unseen queries.
The metrics in the TOFU benchmark [1], particularly model utility (MU), already account for this concern. LEAF-OPD’s strong MU performance on TOFU suggests that it does not exhibit substantial over-refusal on unrelated questions.
Weakness 7. Evaluate robustness to relearning attacks and how quickly the unlearned model relearns the forget responses.
This is a very interesting question. In this work, we primarily focus on LEAF-OPD’s unlearning quality and its preservation of model utility through a frozen teacher. Adversarial settings, including relearning attacks, will be a key focus of our future research.
We look forward to your response and would be happy to discuss any aspect of the paper with you.
[1] TOFU: A Task of Fictitious Unlearning for LLMs, COLM2024
[2] RULE: Reinforcement UnLEarning Achieves Forget-Retain Pareto Optimality, nips2025