acceptodds
Under review as a conference paper at ICLR 2027

Token-Selective On-Policy Self-Distillation for LLM Unlearning

Abstract

Large language model (LLM) unlearning aims to suppress designated knowledge while preserving non-target capabilities, without retraining from scratch. However, existing parameter-updating methods often rely on externally constructed supervision, while directly distilling in-context unlearning (ICU) may also transfer prompt-induced changes unrelated to forgetting. In this work, we propose Token-Selective On-Policy Logit Distillation (TOLD), a response-label-free unlearning method that uses ICU as supervision for parameter updating. TOLD compares a frozen teacher's predictions with and without the ICU prompt along student-generated trajectories. It then selectively transfers token-level changes for suppression and promotion, limiting unrelated changes during distillation and enabling prompt-free inference. Experiments on the RWKU benchmark across two backbones show that TOLD reduces mean forget-set recall by 39.6-42.5% relative to the base models while largely preserving neighboring knowledge and general utility. TOLD also demonstrates robust forgetting across nine adversarial attacks. Additional experiments on MUSE-Books extend these findings beyond entity-level unlearning. These results demonstrate the effectiveness of selective on-policy distillation for prompt-free behavioral unlearning.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Model Unlearning, On-Policy Self-Distillation, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bb9wOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes TOLD, a query-only unlearning method that distills an in-context unlearning prompt into model parameters. A frozen teacher evaluates student-generated responses with and without the prompt. TOLD selectively transfers probability changes for vocabulary tokens through a KL objective. Experiments cover single-target unlearning on RWKU and additional evaluation on MUSE-Books.

Critical strengths
  1. Avoiding reference answers and a separate external teacher is practically useful. The method uses the base model to provide supervision and removes the need for an unlearning prompt at inference time. This reduces response-curation requirements and offers a straightforward way to turn prompt-based behavior into a parameter update.
  2. The evaluation considers neighboring knowledge as well as forgetting, which is important for entity-level unlearning. TOLD preserves neighboring knowledge well on Llama, and the Full-KLD comparison provides some evidence that selective distillation can reduce collateral damage. The additional MUSE-Books results also provide an initial test beyond individual entities.
Critical weaknesses
  1. The model coverage is too narrow. The experiments use only Llama-3.2-3B and Phi-3-mini, both from 2024 and both in the 3–4B range. There is no meaningful scaling study and no evaluation on recent Qwen or Gemma models. Since TOLD relies on the model’s own prompt-following ability to generate supervision, its effectiveness may depend strongly on the backbone.
  2. The main results do not establish a clear advantage. Averaging the three forgetting columns in Table 1, TOLD’s Llama forget recall is about 0.294, compared with about 0.280 for both NPO and Full-KLD, where lower is better. TOLD improves neighboring retention, but this is a trade-off rather than stronger forgetting. On Phi, it improves QA and adversarial forgetting, but has the lowest reported fluency score, 5.848 versus 6.737 for the base model. These mixed results require comparisons at matched forgetting or retention levels, not just isolated operating points.
  3. The baseline comparison is not sufficiently controlled. Table 2 uses different learning rates, effective batch sizes, and training lengths, but provides no comparable search budgets or common selection rule. TOLD receives backbone-specific β\beta selection, while baselines use settings from their released implementations. The total search budget for TOLD is also unclear. A documented tuning protocol, such as the one used in OpenUnlearning, is needed. Moreover, TOLD trains on 300 non-target queries, whereas the listed GA and NPO objectives contain no retain term. Comparisons with GradDiff or NPO plus retention, using the same non-target data, would better test the claimed retention advantage.
  4. The central design choices are not cleanly validated. There is no controlled on-policy versus off-policy comparison. DUET also differs in which response positions are distilled, so beating DUET does not isolate the benefit of on-policy sampling. The Full-KLD comparison has another potential confound: TOLD uses β=2\beta=2 on Phi, while Full-KLD is described as directly matching the ICU distribution. It is unclear whether intervention strength and clipping are matched. Similarly, bidirectional selection can modify up to 2K2K vocabulary entries, but the paper does not establish that the one-sided ablations use the same total token budget. Without these controls, the gains cannot be confidently attributed to the proposed components.
  5. The privacy and robustness evidence is weak. The forget-member loss barely changes: from 2.429 to 2.432 on Llama and from 1.932 to 1.933 on Phi. These results provide little evidence of reduced memorization under the paper’s own measure. The “MIA” columns report average passage losses, not attack performance such as ROC-AUC or attack success at a fixed false-positive rate. Robustness is assessed using the provided RWKU prompt suite and ROUGE-L, without adaptive extraction tests or a semantic check of whether target facts remain recoverable. The paper acknowledges behavioral suppression, but even within that scope, stronger evidence is needed.
  6. Reproducibility and computational cost are insufficiently documented. The anonymous repository lacks a README, leaving no clear starting point for reproducing the experiments.
Questions for authors
  1. What search spaces, trial counts, and selection criteria were used for every method, including TOLD? Does TOLD retain its advantage after comparable tuning and after giving retention-aware baselines access to the same non-target data?
  2. Can the authors provide controlled on-policy versus off-policy results? For Full-KLD and the one-sided ablations, were β\beta, clipping, and the total number of selected vocabulary entries matched?
  3. Why do the forget-member losses remain almost unchanged? Can the authors report actual membership-inference performance and test whether adaptive prompts recover target facts that the current evaluation misses?
  4. Can the authors show that the findings hold beyond the 3–4B range and on more recent families, such as Qwen3.5 or Gemma 4? In particular, does the forgetting–retention advantage remain after tuning the baselines on these models?
  5. Can the authors provide a README, exact reproduction commands, search configurations, and per-target runtime and memory measurements, including student generation and teacher evaluation?
Reviewer_EnosOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the reference-free unlearning problem for large language models and proposes Token-Selective On-Policy Logit Distillation (TOLD). Specifically, trajectories are generated online by the student model without the unlearning prompt, and the frozen base model computes the output distributions for the same prefix with and without the unlearning prompt, respectively. Based on probability-weighted changes in log-probabilities, TOLD bidirectionally selects tokens to suppress and promote, constructs a local distillation target anchored to the original base model distribution, and updates the student model through token-wise forward KL divergence, thereby suppressing target behaviors at inference without the need for the prompt. Experiments were conducted on two base models based on RWKU (100 entities) and MUSE-Books, and 9 types of robustness attacks were tested. The results show that while TOLD reduces target knowledge recall, it can relatively well preserve neighboring knowledge under some settings.

Critical strengths
  1. Practical starting point and self-consistent method design. TOLD does not require pre-collected target responses, constructed preference pairs, or additional training of an independent teacher model; it directly internalizes the contextually elicited unlearning capability into the model parameters, substantially alleviating the burden of constructing supervised data. The bidirectional token selection addresses both "suppressing the original output" and "promoting alternatives," with the remaining vocabulary positions directly aligned to the original base model distribution. The overall approach is clear and reasonable.
  2. Achieved a good balance between unlearning effectiveness and neighboring knowledge retention. From Table 1, TOLD's neighboring knowledge retention mean on Llama reaches 0.5905, significantly higher than DUET (0.4880) and Full-KLD (0.5500), while maintaining strong unlearning effectiveness; on Phi, its average forgetting metric is reduced by about 31.8% compared to DUET, and neighboring knowledge performance is essentially on par. Supplementary experiments on multiple entities and MUSE-Books across models indicate that the method has considerable empirical effectiveness.
Critical weaknesses
  1. The core claim of "on-policy" lacks rigorous controlled-variable verification. The paper presents on-policy distillation as its core contribution (directly reflected in the title and the list of contributions), but the experimental section mainly compares with DUET. In fact, DUET itself already has query-only training and Top-K distillation, and its appendix also discusses multi-step and full-sequence distillation based on teacher-generated prefixes. Between TOLD and DUET, the trajectory source, supervision positions, token selection rules, and distillation targets are all changed simultaneously; there are too many variables to directly attribute the performance gains to on-policy training. The paper is precisely lacking the most direct ablation experiment: fixing the selective distillation target and training budget, and only replacing the "trajectories currently generated by the student" with the "fixed (off-policy) trajectories generated by the frozen base model." Without this comparison, it is difficult to assess the necessity and actual contribution of the on-policy mechanism itself.
  2. There are concerns about misaligned hyperparameters in the ablation experiments on the token selection mechanism. On one hand, the paper describes Full-KLD as directly fitting the complete ICU distribution, but on Phi, TOLD uses β=2\beta=2, and the paper does not state whether Full-KLD adopts the same intervention strength (β\beta) and clipping rules. On the other hand, TOLD's bidirectional selection takes K=32K=32 for each of suppression and promotion (up to 64 vocabulary items modified per step), but whether the unidirectional ablation (suppression-only or promotion-only) takes K=32K=32 or K=64K=64 to match the total budget is not explained. If the intervention strength or the total number of modified tokens is inherently inconsistent, it is difficult to distinguish whether the existing results' gains come from the "bidirectional selection mechanism" itself, or simply because the scope of the vocabulary or the intensity of intervention is greater.
Questions for authors
  1. If variables are strictly controlled (keeping the same queries, selective distillation targets, number of supervised tokens, and update steps), and only the training trajectories are replaced from online generation by the student (on-policy) to pre-generation by the frozen base model (off-policy), how much difference would there be in forgetting metrics and neighboring knowledge retention? Can this comparison prove that on-policy training has independent benefits?
  2. What are the specific implementation details of the Full-KLD baseline? Does it directly fit the original pIp_I, or does it set mt(v)≡1m_t(v) \equiv 1 in Equation (12) and maintain the same β\beta and clipping settings as TOLD? Particularly on Phi (where β=2\beta=2), if the intervention strength is aligned, does the advantage of the selective target over Full-KLD still hold?
  3. In the unidirectional selection ablation experiments (promotion-only and suppression-only), what is the value of KK? If K=32K=32, can the results for unidirectional K=64K=64 be added and compared with bidirectional (32+3232+32) to rule out the effect of increased total budget of intervened tokens?
Reviewer_YwvXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TOLD, a token-selective on-policy self-distillation method for LLM unlearning. It uses an in-context unlearning (ICU) prompt to provide supervision from a frozen teacher and selectively transfers token-level probability changes to a student model. The method enables prompt-free inference without requiring reference responses. Experiments on RWKU and MUSE-Books show promising forgetting performance while largely preserving neighboring knowledge and general utility.

Critical strengths
  1. The paper addresses an interesting problem by combining ICU with on-policy self-distillation for LLM unlearning.
  2. The bidirectional token selection mechanism is intuitive and reasonably motivated.
  3. The evaluation is fairly comprehensive, covering forgetting, neighboring knowledge, general utility, membership inference, and adversarial robustness.
  4. The ablation studies provide useful evidence for the benefits of selective distillation and bidirectional token selection.
Critical weaknesses
  1. The choice of the Top-K selection budget is not sufficiently justified. Token selection is a central component of TOLD, but the paper fixes K=32 without analyzing its sensitivity. The number of selected tokens directly controls how much ICU-induced distribution change is transferred to the student and may affect the forgetting–utility trade-off. I would like to see experiments with different K values to better understand the effectiveness and stability of this design choice.

  2. The distinction between behavioral suppression and knowledge removal remains unclear. Although TOLD substantially reduces generation-based recall, its forget-member MIA scores remain almost unchanged compared with the base model. This raises the question of whether the method primarily learns refusal behavior rather than reducing the underlying knowledge. The authors acknowledge that TOLD mainly achieves behavioral suppression, but further analysis or relearning-based evaluation would help clarify the nature and robustness of the achieved forgetting.

Questions for authors
  1. On-policy vs. off-policy training: Can the authors compare on-policy and off-policy training while keeping the same selective distillation target and training budget? This would help isolate the contribution of on-policy sampling from the token selection mechanism.
  2. Top-K sensitivity: How sensitive is TOLD to the selection budget K? Does the forgetting–utility trade-off remain stable across different K values?
  3. Behavioral suppression: How should we interpret the substantial reduction in generation-based recall alongside the nearly unchanged forget-member MIA scores? Would relearning-based evaluation help distinguish behavioral suppression from more persistent forgetting?
  4. ICU prompt design: The method relies on a manually designed ICU prompt to provide the unlearning signal, but its effectiveness may depend on the quality of this prompt. Could the authors discuss how this prompt was chosen and how sensitive TOLD is to different ICU prompts? It would also be interesting to explore whether a more systematic or optimizable prompt construction strategy could provide better and more reliable supervision.

I am open to raising my score if the authors adequately address these concerns and questions during the rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.