- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors propose Hybrid-Policy Self-Editing (HPSE), which distills a frozen, passage-conditioned version of the original model into the edited model. During rollout generation, the teacher intervenes when its confidence and its disagreement with the student exceed predefined thresholds. Training combines distillation along these hybrid trajectories with a passage-level NLL objective. Experiments evaluate HPSE with two editors across four models under single and continual editing.
Critical strengths
-
The paper addresses a meaningful limitation of knowledge editing: reproducing an edited passage does not establish that its facts can be independently retrieved or combined.
-
The evaluation covers multiple models, editors, and editing settings. HPSE consistently improves benchmark averages, with substantial multi-hop gains in some configurations.
-
The ablations include useful comparisons against pure on-policy distillation, on-policy distillation with NLL, and fully teacher-generated rollouts.
Critical weaknesses
-
The paper clearly distinguishes decomposition and composition as two challenges in unstructured knowledge editing. However, the connection between these challenges and the proposed technical solution is less clear. Why should OPSD with hybrid rollouts specifically improve fact-specific retrieval and multi-hop composition? The coverage analysis concerns access to factual token prefixes, but does not establish how improved coverage translates into either capability.
-
The connection to DAgger is acknowledged, but the distinction from existing teacher-intervention and hybrid-policy approaches is insufficiently established. What constitutes the substantive methodological advance beyond adapting these ideas to knowledge editing?
-
In Table 2, MQuAKE-uns composition accuracy increases from 32.0 for LoRA to 51.3 for OPSD with NLL, while full HPSE reaches 54.7. Thus, much of the improvement is already obtained without hybrid rollouts. Although the additional gains under continual editing are encouraging, can the authors establish the intervention’s contribution across more models and report uncertainty for these comparisons?
-
The implementation approximates distillation using the teacher’s top-16 tokens, whereas the theoretical analysis uses full-distribution KL. Truncation and rollout coverage are distinct issues, but both may affect learning. Does HPSE retain its advantage with larger top-k targets or full-vocabulary KL?
Questions for authors
Please see the Weaknesses.
4 comments
Contribution of hybrid rollouts beyond OPSD+NLL, across more models and with uncertainty.
Following the reviewer's suggestions, we conducted additional experiments comparing HPSE and OPSD+NLL on two more backbones (Llama3.1 and Gemma2) and reported standard errors (detailed below) as a measure of uncertainty. Results (new ones on Llama3.1 and Gemma2, and existing ones on Qwen2.5) are summarized in the table below. We noted that the composition gain of HPSE over OPSD+NLL holds on all three backbones, and becomes larger on Llama3.1 and Gemma2, where OPSD+NLL alone does not improve composition over LoRA. The reported uncertainty also indicates that these improvements are not merely random noise, with the overall composition gain reaching 8.4 pts. For an intuitive explanation, we view OPSD+NLL as a coarse, rollout-level form of step-in, which combines on-policy OPSD with off-policy NLL on the fixed passage, and interpret the empirical gap as the benefit of token-level step-in in HPSE.
We have added the results on more backbones and the discussions to the revised paper.
| Backbone | Method | UnKE Jnt. | UnKE Dmp. | UnKE Div. | UnKE Avg. | MQuAKE Ind. | MQuAKE Cmp. | MQuAKE Avg. |
|---|---|---|---|---|---|---|---|---|
| Qwen2.5 | LoRA | 63.2±3.1 | 47.8±2.7 | 63.0±1.8 | 58.0±1.3 | 73.3±2.1 | 32.0±4.6 | 52.6±2.7 |
| OPSD+NLL | 72.5±3.0 | 62.6±2.8 | 62.0±1.5 | 65.7±1.4 | 81.7±1.8 | 51.3±4.3 | 66.5±2.7 | |
| HPSE | 75.0 | 62.5 | 60.9 | 66.1 | 83.2 | 54.7 | 68.9 | |
| Llama3.1 | LoRA | 78.5±3.1 | 62.5±2.7 | 45.0±2.1 | 62.0±1.3 | 75.8±2.4 | 46.7±5.0 | 61.3±2.8 |
| OPSD+NLL | 73.1±2.9 | 62.8±2.8 | 60.9±2.2 | 65.6±1.5 | 76.1±2.5 | 33.3±5.0 | 54.7±2.9 | |
| HPSE | 78.9 | 61.9 | 59.9 | 66.9 | 81.0 | 50.0 | 65.5 | |
| Gemma2 | LoRA | 85.0±3.3 | 69.4±2.6 | 34.3±2.4 | 62.9±1.4 | 79.1±2.1 | 41.3±4.5 | 60.2±2.7 |
| OPSD+NLL | 83.9±2.6 | 68.3±2.3 | 56.0±2.0 | 69.4±1.2 | 80.8±2.0 | 40.7±3.6 | 60.8±2.3 | |
| HPSE | 83.1 | 67.9 | 53.6 | 68.2 | 81.5 | 46.0 | 63.7 | |
| Overall | LoRA | 75.6±1.9 | 59.9±1.6 | 47.5±1.3 | 61.0±0.8 | 76.1±1.3 | 40.0±2.7 | 58.0±1.6 |
| OPSD+NLL | 76.5±1.6 | 64.6±1.5 | 59.6±1.1 | 66.9±0.8 | 79.6±1.2 | 41.8±2.5 | 60.7±1.5 | |
| HPSE | 79.0 | 64.1 | 58.1 | 67.1 | 81.9 | 50.2 | 66.1 |
± denotes the standard error of the per-edit paired difference from HPSE.
Top-16 KL approximation versus full-vocabulary KL.
We adopted the truncated KL design in the literature [1-3] for better computational efficiency and to avoid potential numerical instability and random noise in the tail of the distribution. As suggested by the reviewer, we experimented with full-vocabulary KL. Based on our results, HPSE remains effective under full-vocabulary KL. We also observed lower performance on MQuAKE-uns for both OPSD+NLL and HPSE compared with their top-16 counterparts, suggesting that including the full distribution does not necessarily improve knowledge transfer, possibly due to the noise in its tail. At the same time, we agree with the reviewer that the choice of may affect learning. As we did not extensively tune , a more suitable choice may further improve performance.
| Method | KL target | UnKE Jnt. | UnKE Dmp. | UnKE Div. | UnKE Avg. | MQuAKE Ind. | MQuAKE Cmp. | MQuAKE Avg. |
|---|---|---|---|---|---|---|---|---|
| LoRA | – | 63.2 | 47.8 | 63.0 | 58.0 | 73.3 | 32.0 | 52.6 |
| OPSD+NLL | top-16 | 72.5 | 62.6 | 62.0 | 65.7 | 81.7 | 51.3 | 66.5 |
| HPSE | top-16 | 75.0 | 62.5 | 60.9 | 66.1 | 83.2 | 54.7 | 68.9 |
| OPSD+NLL | full | 72.4 | 64.1 | 61.3 | 65.9 | 81.7 | 43.3 | 62.5 |
| HPSE | full | 77.4 | 62.6 | 61.2 | 67.0 | 83.4 | 45.3 | 64.4 |
[1] Entropy-Aware On-Policy Distillation of Language Models, 2026
[2] Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes, 2026
[3] Reinforcement Learning via Self-Distillation, 2026
Thank you for the additional experiments. The composition gains over OPSD+NLL across all three backbones strengthen the case for hybrid rollouts, and the full-vocabulary KL experiment addresses my concern about truncation.
Could you replace only HPSE’s gate with Speculative KD’s top- acceptance rule [1] and compare how often the edited facts appear in the rollouts and how accurately the edited model answers multi-hop questions, using the same teacher and comparable intervention rates?
[1] Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling, 2025
Dear Reviewer RUw5,
We are happy to see that our responses addressed your other concerns. We provide our new response below to your follow-up questions.
Comparison with Speculative KD.
Following the reviewer's suggestion, we conducted additional experiments that replace the step-in strategy of HPSE with that of Speculative KD (SKD) [1], using the same teacher. We consider two variants: the default SKD, and HPSE using SKD's step-in strategy (HPSE+SKD). Due to time constraints, we directly adopted the default hyperparameters without tuning. Results on MQuAKE-uns are summarized in the table below. We noted that hybrid rollouts are helpful under both step-in strategies within HPSE, and which strategy works better depends on the backbone. Specifically, on Qwen2.5, SKD performed better than HPSE with a lower step-in rate, and HPSE+SKD, with an even lower step-in rate, performed the best. On the other hand, HPSE performed better on Llama3.1 than the two SKD variants, while HPSE+SKD still outperformed SKD. These results confirm the benefit of step-in within HPSE. The step-in can be done in many ways, and the optimal step-in strategy, as included the revised paper, is still an open question that we didn't address.
| Backbone | Method | Step-in rate | Ind. | Cmp. | Avg. |
|---|---|---|---|---|---|
| Qwen2.5 | LoRA | – | 73.3 | 32.0 | 52.6 |
| OPSD+NLL | 0.0% | 81.7 | 51.3 | 66.5 | |
| SKD | 4.4% | 86.7 | 58.0 | 72.4 | |
| HPSE+SKD | 3.1% | 88.3 | 61.3 | 74.8 | |
| HPSE | 13.1% | 83.2 | 54.7 | 68.9 | |
| Llama3.1 | LoRA | – | 75.8 | 46.7 | 61.3 |
| OPSD+NLL | 0.0% | 76.1 | 33.3 | 54.7 | |
| SKD | 3.1% | 65.1 | 31.3 | 48.2 | |
| HPSE+SKD | 2.2% | 79.7 | 38.7 | 59.2 | |
| HPSE | 9.9% | 81.0 | 50.0 | 65.5 |
Step-in rate is the fraction of rollout tokens replaced by the privileged teacher over all training rounds.
[1] Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling, 2025
How often the edited facts appear in the rollouts.
The “coverage failure” we discussed indicates that the student's response can go off-topic and cover other content that is irrelevant to the editing knowledge, leaving the teacher little to supervise. We allow the teacher to proactively step in, which can have two empirical effects: steering the rollout toward the knowledge to be discussed, and adding more edited facts to it (Fig. 11). Empirically, atomic-fact answer tokens form approximately 17.8% of the step-in tokens in the first-round rollouts of HPSE (UnKEBench, Qwen2.5). On the other hand, an “always step-in” (which should lead to very good fact coverage) does not perform well either (Tab. 2). Moreover, based on our new results above, HPSE+SKD and SKD use fewer step-ins (thereby less teacher content) than HPSE, but can still perform comparably in practice (e.g., better on Qwen2.5). HPSE+SKD has consistently fewer step-ins but better performance than SKD. Considering that early step-ins may cause the trajectories to deviate largely, how often the edited facts appear in the rollouts may not be a good proxy for editing performance.
We highly appreciate your effort and time spent reviewing our paper and thank you for your expertise and constructive comments. In the following, we address your comments and questions one by one, which will also be updated in the revised paper.
We study and formulate decomposition and composition as two aspects of knowledge composability, requiring an LLM to recall each individual fact as needed and to combine facts for multi-hop reasoning, respectively. Conceptually, this relates to the general capabilities of LLMs in knowledge retrieval [1] and reasoning [2, 3]. It is widely observed that after KE, the model often fails to apply these capabilities to the new knowledge: it can memorize the new knowledge but fails to use it [4-6]. Therefore, an ideal KE method should inject the new knowledge in a way that these capabilities the model already has remain applicable to it.
Toward this goal, we noted that existing KE methods often passively rely on the fixed editing statement as the sole learning source, and seek proactive on-policy learning instead, inspired by recent findings that on-policy learning generally generalizes better than learning from fixed data [7-9].
For untargeted UKE, we further identify a coverage failure: the student neither knows the new knowledge nor what is being edited, so its response can go off-topic and cover content that is irrelevant to the editing knowledge, leaving the teacher little to supervise on the new knowledge. HPSE therefore allows the teacher to step in proactively. Our coverage analysis addresses this separate point, i.e., how to make on-policy learning effective for UKE. We also agree with the reviewer that a more precise characterization of how on-policy learning preserves these capabilities is an important question that deserves further study. We have updated our paper to discuss this as an open question in the conclusion, and revised other parts for better clarity.
[1] Language Models as Knowledge Bases?, 2019
[2] Large Language Models are Zero-Shot Reasoners, 2022
[3] Do Large Language Models Latently Perform Multi-Hop Reasoning?, 2024
[4] MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions, 2023
[5] MQuAKE-Remastered: Multi-Hop Knowledge Editing Can Only Be Advanced with Reliable Evaluations, 2025
[6] Uncovering Overfitting in Large Language Model Editing, 2024
[7] On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes, 2024
[8] SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training, 2025
[9] Self-Distillation Enables Continual Learning, 2026
HPSE is motivated by and formulated around the coverage failure we identified in untargeted UKE. From a methodological perspective, it lets a privileged teacher supervise, and occasionally step into, the student's own rollouts. Ideas in a similar spirit have been explored in the literature, each motivated by a different problem. For example, DAgger [1] reduces the distribution mismatch through expert feedback on learner-visited states, HDPO [2] addresses vanishing learning signals through privileged self-distillation with verifiable rewards, and Speculative KD [3] corrects low-quality student rollouts by replacing tokens unlikely under the teacher.
Specifically, coverage failure means that the student goes off-topic and omits the editing knowledge, so supervision on its own rollouts teaches little about the new knowledge. This calls for a different criterion for when to intervene: HPSE steps in when the privileged model's preferred token is confident and clearly underestimated by the student, which brings in the knowledge that the student would not mention by itself, even when its own token is reasonable. We further analyze how such step-ins provide supervision on the new knowledge.
We have revised the related work to make this clearer.
[1] A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, 2011
[2] HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation, 2026
[3] Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling, 2025