acceptodds
Under review as a conference paper at ICLR 2027

Hybrid-Policy Self-Editing for Composable Unstructured Knowledge Editing

20 views

Abstract

Pretrained large language models (LLMs) achieve remarkable performance across natural language tasks, yet their internal knowledge quickly becomes outdated in a fast-changing world. This motivates knowledge editing (KE), which updates specific knowledge in an LLM without changing unrelated knowledge. Recent works move from structured knowledge triples toward unstructured KE (UKE), where the edit is a free-form passage that may state multiple facts at once. Nonetheless, existing editors inject the passage but fail to use it: the edited model can recall the passage, but can neither answer atomic questions about its facts nor compose them into multi-hop reasoning. We attribute this missing property, which we term composability, to editors' passive reliance on the fixed passage as the sole learning source. In response, we cast editing as proactive self-distillation from a privileged in-context state of the same model, which requires no external supervision. We further reveal that because the injected knowledge is novel, the pre-edited model's own rollouts rarely cover it, limiting the effectiveness of pure on-policy distillation. To close this gap, we propose HPSE, which builds a hybrid rollout that steps in to place missing facts onto the student's own trajectory precisely where its coverage fails, while staying on-policy elsewhere. We theoretically analyze HPSE's advantage over pure on-policy distillation and empirically establish its plug-and-play improvements across four LLM backbones and two KE editors under various scenarios. We release our datasets and evaluation code at https://anonymous.4open.science/r/hpse-review-C778.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Knowledge Editing, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RUw5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose Hybrid-Policy Self-Editing (HPSE), which distills a frozen, passage-conditioned version of the original model into the edited model. During rollout generation, the teacher intervenes when its confidence and its disagreement with the student exceed predefined thresholds. Training combines distillation along these hybrid trajectories with a passage-level NLL objective. Experiments evaluate HPSE with two editors across four models under single and continual editing.

Critical strengths
  1. The paper addresses a meaningful limitation of knowledge editing: reproducing an edited passage does not establish that its facts can be independently retrieved or combined.

  2. The evaluation covers multiple models, editors, and editing settings. HPSE consistently improves benchmark averages, with substantial multi-hop gains in some configurations.

  3. The ablations include useful comparisons against pure on-policy distillation, on-policy distillation with NLL, and fully teacher-generated rollouts.

Critical weaknesses
  1. The paper clearly distinguishes decomposition and composition as two challenges in unstructured knowledge editing. However, the connection between these challenges and the proposed technical solution is less clear. Why should OPSD with hybrid rollouts specifically improve fact-specific retrieval and multi-hop composition? The coverage analysis concerns access to factual token prefixes, but does not establish how improved coverage translates into either capability.

  2. The connection to DAgger is acknowledged, but the distinction from existing teacher-intervention and hybrid-policy approaches is insufficiently established. What constitutes the substantive methodological advance beyond adapting these ideas to knowledge editing?

  3. In Table 2, MQuAKE-uns composition accuracy increases from 32.0 for LoRA to 51.3 for OPSD with NLL, while full HPSE reaches 54.7. Thus, much of the improvement is already obtained without hybrid rollouts. Although the additional gains under continual editing are encouraging, can the authors establish the intervention’s contribution across more models and report uncertainty for these comparisons?

  4. The implementation approximates distillation using the teacher’s top-16 tokens, whereas the theoretical analysis uses full-distribution KL. Truncation and rollout coverage are distinct issues, but both may affect learning. Does HPSE retain its advantage with larger top-k targets or full-vocabulary KL?

Questions for authors

Please see the Weaknesses.

4 comments

We highly appreciate your effort and time spent reviewing our paper and thank you for your expertise and constructive comments. In the following, we address your comments and questions one by one, which will also be updated in the revised paper.

How on-policy self-distillation with hybrid rollouts connects to knowledge decomposition and composition.

We study and formulate decomposition and composition as two aspects of knowledge composability, requiring an LLM to recall each individual fact as needed and to combine facts for multi-hop reasoning, respectively. Conceptually, this relates to the general capabilities of LLMs in knowledge retrieval [1] and reasoning [2, 3]. It is widely observed that after KE, the model often fails to apply these capabilities to the new knowledge: it can memorize the new knowledge but fails to use it [4-6]. Therefore, an ideal KE method should inject the new knowledge in a way that these capabilities the model already has remain applicable to it.

Toward this goal, we noted that existing KE methods often passively rely on the fixed editing statement as the sole learning source, and seek proactive on-policy learning instead, inspired by recent findings that on-policy learning generally generalizes better than learning from fixed data [7-9].

For untargeted UKE, we further identify a coverage failure: the student neither knows the new knowledge nor what is being edited, so its response can go off-topic and cover content that is irrelevant to the editing knowledge, leaving the teacher little to supervise on the new knowledge. HPSE therefore allows the teacher to step in proactively. Our coverage analysis addresses this separate point, i.e., how to make on-policy learning effective for UKE. We also agree with the reviewer that a more precise characterization of how on-policy learning preserves these capabilities is an important question that deserves further study. We have updated our paper to discuss this as an open question in the conclusion, and revised other parts for better clarity.

[1] Language Models as Knowledge Bases?, 2019

[2] Large Language Models are Zero-Shot Reasoners, 2022

[3] Do Large Language Models Latently Perform Multi-Hop Reasoning?, 2024

[4] MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions, 2023

[5] MQuAKE-Remastered: Multi-Hop Knowledge Editing Can Only Be Advanced with Reliable Evaluations, 2025

[6] Uncovering Overfitting in Large Language Model Editing, 2024

[7] On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes, 2024

[8] SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training, 2025

[9] Self-Distillation Enables Continual Learning, 2026

Methodological advance beyond existing approaches.

HPSE is motivated by and formulated around the coverage failure we identified in untargeted UKE. From a methodological perspective, it lets a privileged teacher supervise, and occasionally step into, the student's own rollouts. Ideas in a similar spirit have been explored in the literature, each motivated by a different problem. For example, DAgger [1] reduces the distribution mismatch through expert feedback on learner-visited states, HDPO [2] addresses vanishing learning signals through privileged self-distillation with verifiable rewards, and Speculative KD [3] corrects low-quality student rollouts by replacing tokens unlikely under the teacher.

Specifically, coverage failure means that the student goes off-topic and omits the editing knowledge, so supervision on its own rollouts teaches little about the new knowledge. This calls for a different criterion for when to intervene: HPSE steps in when the privileged model's preferred token is confident and clearly underestimated by the student, which brings in the knowledge that the student would not mention by itself, even when its own token is reasonable. We further analyze how such step-ins provide supervision on the new knowledge.

We have revised the related work to make this clearer.

[1] A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, 2011

[2] HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation, 2026

[3] Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling, 2025

Contribution of hybrid rollouts beyond OPSD+NLL, across more models and with uncertainty.

Following the reviewer's suggestions, we conducted additional experiments comparing HPSE and OPSD+NLL on two more backbones (Llama3.1 and Gemma2) and reported standard errors (detailed below) as a measure of uncertainty. Results (new ones on Llama3.1 and Gemma2, and existing ones on Qwen2.5) are summarized in the table below. We noted that the composition gain of HPSE over OPSD+NLL holds on all three backbones, and becomes larger on Llama3.1 and Gemma2, where OPSD+NLL alone does not improve composition over LoRA. The reported uncertainty also indicates that these improvements are not merely random noise, with the overall composition gain reaching 8.4 pts. For an intuitive explanation, we view OPSD+NLL as a coarse, rollout-level form of step-in, which combines on-policy OPSD with off-policy NLL on the fixed passage, and interpret the empirical gap as the benefit of token-level step-in in HPSE.

We have added the results on more backbones and the discussions to the revised paper.

BackboneMethodUnKE Jnt.UnKE Dmp.UnKE Div.UnKE Avg.MQuAKE Ind.MQuAKE Cmp.MQuAKE Avg.
Qwen2.5LoRA63.2±3.147.8±2.763.0±1.858.0±1.373.3±2.132.0±4.652.6±2.7
OPSD+NLL72.5±3.062.6±2.862.0±1.565.7±1.481.7±1.851.3±4.366.5±2.7
HPSE75.062.560.966.183.254.768.9
Llama3.1LoRA78.5±3.162.5±2.745.0±2.162.0±1.375.8±2.446.7±5.061.3±2.8
OPSD+NLL73.1±2.962.8±2.860.9±2.265.6±1.576.1±2.533.3±5.054.7±2.9
HPSE78.961.959.966.981.050.065.5
Gemma2LoRA85.0±3.369.4±2.634.3±2.462.9±1.479.1±2.141.3±4.560.2±2.7
OPSD+NLL83.9±2.668.3±2.356.0±2.069.4±1.280.8±2.040.7±3.660.8±2.3
HPSE83.167.953.668.281.546.063.7
OverallLoRA75.6±1.959.9±1.647.5±1.361.0±0.876.1±1.340.0±2.758.0±1.6
OPSD+NLL76.5±1.664.6±1.559.6±1.166.9±0.879.6±1.241.8±2.560.7±1.5
HPSE79.064.158.167.181.950.266.1

± denotes the standard error of the per-edit paired difference from HPSE.

Top-16 KL approximation versus full-vocabulary KL.

We adopted the truncated KL design in the literature [1-3] for better computational efficiency and to avoid potential numerical instability and random noise in the tail of the distribution. As suggested by the reviewer, we experimented with full-vocabulary KL. Based on our results, HPSE remains effective under full-vocabulary KL. We also observed lower performance on MQuAKE-uns for both OPSD+NLL and HPSE compared with their top-16 counterparts, suggesting that including the full distribution does not necessarily improve knowledge transfer, possibly due to the noise in its tail. At the same time, we agree with the reviewer that the choice of KK may affect learning. As we did not extensively tune KK, a more suitable choice may further improve performance.

MethodKL targetUnKE Jnt.UnKE Dmp.UnKE Div.UnKE Avg.MQuAKE Ind.MQuAKE Cmp.MQuAKE Avg.
LoRA–63.247.863.058.073.332.052.6
OPSD+NLLtop-1672.562.662.065.781.751.366.5
HPSEtop-1675.062.560.966.183.254.768.9
OPSD+NLLfull72.464.161.365.981.743.362.5
HPSEfull77.462.661.267.083.445.364.4

[1] Entropy-Aware On-Policy Distillation of Language Models, 2026

[2] Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes, 2026

[3] Reinforcement Learning via Self-Distillation, 2026

Thank you for the additional experiments. The composition gains over OPSD+NLL across all three backbones strengthen the case for hybrid rollouts, and the full-vocabulary KL experiment addresses my concern about truncation.

Could you replace only HPSE’s gate with Speculative KD’s top-kk acceptance rule [1] and compare how often the edited facts appear in the rollouts and how accurately the edited model answers multi-hop questions, using the same teacher and comparable intervention rates?

[1] Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling, 2025

Dear Reviewer RUw5,

We are happy to see that our responses addressed your other concerns. We provide our new response below to your follow-up questions.

Comparison with Speculative KD.

Following the reviewer's suggestion, we conducted additional experiments that replace the step-in strategy of HPSE with that of Speculative KD (SKD) [1], using the same teacher. We consider two variants: the default SKD, and HPSE using SKD's step-in strategy (HPSE+SKD). Due to time constraints, we directly adopted the default hyperparameters without tuning. Results on MQuAKE-uns are summarized in the table below. We noted that hybrid rollouts are helpful under both step-in strategies within HPSE, and which strategy works better depends on the backbone. Specifically, on Qwen2.5, SKD performed better than HPSE with a lower step-in rate, and HPSE+SKD, with an even lower step-in rate, performed the best. On the other hand, HPSE performed better on Llama3.1 than the two SKD variants, while HPSE+SKD still outperformed SKD. These results confirm the benefit of step-in within HPSE. The step-in can be done in many ways, and the optimal step-in strategy, as included the revised paper, is still an open question that we didn't address.

BackboneMethodStep-in rateInd.Cmp.Avg.
Qwen2.5LoRA–73.332.052.6
OPSD+NLL0.0%81.751.366.5
SKD4.4%86.758.072.4
HPSE+SKD3.1%88.361.374.8
HPSE13.1%83.254.768.9
Llama3.1LoRA–75.846.761.3
OPSD+NLL0.0%76.133.354.7
SKD3.1%65.131.348.2
HPSE+SKD2.2%79.738.759.2
HPSE9.9%81.050.065.5

Step-in rate is the fraction of rollout tokens replaced by the privileged teacher over all training rounds.

[1] Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling, 2025

How often the edited facts appear in the rollouts.

The “coverage failure” we discussed indicates that the student's response can go off-topic and cover other content that is irrelevant to the editing knowledge, leaving the teacher little to supervise. We allow the teacher to proactively step in, which can have two empirical effects: steering the rollout toward the knowledge to be discussed, and adding more edited facts to it (Fig. 11). Empirically, atomic-fact answer tokens form approximately 17.8% of the step-in tokens in the first-round rollouts of HPSE (UnKEBench, Qwen2.5). On the other hand, an “always step-in” (which should lead to very good fact coverage) does not perform well either (Tab. 2). Moreover, based on our new results above, HPSE+SKD and SKD use fewer step-ins (thereby less teacher content) than HPSE, but can still perform comparably in practice (e.g., better on Qwen2.5). HPSE+SKD has consistently fewer step-ins but better performance than SKD. Considering that early step-ins may cause the trajectories to deviate largely, how often the edited facts appear in the rollouts may not be a good proxy for editing performance.

Reviewer_CLEuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the problem of composability in Unstructured Knowledge Editing (UKE), emphasizing that an edited model should not only memorize newly injected knowledge but also decompose a passage into individual facts (decomposition) and combine related facts to perform multi-hop reasoning (composition). The authors argue that conventional training on fixed passage contexts may lead to context reliance, while pure On-Policy Self-Distillation (OPSD) can suffer from coverage failure because the student's generated trajectories may rarely encounter newly introduced knowledge.

To address these challenges, the paper proposes Hybrid-Policy Self-Editing (HPSE). The method constructs a privileged teacher using a frozen base model conditioned on the editing passage. During student-generated rollouts, the teacher selectively intervenes based on teacher–student prediction discrepancies and teacher confidence. The student is then trained through KL distillation on these hybrid trajectories, with an additional passage-level NLL objective serving as an anchor.

Experiments are conducted on four LLM backbones, two gradient-based editors (FT-M and LoRA), and two benchmarks (UnKEBench and MQuAKE-uns). HPSE improves the aggregate scores across all 16 single-editing configurations, with notable improvements in several decomposition and multi-hop composition metrics.

Critical strengths

1. The paper addresses a meaningful and practically relevant research problem.

The paper extends the objective of UKE beyond simply reproducing injected passages to effectively decomposing and composing newly acquired knowledge. Conventional NLL training on fixed passage prefixes can lead to context reliance, potentially limiting the model's ability to retrieve edited facts under different queries or use them in multi-hop reasoning. Investigating decomposition and composition within a unified composability framework provides a valuable perspective on the limitations of existing knowledge editing methods.

2. HPSE introduces a meaningful mechanism for improving trajectory coverage, supported by extensive experimental results.

To address the coverage failure of OPSD, where student-generated trajectories may fail to reach prefixes associated with newly introduced knowledge, HPSE proposes a selective teacher intervention mechanism based on teacher–student prediction discrepancies and teacher confidence. Rather than merely modifying the distillation objective, the method changes the training prefixes covered during distillation. This approach is compatible with different gradient-based knowledge editors.

Experimental results demonstrate consistent improvements in aggregate scores across all 16 single-editing configurations. For example, on MQuAKE-uns with Qwen2.5 + LoRA, the multi-hop composition score increases from 32.0 to 54.7, representing an absolute improvement of 22.7 percentage points.

Critical weaknesses

1. The independent contribution of the selective teacher intervention mechanism requires more targeted validation.

Although the paper includes several ablation studies, it lacks a controlled comparison with random teacher intervention under a matched intervention budget. Specifically, a baseline that randomly selects intervention positions while maintaining the same number of teacher interventions and using teacher-generated tokens would provide a more direct assessment of the proposed gating mechanism.

Without such a comparison, it remains unclear to what extent the improvements arise from injecting additional teacher-generated tokens, rather than from the specific intervention-position selection strategy based on teacher confidence and teacher–student disagreement.

2. The additional computational overhead is insufficiently quantified.

HPSE requires repeatedly constructing hybrid rollouts involving both the student and the privileged teacher, followed by token-level distillation. The default training configuration consists of five rollout rounds, each involving eight gradient updates, resulting in 40 updates per editing request.

Compared with directly optimizing the passage-level NLL objective, this procedure introduces additional computational overhead. However, the paper does not sufficiently report end-to-end editing time, computational cost, or performance comparisons under matched compute budgets. Consequently, the trade-off between the observed performance improvements and the additional computational expense remains unclear.

Questions for authors

1. Independent benefits of selective intervention: Could the authors include a random teacher intervention baseline with a matched intervention budget? Specifically, this baseline would maintain the same number of teacher interventions as HPSE but randomly select intervention positions, while still using teacher-generated tokens rather than random vocabulary tokens. Such a comparison would help establish whether the proposed confidence- and disagreement-based gating mechanism is necessary.

2. Computational efficiency: Could the authors report the per-edit wall-clock time, training FLOPs, or number of teacher/student forward passes for HPSE compared with NLL and OPSD + NLL? Additionally, could the authors provide performance comparisons under matched computational budgets? These results would help clarify whether HPSE's additional performance gains justify the increased rollout generation and distillation costs.

3 comments

We highly appreciate your effort and time spent reviewing our paper and thank you for your expertise and constructive comments. In the following, we address your comments and questions one by one, which will also be updated in the revised paper.

Comparison with random teacher intervention.

Following the reviewer's suggestions, we conducted an additional ablation that replaces the HPSE gate with random intervention, using teacher-generated tokens and the same step-in budget as HPSE. Results are reported in the table below. We noted that random intervention leads to worse composition, so the benefit of HPSE comes not only from injecting teacher tokens, but also from where the teacher steps in. We have added these results and discussions to the revised paper.

MethodStep-in rate (UnKE / MQuAKE)UnKE Jnt.UnKE Dmp.UnKE Div.UnKE Avg.MQuAKE Ind.MQuAKE Cmp.MQuAKE Avg.
LoRA–63.247.863.058.073.332.052.6
w/o HP0% / 0%72.562.662.065.781.751.366.5
w/o sel (random, matched budget)8.1% / 13.3%75.663.360.366.483.248.065.6
HPSE8.0% / 13.1%75.062.560.966.183.254.768.9

Computational overhead of HPSE and comparison under matched compute budgets.

As suggested by the reviewer, we studied whether the baselines, LoRA and “w/o HP”, can benefit from more compute. To this end, we trained the two baselines with more updates so that their estimated wall-clock time approximately matches that of HPSE, which results in more FLOPs than HPSE. According to the results, additional compute brings only a slight gain to LoRA and no gain to “w/o HP”, which even loses composition on MQuAKE-uns, and neither closes the gap to HPSE. We have added the results and discussions here to the revised paper.

MethodUpdatesWall-clock (s/edit)FLOPs (×10^12)UnKE Jnt.UnKE Dmp.UnKE Div.UnKE Avg.MQuAKE Ind.MQuAKE Cmp.MQuAKE Avg.
LoRA406.217863.247.863.058.073.332.052.6
LoRA425≈66188767.752.460.360.275.334.755.0
w/o HP404438972.562.662.065.781.751.366.5
w/o HP110≈72100268.763.365.665.878.733.356.0
HPSE4071.941875.062.560.966.183.254.768.9

Thanks for the reply, I choose to maintain my positive score.

Dear reviewer CLEu,

Thank you for your reply and for your positive assessment of our paper. We are glad that our responses addressed your questions. Please feel free to share any further suggestions that could help improve the quality of our paper.

Best regards,

Authors

Reviewer_XCKQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies unstructured knowledge editing. The authors find that existing methods struggle to enable models to extract and combine newly injected facts. They also identify a knowledge coverage problem in on-policy self-distillation. To address this, they propose HPSE, which selectively lets the teacher intervene during generation based on teacher confidence and the teacher–student prediction gap. Training combines distillation on these hybrid trajectories with a likelihood objective on the original passage. Experiments across four models and two editing methods show improvements in knowledge decomposition and composition, with gains also observed under continual editing.

Critical strengths
  1. Unstructured knowledge editing is of practical significance and is closer to real-world knowledge update needs.
  2. The research motivation is well-grounded, as it uniformly examines decomposition and composition capabilities and reveals the shortcomings of existing editing methods.
  3. The method is simple and effective, applicable to different knowledge editing methods, and exhibits good generalization.
  4. Experiments cover multiple models and validate the effectiveness of the method in both single and continual editing scenarios.
Critical weaknesses
  1. Existing ablations compare no intervention, selective intervention, and full intervention. Although threshold sensitivity is examined, neither gating condition is removed individually. The experiments therefore do not establish whether teacher confidence and the teacher–student prediction gap are both necessary.
  2. Although the teacher and student originate from the same base model, the teacher additionally receives the editing passage. Prediction gaps may therefore reflect differences in wording or formatting as well as factual content. The paper does not quantify how often interventions introduce or correct factual information rather than merely change its expression.
  3. The paper emphasizes avoiding external teachers, but does not compare self-distillation with offline and online distillation from a stronger teacher given the same editing passage. These comparisons would help establish the practical advantages and limitations of choosing self-distillation.
Questions for authors
  1. When the proportion of tokens generated by the teacher is held approximately constant, does the full gate outperform random teacher intervention, intervention based only on teacher confidence, and intervention based only on the teacher–student prediction gap?
  2. How often do teacher interventions supply missing facts or correct factual errors? How often do they only change wording or formatting without changing the factual content?
  3. Given access to stronger teachers, what motivates choosing self-distillation? Could you compare HPSE with both offline distillation using pre-generated teacher responses and online distillation using teacher supervision on student-generated responses? In both comparisons, the stronger teacher should receive the same editing passage used by HPSE.
2 comments

We highly appreciate your effort and time spent reviewing our paper and thank you for your expertise and constructive comments. In the following, we address your comments and questions one by one, which will also be updated in the revised paper.

Ablation study of teacher confidence, the teacher–student prediction gap, and step-in selection in HPSE.

Following the reviewer's suggestions, we conducted additional ablations that remove each condition individually and compare the HPSE gate with random intervention. Results are reported in the table below. From these results, we noted that keeping these conditions gives better overall performance and particularly benefits composition. These new results, together with the sensitivity analysis on step-in hyperparameters in App. D, confirm the effectiveness of our method.

Namely, our experiments were conducted under the same setting as the ablation study in Sec. 4.4. From the table, removing either the confidence condition (“w/o conf.”) or the prediction-gap condition (“w/o gap”) leads to worse overall performance. We also noticed that these two cases can be seen as the limiting cases of our sensitivity analysis (“w/o conf.” indicates κ=0\kappa=0, and “w/o gap” can be seen as having τ→−∞\tau \rightarrow -\infty). Based on this, we interpret the results here as showing that the two conditions are best used jointly, and one can tune their exact thresholds for better performance. On the other hand, we tested random intervention (“w/o sel”) with the same step-in budget as HPSE, and we noted that this also leads to worse composition. We have added these results and discussions to the revised paper.

MethodUnKE Jnt.UnKE Dmp.UnKE Div.UnKE Avg.MQuAKE Ind.MQuAKE Cmp.MQuAKE Avg.
LoRA63.247.863.058.073.332.052.6
w/o HP72.562.662.065.781.751.366.5
w/o conf.73.859.861.164.983.450.066.7
w/o gap70.854.763.963.179.942.761.3
w/o sel75.663.360.366.483.248.065.6
HPSE75.062.560.966.183.254.768.9

Factual content and the role of step-ins in knowledge coverage.

In our paper, we discussed “coverage failure” that can make OPSD ineffective for UKE: due to the lack of the new knowledge (by definition), and the unawareness of what is being edited in untargeted UKE, the student's response can go off-topic and cover other content that is irrelevant to the editing knowledge. As a result, the teacher can provide little supervision to “correct” the generation. We allow the teacher to proactively step in for correction. This can steer the rollout toward the knowledge to be discussed, and correct errors along the trajectory.

Fig. 11 presents such a coverage-failure example. Given “Introduce Tyler, the Creator”, the student rollout drifts to biographical details unrelated to the editing knowledge. Privileged step-ins such as “is a” steer the rollout toward the supplied knowledge about albums and collaborators. Empirically, step-ins change both which knowledge is discussed and what the content is. Following the reviewer's suggestions, we examined the first-round rollouts (from the unedited student) in our ablation study (UnKEBench, Qwen2.5), and found that atomic-fact answer tokens form approximately 17.8% of the step-in tokens.

We have revised our paper to make this clearer.

The choice of self-distillation versus offline and online distillation from a stronger teacher.

We used self-distillation to have the model itself play the role of teacher instead of requiring an external one. This is more computationally efficient in the sense that only one model needs to be loaded. Following the reviewer's suggestions, we conducted additional experiments on “stronger” teachers. Due to limited computing resources, we use Qwen3-4B as the student. For the offline distillation, we view the original passage (the one the teacher reads for in-context learning) as a given knowledge statement sampled offline from an unknown but high-quality policy. For the online distillation, we use the frozen base model and Qwen3-8B as the teachers, both performing ICL. All other experimental settings follow our ablation study.

Based on the results below, we noted that simply using a stronger teacher for on-policy distillation does not always lead to better performance, and is not sufficient to close the gap to HPSE. This is consistent with our finding in the paper that step-in is important for effective proactive learning in UKE.

MethodUnKE Jnt.UnKE Dmp.UnKE Div.UnKE Avg.MQuAKE Ind.MQuAKE Cmp.MQuAKE Avg.
LoRA (offline)72.063.448.761.469.434.051.7
OPSD51.237.566.851.841.420.030.7
OPD with 8B teacher51.744.367.954.639.614.727.1
HPSE81.566.752.867.082.545.363.9

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revision on the paper, and we thank all reviewers for providing valuable feedbacks.

Dear Reviewers and Area Chairs,

We sincerely appreciate the time and effort you have put into our paper. We are glad to see that the reviewers found our problem meaningful, our method effective, and our experiments extensive. Your comments and suggestions have also been very helpful in improving the quality of our paper. Following them, we have uploaded an updated paper with the following revisions (revisions are colored in blue).

  1. We expanded our ablation study with more variants of the step-in mechanism, more backbones, and comparisons under increased compute budgets. The new results are consistent with our original findings and again support the design of HPSE. We also reorganized this part for better readability.
  2. We refined the description of coverage failure and the role of step-in, extended the discussions on related approaches and open questions, and reworded some less accurate statements.

Additional results that are specific to individual questions are provided in the responses below.

Thank you again for your time. We look forward to hearing from you if you have any additional questions.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.