- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+2)
Read review
Paper summary
In this paper, the authors examine the reproducibility-usability gap in unstructured knowledge editing, in which models can reproduce a document with high accuracy but are unable to answer related questions. To address the identified issue, the authors propose ECoA, a wrapper for existing base editors that applies low-rank adapters to selected layers. The empirical results across two benchmarks and three language models show that ECoA consistently improves the base editors' performance in both single-document and batched document editing.
Critical strengths
- The highlighted issue of the gap between factual recall and actually usable knowledge is a timely and important issue in knowledge editing.
- The authors motivate their paper well and explain their proposed method clearly.
- The empirical analysis of the proposed method covers three language models and two datasets. In addition to editing performance, the authors investigate the effect of the conditioning signal, perform component ablation, and examine the preservation of general capability.
Critical weaknesses
- The questions about the document are highly underspecified. The authors motivate their work by emphasizing the distinction between reproducing knowledge and using it. However, the evaluation does not sufficiently characterize the types of questions used to assess the usability of knowledge. Different question types probe different levels of knowledge use. For example, a question might simply require recalling a fact explicitly stated in the document, while another might require applying the acquired knowledge to solve a more complex reasoning task. As these different levels of knowledge use may exhibit substantially different behavior, it is important for the authors to clarify which aspects of knowledge usability their evaluation captures and align their claims about the capabilities enabled by their method accordingly.
- The empirical analysis evaluates the method in the setting of independently editing single documents and batched documents. The setting of editing only a single document represents a strong simplification compared to real-world settings. In reality, editing sequences of individual documents or batches of documents is the standard use case. This is especially important as sequential updates have been shown to be a real issue for knowledge editing and particularly locate-then-edit methods [1,2].
- The results are missing an RAG baseline. While RAG is not directly comparable to editing, it provides a practical alternative for solving the underlying problem of answering questions relating to a given document. In addition to a RAG baseline, an oracle ICL baseline (where the model answers the given questions with the corresponding document in context) would provide an informative soft upper boundary reference point.
- While I appreciate the authors' effort to evaluate the preservation of general capabilities, the current evaluation is focused on evaluating post-edit language understanding. To align the content of the section with the section title, the authors either need to extend their evaluation using an evaluation suite that covers general capabilities more holistically (e.g., [3]) or change the section title and the corresponding claims from general capabilities to language understanding.
[1] Gupta et al., 2024, Model Editing at Scale leads to Gradual and Catastrophic Forgetting [2] Gupta et al., 2024, Rebuilding ROME : Resolving Model Collapse during Sequential Model Editing [3] Thede et al., 2026, CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training
Questions for authors
- Which type of questions are used to test the usability of knowledge? What level of "knowledge usability" does ECoA facilitate?
5 comments
Response to Reviewer YXxu (2/2)
W3. RAG and oracle ICL baselines
As the reviewer notes, RAG and oracle ICL access external documents at inference time and thus cannot be directly compared with parameter-based editing, but they serve as practical alternatives and reference points for the same problem. We therefore added both baselines. RAG prepends the top-1 document retrieved by BM25 from the 500 edited documents to the question, and oracle ICL directly provides the gold document for the question.
| Method | Document access at inference | ROUGE-L | BERTScore |
|---|---|---|---|
| UnKE+ | ✗ | 0.286 | 0.260 |
| UnKE+ w/ ECoA | ✗ | 0.431 | 0.321 |
| AlphaEdit+ | ✗ | 0.361 | 0.273 |
| AlphaEdit+ w/ ECoA | ✗ | 0.451 | 0.325 |
| RAG | ✓ | 0.697 | 0.491 |
| Oracle ICL | ✓ (gold document) | 0.699 | 0.492 |
Methods that provide external documents at inference time outperform parameter-based editing, and oracle ICL provides a soft upper bound. This result supports the motivation of our work. The same model can answer these questions with a ROUGE-L of 0.699 when the gold document is given in context, and the base editor reproduces the document with high accuracy (AlphaEdit+ 0.956, Figure 1). Nevertheless, the edited model's question performance remains at 0.361. The bottleneck is thus neither the model's ability to answer nor the incorporation of knowledge itself, but the use of knowledge stored in the parameters in new question contexts. ECoA closes 35% (UnKE+) and 27% (AlphaEdit+) of this gap.
RAG performs nearly on par with oracle ICL on this benchmark because BM25 retrieves the gold document for most questions. Meanwhile, RAG and ICL require adding a document to the context for every question and maintaining a document store and retrieval index, whereas parameter-based editing uses knowledge inside the model without additional context. We view the two approaches as complementary methods with different trade-offs and will add this comparison to the paper.
W4. General capability preservation
We agree with the reviewer. Since the current GLUE-based evaluation focuses on language understanding, we will rename Section 4.7 to “Language Understanding Preservation” and restrict the related claims accordingly.
To broaden the evaluation, we also added the MMLU-based evaluation of UnKEBench used in prior work [1].
| Method | MMLU accuracy |
|---|---|
| Pre-edit | 0.732 |
| LoRA | 0.226 |
| FT-L | 0.657 |
| AlphaEdit | 0.700 |
| MEMIT | 0.702 |
| UnKE | 0.730 |
| COIN | 0.732 |
| UnKE+ | 0.734 |
| AlphaEdit+ | 0.566 |
| UnKE+ w/ ECoA | 0.731 |
| AlphaEdit+ w/ ECoA | 0.728 |
UnKE+ w/ ECoA maintains the pre-edit level. The drop for AlphaEdit+ largely stems from output-format failures (valid response ratio 50.1%); after applying ECoA, the valid response ratio recovers to 88.9%, and the accuracy drop on questions valid for both the edited and pre-edit models decreases from 0.070 to 0.017. On the other hand, on GLUE (Figure 4), the mean accuracy of AlphaEdit+ w/ ECoA is 0.024 lower than that of the base editor, and we will report this result in the paper as well.
We will incorporate the above results and revisions into the revised paper. We thank the reviewer again for the careful review.
[1] Deng et al., Everything is Editable: Extend Knowledge Editing to Unstructured Data in Large Language Models, ICLR 2025.
[2] Wu et al., AKEW: Assessing Knowledge Editing in the Wild, EMNLP 2024.
[3] Gupta et al., Model Editing at Scale leads to Gradual and Catastrophic Forgetting, Findings of ACL 2024.
[4] Gupta et al., Rebuilding ROME: Resolving Model Collapse during Sequential Model Editing, EMNLP 2024.
Thank you for the extensive rebuttal. The clarifications and new analyses address most of my concerns. The revision clearly explains what the evaluation measures, and the sequential experiment shows that ECoA improves factual recall over both base editors after 100 edits.
My remaining concern stems primarily from the low absolute performance of the underlying parameter-editing methods. ECoA delivers clear gains over both editors, but those gains start from a low baseline, and performance remains limited, particularly on multi-hop questions. I would appreciate more discussion of which use cases the resulting models could realistically support at the demonstrated performance level.
Response to Reviewer YXxu (1/2)
We thank the reviewer for the positive assessment of the importance of the reproduction–usability gap and of ECoA’s motivation and methodology. The reviewer’s points on the clarity of the evaluation scope, the connection to real-world settings, and the need for reference baselines have helped us make the paper's claims more precise.
W1 & Q1. Types of evaluation questions and the scope of knowledge usability
As the reviewer points out, knowledge usability spans multiple levels, from direct factual recall to reasoning that applies acquired knowledge, and the current paper does not describe the scope of our evaluation clearly enough. We agree and clarify it as follows.
(1) Types of evaluation questions
Our evaluation uses the fact-specific questions provided by the three benchmarks, each of which asks about a single fact stated in the edited document in a form different from the original editing prompt. The UnKEBench questions were constructed by decomposing each document into sub-question and sub-answer pairs via knowledge decomposition [1], and the AKEW-CounterFact questions ask about the counterfactual fact contained in the document (the object of a subject–relation–object triple) [2]. For AKEW-MQuAKE, the paper uses single-hop sub-questions. For example, the document in Appendix C.7 states that Nathan Purdee managed the cultivation and harvesting of grapes, berries, and other fruits, and the evaluation question is “What crops has Nathan Purdee managed the cultivation and harvesting of?” The evaluation questions are not used for training; ECoA is trained only on continuation and cloze views constructed from the target document.
(2) Level supported by ECoA
The level directly demonstrated by our results is therefore single-hop factual use: recalling an individual fact stated in the edited document in response to a new question, without the original context. To examine levels beyond this, we also evaluated the multi-hop questions of AKEW-MQuAKE, which require composing multiple facts (Qwen2.5-7B-Instruct).
ECoA improves ROUGE-L on multi-hop questions by 0.220 and 0.116, respectively. However, ECoA’s training views do not directly train the composition of multiple facts, and absolute performance is also lower than on fact-specific questions.
Accordingly, we will replace the term “knowledge usability” with “query-conditioned factual use” in the paper and state explicitly that our claims concern the ability to use individual facts of an edited document in new question contexts. We will describe the results on complex reasoning, together with the multi-hop analysis above, as a limitation.
W2. Sequential editing
As the reviewer points out, edits are made sequentially in real-world settings, and the fact that successive updates cause forgetting and model collapse in locate-and-edit methods [3,4] is an important consideration. We agree and have added a sequential editing experiment.
We sequentially edited 100 UnKEBench documents one at a time, updating the adapters at each step with conditioning on the cumulative update Δt=θt−θ0, and evaluated on all edited documents after the final step (Qwen2.5-7B-Instruct).
In sequential editing as well, ECoA improves the ROUGE-L of both base editors by about 2×. This appears to be because ECoA does not modify the edited base weights and updates only separate adapters, so it does not accumulate additional modifications on top of the base-weight changes caused by successive updates. We will add this result to the paper.