acceptodds
Under review as a conference paper at ICLR 2027

ResidualKV: Dynamic Knowledge Editing via an Editable Attention Key-Value Space

Abstract

Knowledge editing aims to revise specific facts in large language models (LLMs) without costly full-model retraining. Existing editors largely follow a static editing paradigm, performing localized factual updates in feed-forward network (FFN) layers. While effective, this fixed editing substrate can limit factual writing and generalization across query reformulations. To overcome this limitation, we introduce Residual Key-Value Editing (ResidualKV), a dynamic knowledge-editing framework that rethinks factual writing and access through attention. ResidualKV stores corrections in a persistent editable key-value (KV) space and dynamically invokes them through attention during inference, enabling stronger factual writing and improved generalization across query reformulations. Further, to reduce interference from editable-KV writing, we propose Factual Residual Distillation (FRD), which distills only the required old-to-new factual transition into the editable KV space rather than reconstructing the complete factual state. By avoiding redundant encoding, FRD uses the limited KV capacity more efficiently, reducing disruption to pretrained model behavior while strengthening factual writing. Experiments across three recent LLM backbones and two benchmarks demonstrate strong editing efficacy and generalization, competitive post-edit fluency, and robust large-scale sequential editing. Further ablation and mechanistic analyses support the proposed design.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Knowledge Editing, Large Language Models, Attention Key-Value Space, Factual Residual Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xGYPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ResidualKV, a knowledge editing method that stores factual corrections in trainable attention KV states while keeping the pretrained model frozen. Factual Residual Distillation (FRD) aligns the edit-induced predictive change with an old-to-new factual transition. Independently trained KV modules are stored in a bank and selected by a router at inference. Experiments evaluate editing effectiveness, generalization, capability preservation, and scalability across three model backbones and two benchmarks.

Critical strengths
  1. The paper is clearly written, with well-organized explanations of the editing mechanism and training objective.

  2. Attention KV editing and factual residual distillation offer meaningful novelty, supported by extensive experiments and ablations. The scalability evaluation with increasing edit batch sizes and accumulated edits is particularly thorough.

Critical weaknesses
  1. The evaluation protocol raises serious concerns. All three CounterFact metric descriptions appear to differ from AlphaEdit’s [1] original protocol without explicit justification. Several baselines also perform substantially worse than their published results. The paper does not explain why the evaluation change affects different methods so unevenly, making the reported comparative gains difficult to interpret.

[1] https://arxiv.org/abs/2410.02355

Questions for authors
  1. What motivates the apparent departure from AlphaEdit’s original evaluation protocol? Could you also report results using that protocol?

  2. Why does changing the evaluation protocol appear to cause much larger performance drops for some methods than others? Could controlled experiments or theoretical analysis explain this disparity and clarify what it reveals about the methods’ editing behavior?

6 comments
AuthorsOpenReview ↗

Response to Reviewer xGYP (1/2)

Dear Reviewer xGYP,

We sincerely thank you for your constructive comments and recognition of the novelty of our method and extensive experimental evaluations. Following your suggestions, we have conducted additional experiments and provide the following clarifications.

Q1. Evaluation Protocol Clarification

Thank you for raising this concern. Our CounterFact evaluation follows EasyEdit [2], a widely adopted knowledge editing framework, rather than a customized protocol. Unlike AlphaEdit [1], which compares the probabilities of new and old answers, EasyEdit evaluates target-answer prediction accuracy. For single-token answers, the distinction is:

\;\nRightarrow\; o_{\mathrm{new}}=\arg\max_{v\in\mathcal V}P_\theta(v\mid x).$$ Under AlphaEdit's protocol, a higher probability for the new answer than the old one does not necessarily indicate successful editing, as other answers may still be more probable. **This may overestimate actual editing success**, motivating our use of EasyEdit to directly assess whether the model correctly predicts the target answer. Following your suggestion, we additionally evaluated both AlphaEdit and ResidualKV under AlphaEdit's original protocol. As shown in Table 1, both methods obtain substantially higher scores, suggesting that relative-probability evaluation may overestimate actual editing success. Nevertheless, ResidualKV still consistently outperforms AlphaEdit across all three metrics on all three backbones. We have clarified these evaluation differences in Appendix A.2 of the revised manuscript. **Table 1: Comparison under AlphaEdit's original CounterFact evaluation protocol (%).** | Backbone | Method | Efficacy ↑ | Generalization ↑ | Specificity ↑ | |---|---|---:|---:|---:| | Llama-3.1-8B | AlphaEdit | 88.80 | 80.45 | 56.78 | | | **ResidualKV** | **99.70** | **89.11** | **89.25** | | Qwen3-8B | AlphaEdit | 80.75 | 51.03 | 49.19 | | | **ResidualKV** | **98.65** | **60.80** | **83.16** | | Phi-4-mini-3.8B | AlphaEdit | 87.65 | 60.58 | 64.14 | | | **ResidualKV** | **99.35** | **69.48** | **80.41** | **References** [1] Fang et al. *AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models.* ICLR, 2025. [2] Wang et al. *EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models.* ACL, 2024.
AuthorsOpenReview ↗

Response to Reviewer xGYP (2/2)

Q2. Evaluation Sensitivity

Thank you for this insightful question. We attribute the observed discrepancies primarily to evaluation protocols and backbone sensitivity.

1. Evaluation Protocol. As clarified in Q1, relative-probability evaluation may count edits as successful even when the target answer is not the top prediction. This may overestimate actual editing success, as further illustrated by our additional results in Table 1.

2. Backbone Sensitivity. Knowledge editing methods exhibit substantial performance variations across model backbones. Existing studies, such as AlphaEdit [1], primarily evaluate earlier models, including GPT2-XL, GPT-J, and LLaMA3-8B. In contrast, we evaluate three more recent backbones: Llama-3.1-8B, Qwen3-8B, and Phi-4-mini-3.8B. Table 2 further compares our results with AlphaEdit [1] and Ren et al. [3].

Table 2: Comparison of sequential editing Efficacy on ZsRE (%).

StudyBackbone# EditsROMEMEMITAlphaEdit
AlphaEdit [1]LLaMA3-8B2,0002.0134.6294.47
Ren et al. [3]Llama-3.1-8B1,0000.000.0074.00
OursLlama-3.1-8B2,0000.160.0080.58

As shown in Table 2, Ren et al. [3] independently report near-zero Efficacy for ROME and MEMIT on Llama-3.1-8B, consistent with our observations. Notably, the performance gap between LLaMA3-8B and Llama-3.1-8B suggests strong backbone sensitivity in existing editing methods. Our experiments reveal similar trends across model families: AlphaEdit achieves 80.58% and 72.59% Efficacy on Llama-3.1-8B and Qwen3-8B, respectively, but drops sharply to 4.24% on Phi-4-mini, whereas ROME and MEMIT perform better on Phi-4-mini than on Llama-3.1-8B. In contrast, ResidualKV consistently achieves strong performance across all three backbones, demonstrating greater cross-backbone robustness.

We sincerely appreciate your constructive feedback. We hope that our clarifications and additional experimental results address your concerns and that you will reconsider your assessment of our work.

References

[1] Fang et al. AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models. ICLR, 2025.

[2] Wang et al. EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models. ACL, 2024.

[3] Ren et al. Revisiting Parameter-Based Knowledge Editing in Large Language Models: Theoretical Limits and Empirical Evidence. ICML, 2026.

Thank you for your detailed response. I have one further question: why does ResidualKV substantially outperform AlphaEdit under the current evaluation metrics? I find this observation quite interesting and would appreciate the authors’ discussion on this point.

Reviewer_pa3EOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

A method is introduced for "editing facts" in a language model without changing its weights. For each batch of edits it trains a small set of extra attention keys and values (prefixes) at layers 20–24, while the backbone stays frozen. The model’s own keys and the added keys are concatenated together in the same attention softmax. A separately trained MLP router looks at the mean-pooled hidden states of the subject tokens and either selects a signle batch of edits or rejects all of them.

Critical strengths
  1. Large reported improvements across multiple backbones.

  2. Appealing modularity and initial routing evidence: Independent states can be detached without modifying backbone weights, and new writes do not directly overwrite earlier states.

Critical weaknesses
  1. The method is not model editing in the useful sense, and the obvious baseline is missing. Editing weights matters because the result is a drop-in replacement for the original model. ResidualKV does not produce one: the model stays frozen, and the system adds a bank of stored states plus a router that decides, for each query, whether an edit applies. Once such a router exists, the simplest solution is to have it retrieve the edited fact as text and place it in the prompt. The paper cites context-based editors such as IKE but does not compare with any of them. FRD itself is trained on exactly this signal: its targets come from the frozen model with the old or new fact placed in context, so the in-context model is the teacher that the stored states imitate. The paper should (a) compare against the same router followed by the fact, or the facts of the selected batch, in context, and against an LLM that is simply asked whether the query concerns an edited fact, reporting accuracy, context length and latency; and (b) explain what the learned key-value states offer beyond that baseline. Without this, the motivation for the method is unclear.

  2. Matched controls that share the router are also missing. First, plain prefix tuning with the same router, layers and prefix length: the target-only KV row of Tables 2 and 4 may be this, but it is not described that way, and the FFN row is not defined. Second, FRD applied to a LoRA or FFN editor with the same router, to separate the loss from where the edit is stored. Third, the closest prior systems of this kind: RECIPE [1], Cartridges [2] and GRACE [3].

  3. Benchmark annotations may supply subject spans that are unavailable in deployment, and the paper does not say whether gold spans are used at inference.

4.Causal subject-token states cannot see relation words that follow them. Two queries sharing a prefix through the subject but asking different subsequent relations can therefore have identical routing features. The selected state can still respond differently downstream, but the router itself cannot tell such queries apart. Same-subject/different-relation, alias, and pronoun tests are needed.

  1. Table 1 reports MEMIT at 0.00/0.00 efficacy/generalization on Llama for both datasets, AlphaEdit at 0.15/0.20 on Qwen/CounterFact, and UltraEdit [6] at 0.00/0.00 on Phi/ZsRE. Degradation under sequential editing is expected, but scores of exactly zero need an explanation.

  2. Table 1 reports the authors’ own reruns of the baselines, and they are far below the published numbers. On Llama-3 with 2,000 edits, AlphaEdit [4] reports 94.47/91.13 efficacy/generalization on ZsRE and 98.90/94.22 on CounterFact, and NeuralDB [5] reports 96.3/92.0 and 99.9/86.6. Table 1 gives AlphaEdit 80.58/70.91 and 66.53/42.85, and NeuralDB 29.50/28.48 and 3.65/2.22, on Llama-3.1. MEMIT is 0.00/0.00 in Table 1 but 34.62/31.28 (ZsRE) and 65.65/64.65 (CounterFact) in [4]. Against the published values, ResidualKV’s 91.39/91.02 and 91.12/90.85 would not be the best result. The metrics also differ: published ZsRE specificity is about 32 for these methods, whereas Table 1 reports 69–100.

  3. The distinction between static FFN editing and dynamic attention editing needs qualification. A fixed FFN parameter update can have input-dependent effects.

  4. Limited novelty. The paper explicitly acknowledges prefix tuning when introducing Equation 4. Concatenating persistent learned keys and values to native attention is therefore an adopted parameterization. The mixture decomposition in Equation 6 is an algebraic property of it: He et al. [8, Section 3.1, Equation 7] already express the mixture of native and prefix attention with input-dependent weights. This identity should be attributed; the more credible contribution is its use for factual editing with the particular residual-distillation objective and routed batch storage.

RECIPE [1] encodes factual statements into continuous input prompts and uses a learned retriever with a Knowledge Sentinel for selective application and fallback. Its encoder/retriever and sentinel differ from ResidualKV’s independently optimized layerwise KVs and fixed-threshold batch classifier, but the common retrieved-prompt structure makes it a close editing baseline.

Cartridges [2] freezes an LLM, directly optimizes persistent prefix-style KVs, and distills a context-conditioned teacher into the cache-conditioned model using a token-distribution KL loss. Its task is amortized long-context representation rather than sequential counterfactual editing, and it does not establish FRD’s old/new residual or this router. It nevertheless provides a close precedent for the writable state and FSD control.

SERAC [9] combines an edit cache, scope classifier, counterfactual predictor, and base-model fallback. GRACE [3] uses an activation-space codebook with distance-based deferral and learned replacement values. Larimar [10] uses a separately trained episodic-memory architecture with least-squares memory operations. These are precedents for selective, query-dependent memory editing, although their architectures and adaptation requirements differ from a shared attention softmax in an arbitrary frozen decoder.

DistillMIKE [11] distills retrieved in-context edits into LoRA, while Kujanpää et al. [12] transfer a knowledge-conditioned teacher into an unconditioned LoRA student. DeCK [13] contrasts contextual and parametric predictions during decoding; DISCO [7] uses differences between original and edited predictions for reasoning.

RECIPE, SERAC, GRACE, Larimar, He et al., DistillMIKE, and Cartridges are absent from the submission’s text and references.

References

[1] Qizhou Chen, Taolin Zhang, Xiaofeng He, Dongyang Li, Chengyu Wang, Longtao Huang, and Hui Xue, "Lifelong Knowledge Editing for LLMs with Retrieval-Augmented Continuous Prompt Learning," in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 13565–13580, 2024. link.

[2] Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Will Tennien, Atri Rudra, James Zou, Azalia Mirhoseini, and Christopher Re, "Cartridges: Lightweight and general-purpose long context representations via self-study," arXiv:2506.06266, 2025.

[3] Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi, "Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors," in Advances in Neural Information Processing Systems 36 (NeurIPS), pp. 47934–47959, 2023. link.

[4] Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Shi Jie, Xiang Wang, Xiangnan He, and Tat-seng Chua, "AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models," in International Conference on Learning Representations (ICLR), 2025. arXiv:2410.02355.

[5] Weizhi Fei, Hao Shi, Jing Xu, Jingchen Peng, Jiazheng Li, Jingzhao Zhang, Bo Bai, Wei Han, Zhenyuan Chen, and Xueyan Niu, "NeuralDB: Scaling Knowledge Editing in LLMs to 100,000 Facts with Neural KV Database," arXiv:2507.18028, 2025.

[6] Xiaojie Gu, Ziying Huang, Jia-Chen Gu, and Kai Zhang, "UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Models," Transactions on Machine Learning Research (TMLR), 2026. arXiv:2505.14679.

[7] Zengkui Sun, Yijin Liu, Jiaan Wang, Fandong Meng, Jinan Xu, Yufeng Chen, and Jie Zhou, "Outdated Issue Aware Decoding for Factual Knowledge Editing," in Findings of the Association for Computational Linguistics: ACL 2024, pp. 9282–9293, 2024. link.

[8] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig, "Towards a Unified View of Parameter-Efficient Transfer Learning," in International Conference on Learning Representations (ICLR), 2022. arXiv:2110.04366.

[9] Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn, "Memory-Based Model Editing at Scale," in International Conference on Machine Learning (ICML), PMLR 162, pp. 15817–15831, 2022. link.

[10] Payel Das, Subhajit Chaudhury, Elliot Nelson, Igor Melnyk, Sarath Swaminathan, Sihui Dai, Aurélie Lozano, Georgios Kollias, Vijil Chenthamarakshan, Jiří Navrátil, Soham Dan, and Pin-Yu Chen, "Larimar: Large Language Models with Episodic Memory Control," in International Conference on Machine Learning (ICML), 2024. arXiv:2403.11901.

[11] Shanbao Qiao, Xuebing Liu, and Seung-Hoon Na, "DistillMIKE: Editing Distillation of Massive In-Context Knowledge Editing in Large Language Models," in Findings of the Association for Computational Linguistics: ACL 2024, pp. 7639–7654, 2024. link.

[12] Kalle Kujanpää, Pekka Marttinen, Harri Valpola, and Alexander Ilin, "Efficient Knowledge Injection in LLMs via Self-Distillation," arXiv:2412.14964, 2024.

[13] Baolong Bi, Shenghua Liu, Lingrui Mei, Yiwei Wang, Junfeng Fang, Pengliang Ji, and Xueqi Cheng, "Decoding by Contrasting Knowledge: Enhancing Large Language Model Confidence on Edited Facts," in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pp. 17198–17208, 2025. link.

Questions for authors
  1. Motivate the work
  2. Address the many missing baselines
  3. Clarify the empirical discrepencies from published work
7 comments
AuthorsOpenReview ↗

Response to Reviewer pa3E (1/4)

Dear Reviewer pa3E,

We sincerely thank you for your detailed review and constructive feedback. To address your concerns regarding novelty and baseline comparisons, we provide the following clarifications and additional experimental evidence.

Q1. Comparison with Adapter-Based and Retrieval-Based Editing

We respectfully clarify that knowledge editing is not limited to modifying pretrained model parameters. As discussed in our Related Work, existing approaches can be broadly categorized into parameter-based, adapter-based, and context-based editing. In particular, adapter-based methods, such as MELO [1], WISE [2], and NeuralDB [3], preserve pretrained parameters while storing factual corrections in auxiliary trainable modules. This separation helps reduce interference with pretrained knowledge and enables modular editing. ResidualKV follows this parameter-preserving direction but introduces a new perspective on knowledge editing, shifting from static FFN parameter rewriting to dynamic factual correction through persistent, editable attention KVs.

To address your concern regarding retrieval-based alternatives, we additionally evaluated IKE [4]. Following EasyEdit's implementation, we directly prepend the corresponding edited fact to each input prompt, providing IKE with ground-truth fact retrieval (100% correct fact selection). As shown in Tables 1 and 2, ResidualKV consistently outperforms IKE in Efficacy and Generalization across all three backbones on ZsRE and CounterFact, even under this favorable setting for IKE.

Table 1: Comparison with IKE on ZsRE (%).

BackboneMethodEfficacy ↑Generalization ↑
Llama-3.1-8BIKE52.8451.98
ResidualKV91.3991.02
Qwen3-8BIKE78.0069.80
ResidualKV84.2782.75
Phi-4-mini-3.8BIKE78.9470.29
ResidualKV89.5875.00

Table 2: Comparison with IKE on CounterFact (%).

BackboneMethodEfficacy ↑Generalization ↑
Llama-3.1-8BIKE80.0061.25
ResidualKV91.1290.85
Qwen3-8BIKE81.0544.75
ResidualKV89.3051.90
Phi-4-mini-3.8BIKE84.7828.41
ResidualKV91.9036.13

We further compared inference efficiency on Phi-4-mini with CounterFact. As shown in Table 3, ResidualKV achieves 2.03× faster inference (147.1 vs. 298.0 ms/batch) with lower peak memory (15,236.1 vs. 15,665.4 MiB). Notably, both methods use comparable edit representation lengths: approximately 14 text tokens for IKE versus 12 editable KV slots for ResidualKV. Unlike IKE, which processes the additional text tokens through all transformer layers, ResidualKV injects learned KVs only at layers 20–24, avoiding the overhead of processing edited facts as textual inputs throughout the model.

Table 3: Inference efficiency on Phi-4-mini-3.8B with CounterFact.

MethodInference Time (ms/batch) ↓Peak Memory (MiB) ↓
IKE298.015,665.4
ResidualKV147.115,236.1

Crucially, providing the correct fact does not necessarily ensure successful factual correction. Even with ground-truth fact retrieval, IKE relies on the frozen model to apply the provided fact, without explicitly optimizing its predictions toward the desired correction. In contrast, FRD explicitly learns the predictive change between old and new facts, optimizing only a compact set of editable KVs to induce the desired correction through attention. The results in Tables 1–3 demonstrate that ResidualKV achieves higher editing efficacy and generalization than IKE while incurring lower inference overhead. We have incorporated these additional experimental results and the corresponding analyses into Section 5.2 and Appendix C.10 of the revised manuscript.

References

[1] Yu et al. MELO: Enhancing Model Editing with Neuron-Indexed Dynamic LoRA. AAAI, 2024.

[2] Wang et al. WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models. NeurIPS, 2024.

[3] Fei et al. NeuralDB: Scaling Knowledge Editing in LLMs to 100,000 Facts with Neural KV Database. ICLR, 2026.

[4] Zheng et al. Can We Edit Factual Knowledge by In-Context Learning? EMNLP, 2023.

AuthorsOpenReview ↗

Response to Reviewer pa3E (2/4)

Q2. Matched Controls and Related Methods

Thank you for these valuable suggestions. We clarify the matched controls and comparisons with related methods as follows.

1. Matched Controls. We apologize for the insufficient descriptions of our ablation settings. The FFN row represents FFN-based editing with the same router, while the target-only KV row corresponds to plain prefix tuning with the same router, edited layers, and KV length. Our original ablations show that even without FRD, editable KVs substantially outperform FFN-based editing, supporting the effectiveness of the KV editing substrate.

We additionally evaluated FFN+FRD under the same routing setting. As shown in Table 4, FFN+FRD improves upon the original FFN baseline but still underperforms KV+FRD. These results demonstrate that FRD is effective beyond the KV editing substrate, while KV-based editing provides additional performance gains. We have clarified these settings and added the corresponding results to Section 5.5 (Tables 3 and 5).

Table 4: Matched comparison of editing substrates and FRD across three backbones on ZsRE (%).

BackboneMethodEfficacy ↑Generalization ↑
Llama-3.1-8BFFN75.8661.25
FFN + FRD78.6062.30
KV + FRD90.3089.25
Qwen3-8BFFN66.9851.77
FFN + FRD72.7565.23
KV + FRD81.0878.91
Phi-4-mini-3.8BFFN66.2958.57
FFN + FRD74.8865.74
KV + FRD89.1972.82

2. Comparisons with Adapter-Based and Retrieval-Based Methods. Regarding GRACE [7] (NeurIPS 2023), our original experiments already extensively evaluate representative adapter-based editors, including WISE [2] (NeurIPS 2024) and NeuralDB [3] (ICLR 2026). These methods employ auxiliary editing mechanisms while preserving pretrained parameters, providing broad experimental coverage of adapter-based knowledge editing.

For RECIPE [5], a closely related retrieval-augmented continuous-prompt editor, we additionally conducted direct comparisons on ZsRE and CounterFact. As shown in Table 5, RECIPE consistently underperforms ResidualKV, further demonstrating the effectiveness of our approach compared with retrieval-based continuous-prompt editing. We have incorporated these results into Section 5.2 (Main Results) of the revised manuscript.

Table 5: Comparison with RECIPE on ZsRE and CounterFact (%).

BackboneMethodZsRE Eff. ↑ZsRE Gen. ↑CounterFact Eff. ↑CounterFact Gen. ↑
Llama-3.1-8BRECIPE41.2540.019.152.48
ResidualKV91.3991.0291.1290.85
Qwen3-8BRECIPE40.8138.844.331.68
ResidualKV84.2782.7589.3051.90
Phi-4-mini-3.8BRECIPE36.4536.285.982.01
ResidualKV89.5875.0091.9036.13

3. Cartridges and Distillation. We acknowledge the relevance of Cartridges [6] and have cited it in the revised manuscript. Cartridges learns persistent KVs through context distillation, using KL divergence to match the teacher's predictive distribution for long-context compression. Importantly, our original ablations already include a closely related KL-based Full-State Distillation (FSD) objective under the same editable-KV architecture. Unlike FSD, which matches the complete new-fact predictive distribution, FRD focuses on the old-to-new factual transition, avoiding redundant supervision over information already captured by the frozen model.

As shown in Table 3 of our manuscript, FRD consistently improves Efficacy, Generalization, and Fluency over FSD. For example, Efficacy/Generalization improve from 87.12/87.35 to 90.30/89.25 on Llama-3.1-8B and from 77.82/75.59 to 81.08/78.91 on Qwen3-8B. Furthermore, the PCA visualizations in Figure 5 and Appendix C.5 show that FRD induces substantially smaller representation shifts than FSD, suggesting that focusing on the old-to-new factual residual enables more effective factual correction with less disruption to pretrained representations.

References

[2] Wang et al. WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models. NeurIPS, 2024.

[3] Fei et al. NeuralDB: Scaling Knowledge Editing in LLMs to 100,000 Facts with Neural KV Database. ICLR, 2026.

[5] Chen et al. Lifelong Knowledge Editing for LLMs with Retrieval-Augmented Continuous Prompt Learning. EMNLP, 2024.

[6] Eyuboglu et al. Cartridges: Lightweight and General-Purpose Long Context Representations via Self-Study. ICLR, 2026.

[7] Hartvigsen et al. Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors. NeurIPS, 2023.

AuthorsOpenReview ↗

Response to Reviewer pa3E (3/4)

Q3–Q4. Subject Identification and Routing Robustness

Thank you for identifying these concerns. We apologize for the unclear description of our routing mechanism. Our implementation does not rely on benchmark-provided gold subject spans at inference. Instead, the router combines stored subject-name embeddings with full-query embeddings, allowing routing decisions to consider both subject identity and relation information, rather than relying solely on causal subject-token representations.

Following your suggestion, we selected challenging cases from ZsRE and CounterFact to evaluate routing robustness under subject-alias and same-subject/different-relation settings. As shown in Table 6, the router maintains high routing accuracy on both datasets, demonstrating its robustness and stability in selecting the appropriate editable KV states.

Table 6: Routing accuracy on challenging queries from ZsRE and CounterFact (%).

DatasetRouting Accuracy ↑
ZsRE95.60
CounterFact98.60

Q5–Q6. Baseline Performance and Evaluation Discrepancies

Thank you for raising these important concerns. Our baselines follow the recommended configurations in EasyEdit [8] and the original studies, including method-specific hyperparameters such as NeuralDB's cosine-similarity gating threshold (γ=0.65\gamma=0.65). The main differences from earlier studies are our use of newer backbones (Llama-3.1-8B, Qwen3-8B, and Phi-4-mini-3.8B) and an evaluation protocol that differs slightly from AlphaEdit [9].

1. Evaluation Protocol. On CounterFact, AlphaEdit evaluates editing success by comparing the probabilities of new and old answers, whereas our EasyEdit evaluation uses target-answer prediction accuracy. Importantly, a higher probability for the new answer than the old one does not necessarily indicate a correct prediction, as other answers may still be more probable. To ensure a fair comparison, we additionally evaluated both methods under AlphaEdit's original protocol. As shown in Table 7, ResidualKV consistently outperforms AlphaEdit across all three evaluation metrics on all three backbones, demonstrating its superior overall editing performance even under AlphaEdit's original evaluation protocol.

Table 7: Comparison under AlphaEdit's original CounterFact evaluation protocol (%).

BackboneMethodEfficacy ↑Generalization ↑Specificity ↑
Llama-3.1-8BAlphaEdit88.8080.4556.78
ResidualKV99.7089.1189.25
Qwen3-8BAlphaEdit80.7551.0349.19
ResidualKV98.6560.8083.16
Phi-4-mini-3.8BAlphaEdit87.6560.5864.14
ResidualKV99.3569.4880.41

2. Backbone Sensitivity. Knowledge editing performance varies across model backbones. Notably, Ren et al. [10] (ICML 2026) also evaluate Llama-3.1-8B-Instruct and report near-zero scores for several parameter-based editors, consistent with our observations. Such degradation may partly result from interference during sequential editing, reflecting the limited cross-backbone robustness of existing methods. In contrast, ResidualKV consistently achieves strong performance across different backbones and datasets, demonstrating its broader applicability.

References

[8] Wang et al. EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models. ACL, 2024.

[9] Fang et al. AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models. ICLR, 2025.

[10] Ren et al. Revisiting Parameter-Based Knowledge Editing in Large Language Models: Theoretical Limits and Empirical Evidence. ICML, 2026.

Reviewer_WEUVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a knowledge editing framework called ResidualKV, which claims to address the poor generalization ability of traditional methods when facing rephrased queries. The core mechanism of the method can be understood as a form of RAG operating within the model's internal attention layers: the model maintains an external vector memory store, and during inference, it retrieves vectors stored in KV-cache format from this external memory based on the input query's features, which then participate in the attention computation to intervene in the model's behavior. These KV-caches are learned through a strategy similar to prefix tuning, with the training objective of distilling the residual difference between the model's outputs for old and new knowledge. Experiments demonstrate strong performance of the proposed method.

Critical strengths
  • The paper has clear presentation, articulating the motivation effectively, and the proposed method follows naturally from it.
  • The method demonstrates strong performance, achieving good results on large-scale editing tasks.
Critical weaknesses
  • Overall, the mechanism of this paper falls under memory-based methods, and its ability to handle sequential editing is trivial, since it only involves modifications to an external knowledge store without needing to consider complex model parameter updates. Therefore, it should be more extensively compared with methods of the same category, such as RAG-based approaches, rather than primarily against parameter-modifying knowledge editing methods. The current baseline selection appears somewhat unfair.

  • Furthermore, I am concerned about whether the distillation-based mechanism for obtaining KV vectors may exhibit disadvantages compared to standard prompt-based RAG on other tasks. For example, a standard RAG approach concatenates knowledge-relevant prompts as prefixes, and models leveraging in-context learning in this setting can often handle complex tasks well, such as multi-hop reasoning. In contrast, since the residual KV vectors are distilled from simple editing samples, it remains unclear whether these KV-caches can be effectively utilized by the model for complex tasks such as multi-hop reasoning when such tasks are encountered as inputs.

  • The evaluation setting of this paper does not adequately support its claims. A core issue highlighted in the introduction is the paraphrase task, yet this is not a particularly challenging problem on the ZsRE and CounterFact datasets, as the paraphrased queries in these datasets share the same subject as the editing samples. Traditional methods can already achieve strong performance on paraphrase under the single-edit setting, and the poor performance of other methods reported in Section 5.2 under the sequential setting is more attributable to the sequential editing configuration rather than paraphrase difficulty per se. To properly support claims regarding paraphrase robustness, evaluations on harder tasks (e.g., subject aliases) and direct comparisons under single-edit settings are necessary.

Questions for authors

How do the distilled KV vectors perform when the edited knowledge needs to be recalled during multi-hop reasoning (e.g., answering "Who is the spouse of the president of the USA?")? Have the authors evaluated whether ResidualKV supports compositional reasoning beyond direct single-hop QA?

3 comments
AuthorsOpenReview ↗

Response to Reviewer WEUV (1/2)

Dear Reviewer WEUV,

We sincerely thank you for your constructive comments and recognition of our method and extensive evaluations. Following your suggestions, we provide the following clarifications and additional experiments.

Q1. Comparison with Memory-Based Methods

We acknowledge that ResidualKV shares characteristics with memory-based editing methods, as it stores factual corrections in independent trainable KV modules. We would like to clarify that our original experiments already include related methods, such as WISE [1] and NeuralDB [2], which also employ auxiliary memory mechanisms for knowledge editing. However, ResidualKV introduces a new perspective on knowledge editing, shifting from static FFN parameter rewriting to dynamic factual correction through editable attention KVs, which are accessed according to the input query. This distinction represents one of our key contributions.

To further address your concern regarding retrieval-based comparisons, we additionally evaluated IKE [3], a retrieval-based in-context editing method. As shown in Tables 1 and 2, ResidualKV consistently outperforms IKE in Efficacy and Generalization across all three backbones on ZsRE and CounterFact. Moreover, ResidualKV achieves 2.03× faster inference with lower peak memory on Phi-4-mini, as shown in Table 3. These results further support the effectiveness and efficiency of our editing approach.

Table 1: Comparison with IKE on ZsRE (%).

BackboneMethodEfficacy ↑Generalization ↑
Llama-3.1-8BIKE52.8451.98
ResidualKV91.3991.02
Qwen3-8BIKE78.0069.80
ResidualKV84.2782.75
Phi-4-mini-3.8BIKE78.9470.29
ResidualKV89.5875.00

Table 2: Comparison with IKE on CounterFact (%).

BackboneMethodEfficacy ↑Generalization ↑
Llama-3.1-8BIKE80.0061.25
ResidualKV91.1290.85
Qwen3-8BIKE81.0544.75
ResidualKV89.3051.90
Phi-4-mini-3.8BIKE84.7828.41
ResidualKV91.9036.13

Table 3: Inference efficiency on Phi-4-mini-3.8B with CounterFact.

MethodInference Time (ms/batch) ↓Peak Memory (MiB) ↓
IKE298.015,665.4
ResidualKV147.115,236.1

Q2. Multi-Hop Reasoning

Thank you for highlighting the importance of complex reasoning. We acknowledge that our original evaluation did not adequately cover multi-hop reasoning. To address this limitation, we additionally evaluated our method on the MQuAKE dataset [4]. As shown in Table 4, AlphaEdit achieves strong Single-hop accuracy (72.17%) but drops substantially to 3.15% on All-of-3. In contrast, IKE achieves higher All-of-3 accuracy (20.85%) despite its lower Single-hop accuracy (47.75%). This contrast highlights that strong single-hop editing performance does not necessarily translate into effective multi-hop reasoning. Notably, ResidualKV consistently achieves the best results across all reasoning depths on both backbones, demonstrating its effectiveness in supporting multi-hop reasoning.

Table 4: Editing efficacy across different reasoning depths on 2,000 MQuAKE instances (%).

BackboneMethodSingle-hop ↑2-hop ↑3-hop ↑All-of-3 ↑
Llama-3.1-8BAlphaEdit72.1712.605.203.15
NeuralDB71.7811.724.622.40
IKE47.7546.7034.4520.85
ResidualKV74.7049.3436.8825.85
Phi-4-mini-3.8BAlphaEdit70.5913.305.093.80
NeuralDB62.677.053.471.50
IKE67.3636.9230.2920.90
ResidualKV73.0148.5534.4525.55

We have incorporated these additional results and the corresponding analyses into Sections 5.2 and 5.3 of the revised manuscript.

References

[1] Wang et al. WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models. NeurIPS, 2024.

[2] Fei et al. NeuralDB: Scaling Knowledge Editing in LLMs to 100,000 Facts with Neural KV Database. ICLR, 2026.

[3] Zheng et al. Can We Edit Factual Knowledge by In-Context Learning? EMNLP, 2023.

[4] Zhong et al. MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions. EMNLP, 2023.

AuthorsOpenReview ↗

Response to Reviewer WEUV (2/2)

Q3. Subject Alias Generalization

Thank you for this valuable suggestion. Conventional paraphrase evaluations often retain the original subject, making it difficult to assess whether editing methods generalize beyond specific subject expressions. To address this concern, we additionally evaluated subject alias generalization on ZsRE and CounterFact using the subject alias evaluation data provided by EasyEdit [5]. All methods were evaluated under the single-edit setting to eliminate the influence of accumulated edits.

As shown in Table 5, AlphaEdit achieves reasonable editing efficacy but suffers substantial degradation when subjects are replaced with aliases, while IKE demonstrates stronger alias generalization. In contrast, ResidualKV consistently achieves the highest Efficacy and Alias Accuracy across both backbones and datasets. Notably, on Llama-3.1-8B with CounterFact, ResidualKV achieves 92.37% Alias Accuracy, substantially outperforming AlphaEdit (30.99%) and IKE (32.39%). These results demonstrate that ResidualKV generalizes beyond exact subject expressions, complementing our original rephrase evaluations and providing stronger evidence of robust factual editing even without sequential updates.

Table 5: Single-edit efficacy and subject alias generalization on ZsRE and CounterFact (%).

BackboneMethodZsRE Eff. ↑ZsRE Alias Acc. ↑CounterFact Eff. ↑CounterFact Alias Acc. ↑
Llama-3.1-8BAlphaEdit62.1224.9183.1030.99
IKE58.0245.3937.5632.39
ResidualKV73.4859.7390.9692.37
Phi-4-mini-3.8BAlphaEdit66.5529.3578.8746.95
IKE78.5053.9288.2671.83
ResidualKV80.7762.3289.9376.53

Overall, ResidualKV introduces a new perspective on dynamic knowledge editing, using attention-KV space instead of conventional FFN parameters for factual correction. Furthermore, our proposed FRD learns the old-to-new factual change, enabling more precise editing with less interference to pretrained knowledge. Our extensive experiments further demonstrate the effectiveness and generalization of these two contributions.

We sincerely thank you again for your time and thoughtful feedback. We hope that our clarifications and additional experimental results address your concerns and that you will reconsider your assessment of our work.

Reference

[5] Wang et al. EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models. ACL, 2024.

AuthorsOpenReview ↗

A Brief Follow-up: Thank You for Your Constructive Feedback

Dear Reviewer WEUV,

Thank you again for your constructive feedback, which has helped us strengthen our experimental evaluation and make our study more comprehensive.

As the discussion period is approaching its end, we would like to kindly follow up on our previous responses. We sincerely hope that our additional experiments on retrieval-based editing, multi-hop reasoning, and subject alias generalization have addressed your concerns. The relevant experimental results and analyses have been incorporated into the revised manuscript and highlighted in blue for your convenience. If any concerns remain, we would be happy to provide further clarifications.

Thank you again for your time and consideration!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary and Manuscript Updates (Nov 19)

Dear Area Chair and Reviewers,

We sincerely appreciate the reviewers' constructive feedback, which has helped us strengthen our experimental evaluation and clarify our methodological contributions. We briefly summarize the key outcomes of our rebuttal below.

  • Reviewer WEUV: To address concerns regarding baseline fairness, multi-hop reasoning, and paraphrase robustness, we conducted additional comparisons with IKE, evaluations on MQuAKE, and subject alias experiments under the single-edit setting. ResidualKV consistently outperforms the evaluated baselines, further demonstrating its effectiveness and generalization.

  • Reviewer pa3E: We provided detailed clarifications regarding our editing paradigm, its distinctions from existing editing methods, and the role of FRD. Additional comparisons with IKE and RECIPE, controlled FFN+FRD experiments, and routing robustness evaluations further substantiate our methodological contributions.

  • Reviewer xGYP: We clarified the evaluation protocol and baseline performance differences, supplemented experiments under AlphaEdit's original protocol, and further explained the reasons behind ResidualKV's performance advantage. We sincerely appreciate the reviewer's recognition of our central finding regarding attention KV-based editing and willingness to increase their score from 2 to 3.

Manuscript Updates and Impact. The relevant experimental results and clarifications have been incorporated into the revised manuscript and highlighted in blue for ease of reference. These additional results further substantiate our two central contributions: a new dynamic knowledge editing paradigm based on editable attention KVs and Factual Residual Distillation (FRD) for targeted factual correction. Our proposed methodology and central conclusions remain unchanged.

We believe that the additional experimental evidence and methodological clarifications provide stronger support for the novelty, effectiveness, and generalization of ResidualKV. We sincerely hope that these contributions and the outcomes of our rebuttal will be considered in the final assessment.

Thank you again for your time, consideration, and valuable feedback.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.