-
The method is not model editing in the useful sense, and the obvious baseline is missing. Editing weights matters because the result is a drop-in replacement for the original model. ResidualKV does not produce one: the model stays frozen, and the system adds a bank of stored states plus a router that decides, for each query, whether an edit applies. Once such a router exists, the simplest solution is to have it retrieve the edited fact as text and place it in the prompt. The paper cites context-based editors such as IKE but does not compare with any of them. FRD itself is trained on exactly this signal: its targets come from the frozen model with the old or new fact placed in context, so the in-context model is the teacher that the stored states imitate. The paper should (a) compare against the same router followed by the fact, or the facts of the selected batch, in context, and against an LLM that is simply asked whether the query concerns an edited fact, reporting accuracy, context length and latency; and (b) explain what the learned key-value states offer beyond that baseline. Without this, the motivation for the method is unclear.
-
Matched controls that share the router are also missing. First, plain prefix tuning with the same router, layers and prefix length: the target-only KV row of Tables 2 and 4 may be this, but it is not described that way, and the FFN row is not defined. Second, FRD applied to a LoRA or FFN editor with the same router, to separate the loss from where the edit is stored. Third, the closest prior systems of this kind: RECIPE [1], Cartridges [2] and GRACE [3].
-
Benchmark annotations may supply subject spans that are unavailable in deployment, and the paper does not say whether gold spans are used at inference.
4.Causal subject-token states cannot see relation words that follow them. Two queries sharing a prefix through the subject but asking different subsequent relations can therefore have identical routing features. The selected state can still respond differently downstream, but the router itself cannot tell such queries apart. Same-subject/different-relation, alias, and pronoun tests are needed.
-
Table 1 reports MEMIT at 0.00/0.00 efficacy/generalization on Llama for both datasets, AlphaEdit at 0.15/0.20 on Qwen/CounterFact, and UltraEdit [6] at 0.00/0.00 on Phi/ZsRE. Degradation under sequential editing is expected, but scores of exactly zero need an explanation.
-
Table 1 reports the authors’ own reruns of the baselines, and they are far below the published numbers. On Llama-3 with 2,000 edits, AlphaEdit [4] reports 94.47/91.13 efficacy/generalization on ZsRE and 98.90/94.22 on CounterFact, and NeuralDB [5] reports 96.3/92.0 and 99.9/86.6. Table 1 gives AlphaEdit 80.58/70.91 and 66.53/42.85, and NeuralDB 29.50/28.48 and 3.65/2.22, on Llama-3.1. MEMIT is 0.00/0.00 in Table 1 but 34.62/31.28 (ZsRE) and 65.65/64.65 (CounterFact) in [4]. Against the published values, ResidualKV’s 91.39/91.02 and 91.12/90.85 would not be the best result. The metrics also differ: published ZsRE specificity is about 32 for these methods, whereas Table 1 reports 69–100.
-
The distinction between static FFN editing and dynamic attention editing needs qualification. A fixed FFN parameter update can have input-dependent effects.
-
Limited novelty. The paper explicitly acknowledges prefix tuning when introducing Equation 4. Concatenating persistent learned keys and values to native attention is therefore an adopted parameterization. The mixture decomposition in Equation 6 is an algebraic property of it: He et al. [8, Section 3.1, Equation 7] already express the mixture of native and prefix attention with input-dependent weights. This identity should be attributed; the more credible contribution is its use for factual editing with the particular residual-distillation objective and routed batch storage.
RECIPE [1] encodes factual statements into continuous input prompts and uses a learned retriever with a Knowledge Sentinel for selective application and fallback. Its encoder/retriever and sentinel differ from ResidualKV’s independently optimized layerwise KVs and fixed-threshold batch classifier, but the common retrieved-prompt structure makes it a close editing baseline.
Cartridges [2] freezes an LLM, directly optimizes persistent prefix-style KVs, and distills a context-conditioned teacher into the cache-conditioned model using a token-distribution KL loss. Its task is amortized long-context representation rather than sequential counterfactual editing, and it does not establish FRD’s old/new residual or this router. It nevertheless provides a close precedent for the writable state and FSD control.
SERAC [9] combines an edit cache, scope classifier, counterfactual predictor, and base-model fallback. GRACE [3] uses an activation-space codebook with distance-based deferral and learned replacement values. Larimar [10] uses a separately trained episodic-memory architecture with least-squares memory operations. These are precedents for selective, query-dependent memory editing, although their architectures and adaptation requirements differ from a shared attention softmax in an arbitrary frozen decoder.
DistillMIKE [11] distills retrieved in-context edits into LoRA, while Kujanpää et al. [12] transfer a knowledge-conditioned teacher into an unconditioned LoRA student. DeCK [13] contrasts contextual and parametric predictions during decoding; DISCO [7] uses differences between original and edited predictions for reasoning.
RECIPE, SERAC, GRACE, Larimar, He et al., DistillMIKE, and Cartridges are absent from the submission’s text and references.
References
[1] Qizhou Chen, Taolin Zhang, Xiaofeng He, Dongyang Li, Chengyu Wang, Longtao Huang, and Hui Xue, "Lifelong Knowledge Editing for LLMs with Retrieval-Augmented Continuous Prompt Learning," in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 13565–13580, 2024. link.
[2] Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Will Tennien, Atri Rudra, James Zou, Azalia Mirhoseini, and Christopher Re, "Cartridges: Lightweight and general-purpose long context representations via self-study," arXiv:2506.06266, 2025.
[3] Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi, "Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors," in Advances in Neural Information Processing Systems 36 (NeurIPS), pp. 47934–47959, 2023. link.
[4] Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Shi Jie, Xiang Wang, Xiangnan He, and Tat-seng Chua, "AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models," in International Conference on Learning Representations (ICLR), 2025. arXiv:2410.02355.
[5] Weizhi Fei, Hao Shi, Jing Xu, Jingchen Peng, Jiazheng Li, Jingzhao Zhang, Bo Bai, Wei Han, Zhenyuan Chen, and Xueyan Niu, "NeuralDB: Scaling Knowledge Editing in LLMs to 100,000 Facts with Neural KV Database," arXiv:2507.18028, 2025.
[6] Xiaojie Gu, Ziying Huang, Jia-Chen Gu, and Kai Zhang, "UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Models," Transactions on Machine Learning Research (TMLR), 2026. arXiv:2505.14679.
[7] Zengkui Sun, Yijin Liu, Jiaan Wang, Fandong Meng, Jinan Xu, Yufeng Chen, and Jie Zhou, "Outdated Issue Aware Decoding for Factual Knowledge Editing," in Findings of the Association for Computational Linguistics: ACL 2024, pp. 9282–9293, 2024. link.
[8] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig, "Towards a Unified View of Parameter-Efficient Transfer Learning," in International Conference on Learning Representations (ICLR), 2022. arXiv:2110.04366.
[9] Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn, "Memory-Based Model Editing at Scale," in International Conference on Machine Learning (ICML), PMLR 162, pp. 15817–15831, 2022. link.
[10] Payel Das, Subhajit Chaudhury, Elliot Nelson, Igor Melnyk, Sarath Swaminathan, Sihui Dai, Aurélie Lozano, Georgios Kollias, Vijil Chenthamarakshan, Jiří Navrátil, Soham Dan, and Pin-Yu Chen, "Larimar: Large Language Models with Episodic Memory Control," in International Conference on Machine Learning (ICML), 2024. arXiv:2403.11901.
[11] Shanbao Qiao, Xuebing Liu, and Seung-Hoon Na, "DistillMIKE: Editing Distillation of Massive In-Context Knowledge Editing in Large Language Models," in Findings of the Association for Computational Linguistics: ACL 2024, pp. 7639–7654, 2024. link.
[12] Kalle Kujanpää, Pekka Marttinen, Harri Valpola, and Alexander Ilin, "Efficient Knowledge Injection in LLMs via Self-Distillation," arXiv:2412.14964, 2024.
[13] Baolong Bi, Shenghua Liu, Lingrui Mei, Yiwei Wang, Junfeng Fang, Pengliang Ji, and Xueqi Cheng, "Decoding by Contrasting Knowledge: Enhancing Large Language Model Confidence on Edited Facts," in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pp. 17198–17208, 2025. link.
Response to Reviewer xGYP (1/2)
Dear Reviewer xGYP,
We sincerely thank you for your constructive comments and recognition of the novelty of our method and extensive experimental evaluations. Following your suggestions, we have conducted additional experiments and provide the following clarifications.
Thank you for raising this concern. Our CounterFact evaluation follows EasyEdit [2], a widely adopted knowledge editing framework, rather than a customized protocol. Unlike AlphaEdit [1], which compares the probabilities of new and old answers, EasyEdit evaluates target-answer prediction accuracy. For single-token answers, the distinction is:
\;\nRightarrow\; o_{\mathrm{new}}=\arg\max_{v\in\mathcal V}P_\theta(v\mid x).$$ Under AlphaEdit's protocol, a higher probability for the new answer than the old one does not necessarily indicate successful editing, as other answers may still be more probable. **This may overestimate actual editing success**, motivating our use of EasyEdit to directly assess whether the model correctly predicts the target answer. Following your suggestion, we additionally evaluated both AlphaEdit and ResidualKV under AlphaEdit's original protocol. As shown in Table 1, both methods obtain substantially higher scores, suggesting that relative-probability evaluation may overestimate actual editing success. Nevertheless, ResidualKV still consistently outperforms AlphaEdit across all three metrics on all three backbones. We have clarified these evaluation differences in Appendix A.2 of the revised manuscript. **Table 1: Comparison under AlphaEdit's original CounterFact evaluation protocol (%).** | Backbone | Method | Efficacy ↑ | Generalization ↑ | Specificity ↑ | |---|---|---:|---:|---:| | Llama-3.1-8B | AlphaEdit | 88.80 | 80.45 | 56.78 | | | **ResidualKV** | **99.70** | **89.11** | **89.25** | | Qwen3-8B | AlphaEdit | 80.75 | 51.03 | 49.19 | | | **ResidualKV** | **98.65** | **60.80** | **83.16** | | Phi-4-mini-3.8B | AlphaEdit | 87.65 | 60.58 | 64.14 | | | **ResidualKV** | **99.35** | **69.48** | **80.41** | **References** [1] Fang et al. *AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models.* ICLR, 2025. [2] Wang et al. *EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models.* ACL, 2024.