- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates an improved method for knowledge editing in LLMs. Existing approaches (e.g., MEMIT, AlphaEdit) typically rely on a pre-computed and stored hidden representation matrix of pre-training knowledge to constrain the editing process and preserve retain knowledge. The primary objective of this paper is to completely eliminate and construct the optimization problem solely based on the representation of new knowledge, . The authors subsequently propose a series of techniques to address the challenges stemming from the removal of . Experimental results demonstrate that the proposed method achieves promising performance while eliminating the pre-computation and storage overhead associated with .
Critical strengths
- The proposed method effectively eliminates the time overhead of pre-computation and reduces the amortized cost per edit at the experimental scale.
- The method demonstrates solid empirical performance, achieving competitive results compared to several baselines designed for sequential editing on large-scale tasks.
Critical weaknesses
- My primary concern regarding this paper lies in its core objective: whether removing the pre-computation of holds practical significance in real-world scenarios. In practice, computing does not appear to be prohibitively expensive, as it only needs to be computed once per LLM model, after which subsequent editing operations can be executed infinitely using the same . Compared to ongoing edits, a few hours of one-time pre-computation cost is relatively low, and its GPU memory overhead during computation is also lower than that of various editing methods.
- The proposed method still exhibits limitations in certain scenarios. For instance, in Table 3 (multi-hop reasoning evaluation), when exceeds 2000, the model's reasoning accuracy shows a noticeable decline compared to traditional methods. This suggests that there remains a performance gap in retaining reasoning capabilities when compared to traditional methods relying on pre-computed .
Questions for authors
- Could the authors provide further clarification on the targeted application scenarios? Specifically, in what practical settings does eliminating the pre-computation of offer a clear, decisive advantage?
- Can the proposed framework be adapted as a general sequential editing method (e.g., by retaining while incorporating the newly introduced constraints to protect unrelated knowledge)? If so, how does its performance compare against other sequential editing baselines under this setup?
Thank you for your thoughtful feedback and for recognizing our method's competitive performance. Below, we respond to each of your concerns in turn.
[Weakness 1 & Question 1. Objective]
We agree that the one-time precomputation cost can be amortized across repeated editing sessions. Our goal is to enable factual association editing without relying on a reference corpus to represent the model's existing knowledge. Moir [1] provides further motivation for examining this dependency. It shows that covariance estimates derived from static reference corpora may not adequately reflect the model’s distribution after post-training. Its findings highlight the importance of selecting reference data that adequately represent the knowledge to be preserved. Consequently, covariance estimates based on K0 may be biased toward representation directions that are prominent in the reference corpus, potentially limiting the preservation of knowledge that is poorly represented in that corpus.
Our method eliminates the need to select such reference data or to construct a corresponding preservation covariance. This is particularly relevant when editing task-adapted models for which a suitable reference corpus representing their existing knowledge and capabilities is difficult to identify or unavailable. We have clarified this scenario in Section 5 in the revised manuscript.
[Weakness 2. Limitation]
We acknowledge the performance gap in multi-hop reasoning when the number of edits exceeds 2,000, as reported in Table 3. In the revised manuscript, the same results previously presented in Table 3 are shown in Figure 6 for easier comparison. We have explicitly discussed this limitation in Appendix A in the revised manuscript. The experiments below address the separate question of whether our framework can be extended to sequential editing. When we applied the extension proposed in the response to Question 2, accuracy decreased from 7.58 at N=1,000 to 4.25 at N=2,000. The drop in accuracy was smaller than before.
[Question 2. Sequential editing]
Yes, our framework can be adapted to sequential editing. Following your suggestion, we compared two variants of our method with existing sequential editing baselines. We conducted experiments on ZSRE using GPT-2 XL with a batch size of 100 for N=1,000.
We evaluated a variant denoted as Ours (w/ K0) that retains K0 for knowledge preservation while applying centering and TSVD to K1. We also developed a variant denoted as Ours** (w/o K0) that supports sequential editing without requiring K0. Specifically, we applied ridge regularization to penalize the update magnitude in all directions and incorporated statistics from previous edits to reduce interference with earlier edits.
We also compared Ours** with UltraEdit on ZSRE using Mistral-7B with a batch size of 10 for N=1,000.
Overall, these experiments support the feasibility of extending the proposed framework to sequential editing without K0.
Thank you again for your constructive feedback. We have included these results in Appendix B in the revised manuscript.
[1] Kwon, J., Kim, J., Kim, D.K. and Cha, M., 2026. Moir: Let the model direct its own story for robust cross-domain knowledge editing. In Advances in Neural Information Processing Systems.