acceptodds
Under review as a conference paper at ICLR 2027

KEY: Unlocking Disentangled Model Updating with a Knowledge Addressing Dictionary

Abstract

Recent years have witnessed growing interest in knowledge updating (e.g., editing and unlearning) in language models. Despite substantial progress, existing methods remain challenged by knowledge entanglement, where different knowledge items share overlapping parameters or representational directions, often causing unintended interference. To address this issue, this paper introduces KEY (Knowledge addrEssing dictionarY), aiming for unlocking distinguishable knowledge structures within the entangled weight space of language models. Specifically, the proposed method first constructs an orthogonal atom dictionary from knowledge-induced gradients by decomposing their input- and output-side directions. It then performs sparse addressing to identify a compact set of atoms and their coefficients for each update. Theoretically, both global and local proximity bounds are established, characterizing conditions under which the learned dictionary preserves the desired knowledge structure and supports stable knowledge addressing. Extensive experiments across both knowledge editing and unlearning tasks demonstrate that KEY consistently achieves competitive performance against recent state-of-the-art baselines, while enabling more interpretable and disentangled knowledge updating.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Model Updating, Knowledge Editing, Machine Unlearning, Dictionary and Sparse Representation, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_P1qkOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes KEY, a unified framework for knowledge editing and unlearning in large language models (LLMs), aiming to mitigate knowledge entanglement during model updates. Through preliminary analyses of parameter overlap and responses to perturbations, the authors demonstrate that overlapping parameter representations do not necessarily imply that different knowledge is inseparable in weight space.

Motivated by these findings, KEY introduces a two-stage framework consisting of Knowledge Addressing Dictionary Construction and Task-Specific Sparse Addressing. The former constructs an orthogonal dictionary from input- and output-side gradients using SVD, while the latter identifies sparse combinations of dictionary atoms to perform targeted knowledge updates. The paper further establishes local and global bounded-proximity guarantees under certain regularity assumptions, providing theoretical support for the stability of knowledge addressing.

Extensive experiments on knowledge editing (CounterFact and FINE-KED) and unlearning (TOFU and MUSE-News/Books), across multiple backbone LLMs, demonstrate that KEY achieves competitive or superior target modification performance while substantially improving neighboring knowledge preservation and general model utility compared with existing baselines.

Critical strengths

1. Novel and well-motivated perspective on knowledge entanglement.

The paper provides an interesting perspective on knowledge entanglement by investigating whether overlapping parameter representations necessarily prevent knowledge separation. The preliminary experiments on parameter overlap and divergent responses to perturbations offer intuitive empirical motivation for the proposed approach. Building an orthogonal dictionary from input- and output-side gradients provides an interpretable mechanism for addressing different knowledge through shared model parameters. This is a meaningful strength because it offers a principled alternative to directly modifying overlapping parameters and provides a coherent motivation for the framework's design.

2. Unified formulation supported by extensive empirical and theoretical analysis.

KEY accommodates both knowledge editing and unlearning within the same dictionary construction and sparse addressing framework, with the two tasks primarily distinguished by their update objectives. This unification is appealing because these tasks are often investigated separately despite sharing the challenge of modifying targeted knowledge while preserving unrelated knowledge. Moreover, the paper evaluates KEY across multiple benchmarks and backbone models, complemented by ablation studies and theoretical analysis of addressing stability. The consistent improvements in knowledge preservation and general utility provide substantial empirical support for the proposed method.

Critical weaknesses

1. Strong dependence on representative global knowledge and retained sets limits practical applicability.

KEY relies heavily on a global knowledge pool P\mathcal{P} for dictionary construction and an explicitly defined retained set R\mathcal{R} for atom filtering and coefficient optimization. The ablation results in Table 3 reveal substantial performance degradation when the dictionary is constructed using only the target knowledge T\mathcal{T}: NS on CounterFact decreases from 83.3% to 56.5%, while FQ on TOFU drops from 0.69 to 0.42. These results suggest that KEY's effectiveness depends strongly on access to sufficiently representative auxiliary knowledge.

This assumption may be difficult to satisfy in realistic open-world settings, where knowledge updates arrive dynamically and the distribution of future editing requests is unknown. Furthermore, collecting and maintaining representative knowledge and retained sets may introduce substantial additional costs. This concern is consequential because it raises questions about whether KEY's reported advantages generalize beyond the relatively controlled experimental settings considered in the paper.

2. Unclear end-to-end scalability and lack of validation for sequential knowledge updates.

KEY introduces potentially substantial upfront computational and storage overhead. According to Table 7, the framework constructs dictionaries for editable weight matrices across all layers by default. This requires collecting token-level activations and gradients, performing SVD, and storing up to Natom=1024N_{\mathrm{atom}}=1024 dictionary atoms for each weight matrix. Although Appendix E.5 demonstrates efficient coefficient optimization after dictionary construction, it remains unclear whether the overall framework is computationally and memory-efficient when preprocessing costs are included, particularly for larger LLMs.

Additionally, the experiments focus on individual updating tasks without evaluating sequential or lifelong knowledge updates. In practical deployments, editing and unlearning requests typically arrive continuously, making it important to understand whether a fixed dictionary remains effective over time, whether repeated updates interfere with previously modified knowledge, and whether dictionary reconstruction is necessary. These limitations collectively leave the scalability and long-term practicality of KEY insufficiently established.

Questions for authors

1. Robustness to the selection and distribution of auxiliary knowledge.

Table 3 suggests that constructing the dictionary solely from target knowledge leads to substantial performance degradation. How sensitive is KEY to the size, diversity, and distributional similarity of the global knowledge pool P\mathcal{P} and retained set R\mathcal{R}? In particular, can the authors provide experiments using smaller or distribution-mismatched auxiliary knowledge sets to clarify the minimum requirements for maintaining effective knowledge addressing?

2. End-to-end computational and memory overhead.

Could the authors report the complete preprocessing cost of dictionary construction, including gradient collection, SVD computation, peak GPU memory consumption, and dictionary storage, alongside the per-update optimization cost? How do these costs scale with model size and the number of dictionary atoms? Such measurements would help determine whether KEY's improvements justify its additional overhead compared with existing editing and unlearning methods.

3. Applicability to sequential knowledge updates.

Can the authors clarify whether a dictionary constructed once can be reused for a sequence of editing or unlearning requests without significant performance degradation? Specifically, how does KEY handle interference between successive updates, and under what circumstances would dictionary reconstruction become necessary? Even a preliminary sequential updating experiment would help assess the framework's practical applicability.

8 comments
AuthorsOpenReview ↗

[Part 1] Respond to reviewer P1qk

Dear Reviewer P1qk,

We sincerely thank the reviewer for the constructive and insightful feedback. The comments have helped us strengthen both the technical analysis and experimental evaluation of KEY. We provide detailed responses and additional results below.

Weakness 1. Strong dependence on representative global knowledge and retained sets limits practical applicability. KEY relies heavily on a global knowledge pool for dictionary construction and an explicitly defined retained set for atom filtering and coefficient optimization. The ablation results in Table 3 reveal substantial performance degradation when the dictionary is constructed using only the target knowledge: NS on CounterFact decreases from 83.3% to 56.5%, while FQ on TOFU drops from 0.69 to 0.42. These results suggest that KEY's effectiveness depends strongly on access to sufficiently representative auxiliary knowledge. This assumption may be difficult to satisfy in realistic open-world settings, where knowledge updates arrive dynamically and the distribution of future editing requests is unknown. Furthermore, collecting and maintaining representative knowledge and retained sets may introduce substantial additional costs. This concern is consequential because it raises questions about whether KEY's reported advantages generalize beyond the relatively controlled experimental settings considered in the paper.

We thank the reviewer for this insightful comment. To further investigate KEY's applicability to dynamic update requests and unknown distribution of future updates, the following experiment with dynamic dictionary updating is introduced.

Specifically, an initial general knowledge pool P′P' is constructed by randomly sampling question--answer pairs from TOFU [1], CounterFact [2], MMLU [3], TriviaQA [4], and SciQ [5], with each dataset contributing 20% of the pool. The total size matches that used in the main experiments, i.e., ∣P′∣=∣P∣|P'|=|P|. All sampled items are standardized into a common question--answer format, and the initial dictionary DD is constructed following Section 4.1. Subsequently, target knowledge items T\mathcal{T} are sampled independently, ensuring T∩P′=∅\mathcal{T}\cap P'=\varnothing, without assuming prior knowledge of future update requests.

For each incoming knowledge item i∈Ti\in\mathcal{T}, dictionary coverage is evaluated to determine whether the existing bases sufficiently capture its gradient directions. Specifically, given the knowledge-induced gradient Gi=∇Wsi(W)G_i=\nabla_W s_i(W) from Eq. (5), the dictionary coverage is measured as

\frac{\|(U^\star)^\top G_i V^\star\|_F} {\|G_i\|_F},$$ where $U^\star$ and $V^\star$ denote the current output- and input-side bases, respectively. If $\mathrm{cov}_i\geq 0.5$, the existing dictionary is reused. Otherwise, the corresponding bases are incrementally updated to accommodate newly observed knowledge directions. Specifically, the new token-level factors are extracted following `Eqs. (6)--(7)` to form $A'$ and $B'$. Following the incremental SVD framework in [6], the existing bases and singular values can be reused to construct the reduced matrices $\begin{bmatrix}\Sigma_A(V^\star)^\top\\ A'\end{bmatrix}$ and $\begin{bmatrix}\Sigma_B(U^\star)^\top\\ B'\end{bmatrix}$. Their leading right singular vectors are then retained as the updated bases $V^\star$ and $U^\star$, respectively. Notably, for $n$ incoming knowledge items and $k$ new factor rows per weight matrix, gradient extraction requires $\mathcal{O}(nC_{\mathrm{FB}})$. The compressed matrices have dimensions $(r_{\mathrm{in}}+k)\times d_{\mathrm{in}}$ and $(r_{\mathrm{out}}+k)\times d_{\mathrm{out}}$. Assuming $r_{\mathrm{in}}+k\ll d_{\mathrm{in}}$ and $r_{\mathrm{out}}+k\ll d_{\mathrm{out}}$, the total complexity is $\mathcal{O}\left(nC_{\mathrm{FB}}+\sum_W\left[d_{\mathrm{in}}(r_{\mathrm{in}}+k)^2+d_{\mathrm{out}}(r_{\mathrm{out}}+k)^2\right]\right)$. When $n\ll N$ and $k\ll K$, incremental updating significantly reduces reconstruction costs by operating on compressed matrices rather than the full accumulated knowledge pool. Then, updated bases are used to produce dictionary atoms following `Eq. (9)`, followed by task-specific atom filtering and coefficient optimization. **Table A. Unlearning performance on TOFU with fixed and dynamic dictionaries.** | Method | FQ ↑ | MU ↑ | | ------ | ---- | ---- | | KEY (Original $P$) | 0.69 | 0.66 | | KEY ($P'$ with fixed dictionary) | 0.46 | 0.53 | | KEY ($P'$ with dynamic dictionary) | 0.65 | 0.62 | **Table B. Editing performance on CounterFact with fixed and dynamic dictionaries.** | Method | ES ↑ | PS ↑ | NS ↑ | | ------ | ---- | ---- | ---- | | KEY (Original $P$) | 99.9 | 98.1 | 83.3 | | KEY ($P'$ with fixed dictionary) | 89.5 | 86.7 | 56.9 | | KEY ($P'$ with dynamic dictionary) | 99.8 | 97.2 | 81.5 |
AuthorsOpenReview ↗

[Part 2] Respond to reviewer P1qk

Continue to weakness 1. As shown in Tables A--B, replacing the original knowledge pool PP with P′P' while keeping the dictionary fixed degrades both unlearning and editing performance. However, dynamic dictionary updating substantially improves performance on both tasks. On TOFU, for instance, FQ increases from 0.46 to 0.65, while MU improves from 0.53 to 0.62. Notably, the resulting FQ ranks third and MU ranks second among the unlearning methods compared in Table 1.

These results suggest that dynamic dictionary updating offers a practical extension of KEY when future update requests are unknown or the initial knowledge pool provides insufficient coverage. Beyond achieving competitive performance, this strategy maintains relatively low computational overhead, as the SVD-based dictionary construction in KEY naturally supports incremental updates by reusing existing singular directions. This provides a promising direction for addressing the challenges of dynamic knowledge updating without relying on a fully representative global knowledge pool.

Detials and experimental settings for dynamic dictionary updating are provided in Appendix E.10.

[1] Maini et al. TOFU: A Task of Fictitious Unlearning for LLMs. COLM 2024.

[2] Meng et al. Locating and Editing Factual Associations in GPT. NeurIPS 2022.

[3] Hendrycks et al. Measuring Massive Multitask Language Understanding. ICLR 2021.

[4] Joshi et al. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. ACL 2017.

[5] Welbl et al. Crowdsourcing Multiple Choice Science Questions. WNUT 2017.

[6] Kalantzis et al. Projection Techniques to Update the Truncated SVD of Evolving Matrices with Applications. ICML 2021.

Weakness 2. Unclear end-to-end scalability and lack of validation for sequential knowledge updates. KEY introduces potentially substantial upfront computational and storage overhead. According to Table 7, the framework constructs dictionaries for editable weight matrices across all layers by default. This requires collecting token-level activations and gradients, performing SVD, and storing up to dictionary atoms for each weight matrix. Although Appendix E.5 demonstrates efficient coefficient optimization after dictionary construction, it remains unclear whether the overall framework is computationally and memory-efficient when preprocessing costs are included, particularly for larger LLMs. Additionally, the experiments focus on individual updating tasks without evaluating sequential or lifelong knowledge updates. In practical deployments, editing and unlearning requests typically arrive continuously, making it important to understand whether a fixed dictionary remains effective over time, whether repeated updates interfere with previously modified knowledge, and whether dictionary reconstruction is necessary. These limitations collectively leave the scalability and long-term practicality of KEY insufficiently established.

Thank you for this constructive comment. The computational and storage cost of KEY is addressed first, followed by new experiments on sequential knowledge updates.

1. Cost of dictionary construction Dictionary construction involves gradient extraction, and SVD. Let N=∣C∣N=|\mathcal C| denote the number of knowledge items used for construction, KK denote the total number of token positions collected across knowledge set C\mathcal C (Line 228), and CFBC_{\mathrm{FB}} the cost of one forward–backward pass per item. For each weight matrix W∈Rdout×dinW\in\mathbb R^{d_{\mathrm{out}}\times d_{\mathrm{in}}}, dind_{\mathrm{in}} and doutd_{\mathrm{out}} denote its input and output dimensions, respectively. Gradient extraction requires O(NCFB)\mathcal O(NC_{\mathrm{FB}}). Meanwhile, SVD composition of AA and BB costs O(K(din2+dout2))\mathcal O(K(d_{\mathrm{in}}^2+d_{\mathrm{out}}^2)). Thus, the total time complexity of these stages is O(NCFB+∑W(K(din2+dout2)))\mathcal{O}\left( NC_{\mathrm{FB}} +\sum_W(K(d_{\mathrm{in}}^2+d_{\mathrm{out}}^2)) \right).

Empirically, dictionary construction is evaluated using the TOFU-finetuned Llama-3.1-8B on a single NVIDIA A100 80 GB GPU, with 4,000 TOFU question–answer pairs. The dictionary covers 7 projection matrices per Transformer layer, giving 224 matrices across 32 layers in total. The input- and output-side ranks are set to rin=rout=48r_{\mathrm{in}}=r_{\mathrm{out}}=48. As shown in Table C, the entire construction takes 17.94 minutes, of which SVD-based decomposition accounts for 61.8%, while gradient extraction and direction aggregation jointly account for 34.0%. The resulting dictionary occupies approximately 0.5 GB on disk, and the measured peak GPU memory usage during construction is approximately 67 GB. This analysis is further detailed in Appendix E.5.

AuthorsOpenReview ↗

[Part 3] Respond to reviewer P1qk

Continue to weakness 2.

Table C. Runtime (in minutes) breakdown for initial dictionary construction, while other overhead includes dictionary serialization and normalization.

\hline \text{Gradient extraction} & \text{SVD} & \text{Other overhead} & \text{Total} \\\\ \hline 6.10 & 11.09 & 0.75 & \mathbf{17.94} \\\\ \hline \end{array}$$ To assess scalability to larger LLMs, `Table D` reports dictionary construction times of 3.87, 17.94, and 31.18 minutes for Llama-3.2-1B, Llama-3.1-8B, and Llama-2-13B, respectively. Although construction time increases with model size, the dictionary for the 13B model can be constructed in approximately 31 minutes. This linear growth in dictionary construction time remains manageable. In sequential updating setting, the preprocessing cost can be amortized across multiple update requests through dictionary reuse. **Table D. Comparison of construction time across model sizes (minutes).** $$\begin{array}{|l|c|c|c|} \hline \text{Model} & \text{Llama-3.2-1B} & \text{Llama-3.1-8B} & \text{Llama-2-13B} \\\\ \hline \text{Construction time (min)} & 3.87 & 17.94 & 31.18 \\\\ \hline \end{array}$$ **2. Sequential knowledge updates** To examine whether a fixed dictionary remains effective over time, sequential-batch experiments are added for both editing and unlearning. In every experiment, the dictionary is constructed once before the first round and kept fixed across all rounds, without reconstruction. *Sequential editing.* Following the experimental setup of AlphaEdit [1], Llama-3-8B is evaluated on CounterFact with 100 facts per round over 20 consecutive rounds, totaling 2,000 edits. As shown in `Table E`, KEY achieves comparable or superior performance to AlphaEdit across all three evaluation metrics at every reported checkpoint. After 2,000 edits, KEY achieves 99.0\% Efficacy Success (ES), 95.3\% Paraphrase Success (PS), and 72.2\% Neighborhood Success (NS), outperforming AlphaEdit by 0.1, 1.0, and 4.4 percentage points, respectively. These results indicate that KEY maintains comparable editing effectiveness while improving paraphrase generalization and preservation of neighboring knowledge throughout sequential updates. **Table E. Performance comparsion (reported as ES / PS / NS (%); higher is better) for sequential-batch knowledge editing on CounterFact with Llama-3-8B.** $$\begin{array}{|l|c|c|c|c|} \hline \text{Method} & \text{Round 5} & \text{Round 10} & \text{Round 15} & \text{Round 20} \\\\ \hline \text{AlphaEdit} & 99.8 / 93.5 / 76.8 & 99.6 / 93.6 / 74.1 & 99.2 / 93.9 / 70.4 & 98.9 / 94.3 / 67.8 \\\\ \hline \mathbf{KEY} & 99.8 / 95.9 / 82.1 & 99.7 / 94.3 / 80.5 & 99.3 / 95.1 / 77.9 & 99.0 / 95.3 / 72.2 \\\\ \hline \end{array}$$ *Sequential unlearning.* Following the experimental setting of ASU [2], Llama-2-Chat-7B is evaluated on TOFU, with each round forgetting either 1\% or 5\% of the dataset's authors over 10 consecutive rounds. For a fair comparison, Forgetting Efficacy (FE) and Model Utility (MU) are adopted from [2] to measure forgetting performance on the current round forget set and on the remaining retain set, respectively. As reported in `Table F`, KEY consistently achieves higher MU than ASU at every reported checkpoint under both settings. With 1\% of authors removed per round, KEY also achieves higher FE at rounds 1, 5, and 10. Under the more demanding 5\% setting, although both methods exhibit fluctuations in FE across rounds, KEY attains a higher FE than ASU. Overall, these results indicate that KEY better preserves model utility while maintaining competitive forgetting effectiveness across sequential unlearning. **Table F. Performance comparsion (reported as FE / MU (%); higher is better) for sequential-batch knowledge unlearning on TOFU with Llama-2-Chat-7B .** $$\begin{array}{|c|l|c|c|c|c|} \hline \text{Authors removed} & \text{Method} & \text{Round 1} & \text{Round 3} & \text{Round 5} & \text{Round 10} \\\\ \text{per round} & & & & & \\\\ \hline 1\\% & \text{ASU} & 81.2 / 76.8 & 78.7 / 75.3 & 78.2 / 74.9 & 74.5 / 74.1 \\\\ \hline 1\\% & \mathbf{KEY} & 82.5 / 78.7 & 76.3 / 77.1 & 81.7 / 76.2 & 76.4 / 74.7 \\\\ \hline 5\\% & \text{ASU} & 76.6 / 73.6 & 75.1 / 72.0 & 75.2 / 71.4 & 77.2 / 67.0 \\\\ \hline 5\\% & \mathbf{KEY} & 78.1 / 75.5 & 79.2 / 74.8 & 75.5 / 74.1 & 77.9 / 71.7 \\\\ \hline \end{array}$$ Both results clearly demonstrate the effectiveness of KEY in sequential-batch knowledge updating, highlighting its ability to preserve selective addressing while reusing a fixed dictionary across successive rounds. We thank the reviewer again for this broader evaluation. Detailed experimental settings, complete results, and further analysis are included in `Section 5.3`.
Reviewer_1ZtjOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Knowledge editing and unlearning in LLMs suffer from knowledge entanglement: updates to one fact can interfere with others because parameters/directions overlap. This paper proposes KEY (Knowledge addrEssing dictionarY). It builds an orthogonal atom dictionary from knowledge-induced gradients by decomposing input- and output-side directions, then sparsely addresses each update with a compact set of atoms and coefficients. The paper provides global/local proximity bounds for stable addressing and evaluates on editing (CounterFact, FINE-KED) and unlearning (TOFU, MUSE) across several LLM backbones. KEY is competitive with recent baselines and shows qualitative evidence of inter-knowledge separability and intra-knowledge consistency in coefficient space. Anonymous code is linked.

Critical strengths
  1. Unified framing. Casting editing and unlearning as sparse addressing over a shared dictionary (Fig. 3) is a clean organizing idea and supports interpretability beyond raw accuracy.

  2. Solid empirics with theory. Competitive results on multiple backbones/benchmarks (Tables 1–2), often strong on locality/retain metrics, plus conditional proximity bounds for address stability (Appendix D).

  3. Direct coefficient analysis. Sec. 5.3 / Fig. 5 (overlap ≈0.05 across knowledge vs ≈0.98 for paraphrases of the same fact) gives concrete evidence that addresses are separable in coefficient space. Dual-use risks are also acknowledged in the Ethics Statement.

Critical weaknesses
  1. Baseline comparison fairness (Sec. 5; Tables 1–2; Appendix E.1). The baseline results are taken directly from the original papers, making it difficult to verify whether all methods are evaluated under comparable conditions. In particular, differences in evaluation protocols, retained-knowledge access, and update settings could affect the reported results. A clearer protocol comparison, including whether each baseline uses retained knowledge during optimization, or matched experiments with strong baselines such as DiKE and PALU would strengthen the empirical claims.

  2. Dictionary protocol, construction cost, and reuse (Sec. 5; Table 3; Appendix E.5). Building the dictionary from the full knowledge pool P before updates is distant from one-off/streaming edits: target-related directions are largely already represented in the dictionary. Although Appendix E.5 reports update efficiency after dictionary construction, the actual time and memory required for initial gradient extraction and SVD are not sufficiently quantified. Moreover, it remains unclear whether a fixed dictionary maintains its effectiveness across repeated sequential knowledge updates or requires periodic reconstruction.

  3. Claim–evidence gap (Sec. 5.3; Appendix D, E.4–E.7). The paper claims disentangled knowledge updating, but the evidence primarily demonstrates separable coefficient representations and selective updates over a shared dictionary. While the additional intervention analyses provide useful support, they do not fully establish semantic or causal separation of knowledge. Appendix D provides conditional bounds for address stability rather than guarantees of knowledge disentanglement. Clarifying the scope of the claims and evaluating more closely related knowledge items would improve persuasiveness.

  4. Limited novelty / positioning (Sec. 2; Appendix A.2). Orthogonal dictionaries and sparse codes over gradient-derived directions overlap with prior work such as DiKE (representation separation), parameter isolation, and subspace / neuron localization. A sharper one-sentence statement of KEY’s distinctive contribution versus these lines would help clarify its methodological novelty.

  5. Trade-offs under-sold as SOTA (Sec. 5.1; Tables 1–2). KEY is not uniformly best, particularly in knowledge unlearning, where some baselines achieve substantially higher Forget Quality on TOFU. Emphasizing the locality–forgetting/utility trade-off would be more convincing than primarily counting the number of best-performing metrics.

Questions for authors

Q1. Since the baseline results are taken from the original papers, how comparable are the evaluation protocols and information available to each method, particularly retained-knowledge access? Could the authors provide a protocol comparison table or matched evaluations against strong baselines such as DiKE and PALU?

Q2. How does performance change if the dictionary is built from a small proxy knowledge set or updated online? Does a fixed dictionary remain effective across sequential knowledge updates, or must it be reconstructed as the model weights change?

Q3. What are the actual wall-clock time and peak memory costs of initial gradient extraction and SVD-based dictionary construction? For practical deployment, could the authors compare end-to-end latency with ROME/MEMIT, both with and without amortizing dictionary construction, and report how the costs scale with dictionary size and model size?

Q4. Could the authors clarify the distinction between sparse knowledge addressing and semantic knowledge disentanglement, and more explicitly position KEY against existing approaches such as DiKE and subspace-constrained editing? Which assumptions underlying the Appendix D bounds are most restrictive in practice?

Q5. When KEY underperforms on Forget Quality, is this mainly due to the addressing mechanism or the update objective? What operating point on the retain–forget Pareto frontier would the authors recommend?

8 comments
AuthorsOpenReview ↗

[Part 1] Respond to 1Ztj

Dear Reviewer 1Ztj,

We sincerely thank the reviewer for these insightful and constructive comments. We appreciate the opportunity to clarify experimental fairness, computational cost, and performance under specific updating scenarios. The following responses provide additional explanations and experimental evidence to address these concerns.

Weakness 1. Baseline comparison fairness (Sec. 5; Tables 1–2; Appendix E.1). The baseline results are taken directly from the original papers, making it difficult to verify whether all methods are evaluated under comparable conditions. In particular, differences in evaluation protocols, retained-knowledge access, and update settings could affect the reported results. A clearer protocol comparison, including whether each baseline uses retained knowledge during optimization, or matched experiments with strong baselines such as DiKE and PALU would strengthen the empirical claims.

We sincerely thank the reviewer for this comment. Specifically, the baseline results in Table 1 are adopted from DiKE [1], which provides comprehensive comparisons among knowledge editing methods. KEY follows the same model backbones, benchmark datasets, independent single-edit setting, and evaluation metrics. Similarly, the baseline results in Table 2 primarily follow PALU [2], with KEY adopting the corresponding model backbones, forget/retain splits, and evaluation protocols.

Regarding retained-knowledge preservation, both DiKE and PALU incorporate them during model optimization. For instance, DiKE constrains changes to retained knowledge representations, while PALU employs the KL regularization on non-sensitive tokens from retained knowledge. In addition, a protocol comparison table is added to Appendix E.1 to explicitly clarify training-time information access to improve the transparency of baseline comparisons.

[1] Zhang et al. Disentangling Knowledge Representations for Large Language Model Editing. ICLR 2026.

[2] Zhai et al. Maximizing Local Entropy Where It Matters: Prefix-Aware Localized LLM Unlearning. ACL 2026.

AuthorsOpenReview ↗

[Part 2] Respond to reviewer 1Ztj

Weakness 2. Dictionary protocol, construction cost, and reuse (Sec. 5; Table 3; Appendix E.5). Building the dictionary from the full knowledge pool P before updates is distant from one-off/streaming edits: target-related directions are largely already represented in the dictionary. Although Appendix E.5 reports update efficiency after dictionary construction, the actual time and memory required for initial gradient extraction and SVD are not sufficiently quantified. Moreover, it remains unclear whether a fixed dictionary maintains its effectiveness across repeated sequential knowledge updates or requires periodic reconstruction.

We sincerely thank the reviewer for raising these concerns regarding dictionary construction costs and reuse.

Firstly, dictionary construction involves gradient extraction and basis decomposition (SVD). Let N=∣C∣N=|\mathcal C| denote the number of knowledge items used for construction, KK denote the total number of token positions collected across knowledge set C\mathcal C (Line 228), and CFBC_{\mathrm{FB}} the cost of one forward–backward pass per item. For each weight matrix W∈Rdout×dinW\in\mathbb R^{d_{\mathrm{out}}\times d_{\mathrm{in}}}, dind_{\mathrm{in}} and doutd_{\mathrm{out}} denote its input and output dimensions, respectively. Gradient extraction requires O(NCFB)\mathcal O(NC_{\mathrm{FB}}). Meanwhile, SVD composition of AA and BB costs O(K(din2+dout2))\mathcal O(K(d_{\mathrm{in}}^2+d_{\mathrm{out}}^2)). Thus, the total time complexity is O(NCFB+∑W(K(din2+dout2)))\mathcal{O}\left( NC_{\mathrm{FB}} +\sum_W(K(d_{\mathrm{in}}^2+d_{\mathrm{out}}^2)) \right).

Empirically, dictionary construction is evaluated using the TOFU-finetuned Llama-3.1-8B on a single NVIDIA A100 80 GB GPU, with 4,000 TOFU question–answer pairs. The dictionary covers 7 projection matrices per Transformer layer, giving 224 matrices across 32 layers in total. The input- and output-side ranks are set to rin=rout=48r_{\mathrm{in}}=r_{\mathrm{out}}=48. As shown in Table A, the entire construction takes 17.94 minutes, of which SVD-based decomposition accounts for 61.8%, while gradient extraction and direction aggregation jointly account for 34.0%. The resulting dictionary occupies approximately 0.5 GB on disk, and the measured peak GPU memory usage during construction is approximately 67 GB. This analysis is further detailed in Appendix E.5.

Table A. Runtime (in minutes) breakdown for initial dictionary construction on TOFU, while other overhead includes dictionary serialization and normalization.

\hline \text{Gradient extraction} & \text{SVD} & \text{Other overhead} & \text{Total} \\\\ \hline 6.10 & 11.09 & 0.75 & \mathbf{17.94} \\\\ \hline \end{array}$$ Secondly, to examine whether a fixed dictionary remains effective across sequential knowledge updates, additional sequential-batch experiments are conducted. Following the experimental setup of AlphaEdit [1], Llama-3-8B is evaluated on CounterFact with 100 facts per round over 20 consecutive rounds, totaling 2,000 edits. As shown in `Table B`, KEY achieves comparable or superior performance to AlphaEdit across all three evaluation metrics at every reported checkpoint. After 2,000 edits, KEY achieves 99.0\% Efficacy Success (ES), 95.3\% Paraphrase Success (PS), and 72.2\% Neighborhood Success (NS), outperforming AlphaEdit by 0.1, 1.0, and 4.4 percentage points, respectively. These results indicate that KEY maintains robust sequential-editing effectiveness. **Table B. Performance comparsion (reported as ES / PS / NS (%); higher is better) for sequential-batch knowledge editing on CounterFact with Llama-3-8B.** $$\begin{array}{|l|c|c|c|c|} \hline \text{Method} & \text{Round 5} & \text{Round 10} & \text{Round 15} & \text{Round 20} \\\\ \hline \text{AlphaEdit} & 99.8 / 93.5 / 76.8 & 99.6 / 93.6 / 74.1 & 99.2 / 93.9 / 70.4 & 98.9 / 94.3 / 67.8 \\\\ \hline \mathbf{KEY} & 99.8 / 95.9 / 82.1 & 99.7 / 94.3 / 80.5 & 99.3 / 95.1 / 77.9 & 99.0 / 95.3 / 72.2 \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

[Part 3] Respond to reviewer 1Ztj

Continue to Weakness 2. On the other hand, following the experimental setting of ASU [2], Llama-2-Chat-7B is evaluated on TOFU, with each round forgetting either 1% or 5% of the dataset's authors over 10 consecutive rounds. For a fair comparison, Forgetting Efficacy (FE) and Model Utility (MU) are adopted from [2] to measure forgetting performance on the current round forget set and on the remaining retain set, respectively. As reported in Table C, KEY consistently achieves higher MU than ASU at every reported checkpoint under both settings. With 1% of authors removed per round, KEY also achieves higher FE at rounds 1, 5, and 10. Under the more demanding 5% setting, although both methods exhibit fluctuations in FE across rounds, KEY still attains a higher FE than ASU. Overall, these results indicate that KEY manages to preserve better model utility while maintaining competitive forgetting effectiveness across sequential unlearning.

Table C. Performance comparsion (reported as FE / MU (%); higher is better) for sequential-batch knowledge unlearning on TOFU with Llama-2-Chat-7B .

\hline \text{Authors removed} & \text{Method} & \text{Round 1} & \text{Round 3} & \text{Round 5} & \text{Round 10} \\\\ \text{per round} & & & & & \\\\ \hline 1\\% & \text{ASU} & 81.2 / 76.8 & 78.7 / 75.3 & 78.2 / 74.9 & 74.5 / 74.1 \\\\ \hline 1\\% & \mathbf{KEY} & 82.5 / 78.7 & 76.3 / 77.1 & 81.7 / 76.2 & 76.4 / 74.7 \\\\ \hline 5\\% & \text{ASU} & 76.6 / 73.6 & 75.1 / 72.0 & 75.2 / 71.4 & 77.2 / 67.0 \\\\ \hline 5\\% & \mathbf{KEY} & 78.1 / 75.5 & 79.2 / 74.8 & 75.5 / 74.1 & 77.9 / 71.7 \\\\ \hline \end{array}$$ These results demonstrate that KEY maintains effective knowledge updating across sequential updates without periodic reconstruction, supporting the feasibility of fixed-dictionary reuse. Detailed experimental settings, results, and analyses are provided in `Appendix E.9`. [1] Fang et al. *AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models.* ICLR 2025. [2] Zade et al. *Attention Smoothing Is All You Need For Unlearning.* ICLR 2026. > Weakness 3. Claim–evidence gap (Sec. 5.3; Appendix D, E.4–E.7). The paper claims disentangled knowledge updating, but the evidence primarily demonstrates separable coefficient representations and selective updates over a shared dictionary. While the additional intervention analyses provide useful support, they do not fully establish semantic or causal separation of knowledge. Appendix D provides conditional bounds for address stability rather than guarantees of knowledge disentanglement. Clarifying the scope of the claims and evaluating more closely related knowledge items would improve persuasiveness. We sincerely thank the reviewer for highlighting the evidence of knowledge distinguishability in `Section 5.3`. To address this concern, an additional case study is provided to examine whether KEY can selectively unlearn semantically related facts. Specifically, for the same 20 knowledge items examined in `Section 5.3`, entities and factual attributes, including author identities, literary genres, occupations, locations, and awards, are extracted from each item. Pairwise relatedness is measured by the degree of overlap between their extracted entities and attributes, with pairs exceeding a threshold of 0.5 considered semantically related. Each fact is independently unlearned using the same fixed dictionary to examine whether closely related knowledge can be distinguished through task-specific coefficients. The results show that the corresponding coefficient vectors $\alpha$ exhibit a mean cosine similarity below 0.10 across the selected pairs. These findings provide empirical evidence that **facts with substantial entity-level overlap can receive distinct knowledge addresses**. Together with the inter-knowledge separability and intra-knowledge consistency analyses in `Section 5.3`, these results further support the effectiveness of KEY in distinguishing and selectively updating closely related knowledge. Detailed experimental settings and case studies are provided in `Appendix E.9`.
Reviewer_XSqyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies knowledge updating in LLMs, where shared parameters can cause interference between target and non-target knowledge. The proposed method, KEY, constructs an orthogonal dictionary from knowledge-induced gradients and represents each update as a sparse combination of dictionary atoms. It extracts input-side and output-side directions, combines them into rank-one atoms, and filters these atoms according to their relevance to target and retained knowledge. The coefficients are then optimized for editing or unlearning. Experiments cover both tasks across several models and benchmarks. The results show a favorable balance between target updating and knowledge preservation. The paper also examines dictionary construction, address consistency across paraphrases, and the effects of intervening on coefficient magnitudes and signs. The theoretical analysis establishes conditional stability bounds for the optimized addresses under continuous input perturbations.

Critical strengths
  1. The proposed KEY is a well-motivated and technically sound approach to knowledge updating. Its shared dictionary and sparse task-specific coefficients provide a clear, interpretable way to perform both editing and unlearning.

  2. The empirical evaluation is convincing across the tested settings, with a balance between update effectiveness and knowledge preservation.

Critical weaknesses
  1. The shared parameter regions used in the motivation experiment may not reflect shared knowledge dependence in the jointly trained model. Section 3 identifies each knowledge item’s update locations through independent training, takes their intersection, and then perturbs those locations in a model trained jointly on all items. Joint training may change the locations. Second, the experiment is also limited to TOFU. Since the method addresses both editing and unlearning, validation on an editing dataset would help.

  2. The paper lacks evaluation of sequential and batch editing or unlearning. This question matters for assessing the practical benefits of dictionary reuse and selective addressing.

Questions for authors
  1. Do the differing perturbation responses also occur on editing datasets such as CounterFact or FINE-KED? A corresponding experiment would help assess whether the observation extends to the editing setting.

  2. Could you report update effectiveness as the number of sequential requests or the batch size increases?

6 comments
AuthorsOpenReview ↗

[Part 1] Response to reviewer XSqy

Dear Reviewer XSqy

Thank you for your thorough review and insightful comments on our manuscript. We have carefully consider your feedback and make the following clarifications to address the concerns raised.

Weakness 1. The shared parameter regions used in the motivation experiment may not reflect shared knowledge dependence in the jointly trained model. Section 3 identifies each knowledge item’s update locations through independent training, takes their intersection, and then perturbs those locations in a model trained jointly on all items. Joint training may change the locations. Second, the experiment is also limited to TOFU. Since the method addresses both editing and unlearning, validation on an editing dataset would help. We thank the reviewer for this constructive suggestion. To address both concerns, the motivation study in Section 3 is extended by identifying shared parameter regions directly within a jointly trained model and conducting additional validation on CounterFact.

The original analysis in Section 3 (Fig. 2(a)) examines whether independently learned knowledge items exhibit overlapping parameter locations. However, these locations may differ from those after joint training. As such, an additional experiment performs both parameter localization and perturbation analysis directly on one jointly trained model.

Specifically, a model WallW_{\mathrm{all}} is first fine-tuned on the same 20 knowledge items. Following [1], the parameter locations associated with each item ii are identified using gradient saliency, ∣∇Wsi(Wall)∣|\nabla_W s_i(W_{\mathrm{all}})|, where sis_i denotes the mean pre-softmax logit assigned to its reference answer. For each weight matrix, parameters with the largest absolute gradients are selected, with the number of selected parameters matched to that in the original independent-training analysis. This ensures a comparable localization budget without relying on additional locations. Pairwise intersections are then computed across all knowledge items, and the perturbation analysis is then repeated explicitly on the newly identified shared parameter regions.

As shown in Table A, substantial parameter overlap also persists within the jointly trained model, with Jaccard similarities ranging from 18% to 56% across all 190 knowledge pairs. Moreover, applying the same perturbation to each pair's shared parameter locations induces different target-logit changes in magnitude across the two knowledge items, similar to the observations in Fig. 2(b). These findings indicate that knowledge items remain distinguishable in their parameter sensitivities despite sharing substantial parameter regions, even when both localization and perturbation are performed within the jointly trained model.

Table A. Pairwise parameter overlapping with jointly trained model.

\hline \text{Dataset} & \text{Localization criterion} & \text{Perturbation model} & \text{Jaccard range (\\%)} \\\\ \hline \text{TOFU (unlearning)} & |\nabla_W s_i(W_{\mathrm{all}})| & W_{\mathrm{all}} & 18\text{--}56 \\\\ \hline \text{CounterFact (editing)} & |\nabla_W s_i(W_0)| & W_0 & 21\text{--}52 \\\\ \hline \end{array}$$ Similarly, the motivation analysis is further extended to CounterFact [2] to examine whether the observed parameter overlap and differentiated perturbation responses generalize to knowledge editing. Following prior work on factual recall and consistency across paraphrases [3,4], 20 facts are selected for which the base model $W_0$ correctly predicts the original answers under both the original and paraphrased prompts, yielding 190 knowledge pairs. Again, following [1], knowledge-specific parameter sets $S_i$ are identified by selecting the largest absolute gradients $|\nabla_W s_i(W_0)|$ within each weight matrix. Pairwise Jaccard similarities quantify their overlap, while identical perturbations are applied to the shared locations $S_{ij}=S_i\cap S_j$ to compare the resulting target-logit responses. As reported in `Table A`, CounterFact also exhibits substantial parameter overlap across distinct knowledge items, while identical perturbations to shared parameter locations induce different target-logit responses. These findings are consistent with the central observation in `Section 3` (`Lines 172--173)`: **substantial parameter overlap does not imply knowledge indistinguishability, as identical perturbations can induce distinct, and potentially opposite, responses across knowledge items.** As such, these results provide further empirical support for this observation across both knowledge editing and unlearning benchmarks. The corresponding parameter-overlap heatmaps and perturbation-response scatter plots are included in `Appendix B`. We sincerely thank the reviewer for this insightful and constructive comment, which motivates a more comprehensive validation of our motivation study.
AuthorsOpenReview ↗

[Part 2] Respond to reviewer XSqy

References for Weakness 1: [1] Fan et al. SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation. ICLR 2024.

[2] Meng et al. Locating and Editing Factual Associations in GPT. NeurIPS 2022.

[3] Hase et al. Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models. NeurIPS 2023.

[4] Saynova et al. Fact Recall, Heuristics or Pure Guesswork? Precise Interpretations of Language Models for Fact Completion. Findings of ACL 2025.

Weakness 2. The paper lacks evaluation of sequential and batch editing or unlearning. This question matters for assessing the practical benefits of dictionary reuse and selective addressing. We sincerely thank the reviewer for highlighting the importance of sequential and batch updating. To address this concern, the following experiments are conducted, with particular attention to dictionary reuse and selective addressing. Specifically, in sequential-batch updating, each round processes a new batch of requests using the model obtained from the preceding round. For KEY, the knowledge-addressing dictionary is constructed only once and reused throughout all updating rounds, while atom filtering and sparse-coefficient optimization are performed separately for each incoming batch.

Firstly, following the experimental setup of AlphaEdit [1], Llama-3-8B is evaluated on CounterFact with 100 facts per round over 20 consecutive rounds, totaling 2,000 edits. As shown in Table B, KEY achieves comparable or superior performance to AlphaEdit across all three evaluation metrics at every reported checkpoint. After 2,000 edits, KEY achieves 99.0% Efficacy Success (ES), 95.3% Paraphrase Success (PS), and 72.2% Neighborhood Success (NS), outperforming AlphaEdit by 0.1, 1.0, and 4.4 percentage points, respectively. These results indicate that KEY maintains comparable sequential-editing effectiveness.

Table B. Performance comparsion (reported as ES / PS / NS (%); higher is better) for sequential-batch knowledge editing on CounterFact with Llama-3-8B.

\hline \text{Method} & \text{Round 5} & \text{Round 10} & \text{Round 15} & \text{Round 20} \\\\ \hline \text{AlphaEdit} & 99.8 / 93.5 / 76.8 & 99.6 / 93.6 / 74.1 & 99.2 / 93.9 / 70.4 & 98.9 / 94.3 / 67.8 \\\\ \hline \mathbf{KEY} & 99.8 / 95.9 / 82.1 & 99.7 / 94.3 / 80.5 & 99.3 / 95.1 / 77.9 & 99.0 / 95.3 / 72.2 \\\\ \hline \end{array}$$ Secondly, following the experimental setting of ASU [2], Llama-2-Chat-7B is evaluated on TOFU, with each round forgetting either 1\% or 5\% of the dataset's authors over 10 consecutive rounds. For a fair comparison, Forgetting Efficacy (FE) and Model Utility (MU) are adopted from [2] to measure forgetting performance on the current round forget set and on the remaining retain set, respectively. As reported in `Table C`, KEY consistently achieves higher MU than ASU at every reported checkpoint under both settings. With 1\% of authors removed per round, KEY also achieves higher FE at rounds 1, 5, and 10. Under the more demanding 5\% setting, although both methods exhibit fluctuations in FE across rounds, KEY still attains a higher FE than ASU. Overall, these results indicate that KEY preserves better model utility while maintaining competitive forgetting effectiveness across sequential unlearning. **Table C. Performance comparsion (reported as FE / MU (%); higher is better) for sequential-batch knowledge unlearning on TOFU with Llama-2-Chat-7B .** $$\begin{array}{|c|l|c|c|c|c|} \hline \text{Authors removed} & \text{Method} & \text{Round 1} & \text{Round 3} & \text{Round 5} & \text{Round 10} \\\\ \text{per round} & & & & & \\\\ \hline 1\\% & \text{ASU} & 81.2 / 76.8 & 78.7 / 75.3 & 78.2 / 74.9 & 74.5 / 74.1 \\\\ \hline 1\\% & \mathbf{KEY} & 82.5 / 78.7 & 76.3 / 77.1 & 81.7 / 76.2 & 76.4 / 74.7 \\\\ \hline 5\\% & \text{ASU} & 76.6 / 73.6 & 75.1 / 72.0 & 75.2 / 71.4 & 77.2 / 67.0 \\\\ \hline 5\\% & \mathbf{KEY} & 78.1 / 75.5 & 79.2 / 74.8 & 75.5 / 74.1 & 77.9 / 71.7 \\\\ \hline \end{array}$$ Both results clearly demonstrate the effectiveness of KEY in sequential-batch knowledge updating, highlighting its ability to preserve selective addressing while reusing a fixed dictionary across successive rounds. We thank the reviewer again for this broader evaluation. Detailed experimental settings, complete results, and further analysis are included in `Section 5.4`. [1] Fang et al. *AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models.* ICLR 2025. [2] Zade et al. *Attention Smoothing Is All You Need For Unlearning.* ICLR 2026.
AuthorsOpenReview ↗

[Part 3]Respond to reviewer XSqy

Question 1. Do the differing perturbation responses also occur on editing datasets such as CounterFact or FINE-KED? A corresponding experiment would help assess whether the observation extends to the editing setting.

Thank you for this helpful suggestion. To examine whether the observation extends to knowledge editing, the motivation analysis in Section 3 is extended to CounterFact using 20 facts correctly recalled by the base model. The results show that substantial parameter overlap persists across distinct facts from the knowledge editing task, while identical perturbations to shared parameter locations also induce different target-logit responses. These findings are consistent with those on unlearning tasks (TOFU), providing strong empirical evidence. Detailed experimental settings and overlap statistics are provided in the response to Weakness 1 and Table A, while the corresponding results are included in Appendix B.

Question 2. Could you report update effectiveness as the number of sequential requests or the batch size increases?

We sincerely thank the reviewer for this helpful suggestion. Tables B and C in the response to Weakness 2 report update effectiveness across successive rounds of knowledge editing and unlearning. As shown, KEY consistently achieves competitive or superior results in sequential updating, while maintaining the highest model utility across all checkpoints. These results demonstrate the effectiveness of selective knowledge addressing and the feasibility of reusing a fixed dictionary as the number of sequential requests or the batch size increases. Detailed settings and results are accordingly provided in Section 5.4.

Thank you again for all your thorough and insightful feedback, which has been instrumental in improving our work. We hope our responses and the additional results have fully addressed your concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of reviewers feedback and authors response

Dear Area Chair and reviewers,

We sincerely appreciate the insightful reviews and constructive discussion. In this note, we summarize our key contributions and the main outcomes of the discussion period. All concerns raised by Reviewers XSqy, 1Ztj, and P1qk have been addressed point by point through detailed clarifications and additional experiments, and all new results have been incorporated into the revised manuscript (highlighted in blue).

Existing knowledge editing and unlearning methods often suffer from knowledge entanglement, where different knowledge items share overlapping parameters, so that updating one item may unintentionally interfere with others. This paper presents KEY, a knowledge addressing dictionary framework that constructs a shared orthogonal atom dictionary from knowledge-induced gradients and represents each update as a sparse, knowledge-specific combination of atoms, enabling selective knowledge updating within overlapping parameters. KEY unifies knowledge editing and unlearning within a single framework and is supported by bounded-proximity analysis for stable knowledge addressing. Across editing and unlearning benchmarks on multiple LLM backbones, KEY achieves competitive performance while substantially improving the preservation of non-target knowledge.

In addition, we group the reviewers’ concerns into the following categories (in order of appearance) and summarize the corresponding experiments and clarifications from our previous responses.

  1. Generality of the motivation study. Repeated the localization and perturbation analyses within a jointly trained model and extended the analysis to CounterFact. Both settings show substantial parameter overlap alongside differentiated perturbation responses, strengthening the empirical motivation for knowledge addressing (Reviewer XSqy W1/Q1).
  2. Sequential updates and dictionary reuse. Added sequential-batch experiments on CounterFact (2,000 edits over 20 rounds, vs. AlphaEdit) and TOFU (10 rounds with 1%/5% forgetting per round, vs. ASU), showing that a dictionary constructed once can be reused without reconstruction, with higher neighborhood success and model utility at every checkpoint (Reviewer XSqy W2/Q2; Reviewer 1Ztj W2/Q2; Reviewer P1qk W2/Q3).
  3. Dependence on auxiliary knowledge. Evaluated dictionaries built using smaller knowledge pools and a distribution-mismatched general pool disjoint from the update targets, and introduced coverage-guided dynamic dictionary updating via incremental SVD to recover most of the performance without prior knowledge of future requests (Reviewer P1qk W1/Q1; Reviewer 1Ztj Q2).
  4. Computational costs and scalability. Reported full dictionary construction costs (time, peak GPU memory, and storage) and evaluated cost scaling across model sizes (1B–13B) and atom budgets. End-to-end evaluation on 2,000 edits shows that KEY is 47.3% and 37.4% faster than ROME and MEMIT, respectively, even with preprocessing included (Reviewer 1Ztj W2/Q3; Reviewer P1qk W2/Q2).
  5. Comparison fairness and performance trade-offs. Clarified fair comparisons under matched evaluation protocols and added harmonic-mean evaluation and Forget Quality analysis to assess performance trade-offs (Reviewer 1Ztj W1/Q1, W5/Q5).
  6. Scope of claims, theory, and positioning. Clarified that the bounds in Appendix D characterize addressing stability, added a case study showing that semantically related facts receive distinct knowledge addresses, and sharpened the positioning of KEY against representation-separation and parameter-constraint methods such as DiKE (Reviewer 1Ztj W3/W4/Q4).

All concerns raised by the reviewers have been addressed through the detailed clarifications, additional experiments, and analyses summarized above. Importantly, all three reviewers have explicitly confirmed that their concerns have been resolved.

The reviewers’ comments have greatly improved the clarity and completeness of the manuscript. In particular, the suggestions on sequential updating and dynamic dictionary construction motivated additional experiments that further demonstrate the practical value of KEY. We sincerely appreciate the reviewers’ and Area Chair’s time and support, and we welcome any further discussion.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.