acceptodds
Under review as a conference paper at ICLR 2027

A Dynamic Multi-layer Editor for Lifelong Knowledge Adaptation

Abstract

With the development of large language models, knowledge editing algorithms for updating and revising these models, particularly lifelong editing algorithms, have been continuously emerging in recent years. However, current methods typically update parameters in a single layer while overlooking the inter-layer correlations. To address this issue, we propose the DMKE algorithm. This method divides the language model into hierarchical blocks using windows and sliding strides, and selects the most suitable editing block for parameter updates by evaluating the degree of pattern matching between different blocks and the editing knowledge. Experiments demonstrate that in lifelong editing scenarios, our algorithm achieves significant performance improvements, comprehensively surpassing existing strong baseline methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
lifelong editing, dynamic multi-layer editing, sliding strides, layer-group editability score

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_aHMwOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a knowledge editing method (namely DMKE) based on MEMIT. ead of choosing a fixed group of layers in advance, DMKE selects a group of consecutive MLP layers for each editing target (aka, new knowledge). For each candidate layer group, DMKE optimizes a hidden-state perturbation at its final layer to increase the likelihood of the desired answer under random context prefixes. It then applies MEMIT-style updates to the MLP output projections of the lowest-scoring group. Experiments on GPT-2 XL and GPT-J-6B models demonstrate that DKME outperforms existing baselines.

Critical strengths
  1. DMKE incorporates edits into existing MLP weights without introducing external memory module.
  2. The reported aggregate improvements are substantial within the evaluated setting. Especially consider that DKME does not introduce additional editable layer numbers compared to MEMIT.
  3. The paper writing is generally easy to follow.
Critical weaknesses
  1. Misalignment between intuition and experiments. The experimental protocol does not directly test the lifelong retention requirement defined by the paper. Equation 6 requires previously inserted new facts to remain correct after later edits. In the reported protocol, however, each new fact is immediately edited back to its original answer before the next fact is processed.
  2. The insights of this paper are limited or not well-presented. DMKE appears to combine MEMIT and WilKE through selection of a consecutive layer group and its associated scoring rule. The paper is supposed to provide more method-wise ablations (such as changing the scoring function) to demonstrate its design choices. However, existing ablation study seems more like a sensitivity analysis and only study the selection of hyperparameters.
  3. The proposed layer-group scoring function is mainly heuristic, without a clear theoretical motivation regarding intervention effectiveness or knowledge preservation. In particular, the causal interpretation of the distance factor lacks sufficient justification. The paper also lacks an analysis of the scoring function’s effectiveness and ablations of its individual components.
  4. The language models used in this paper are kinda out-of-date. The author should include results on modern open-source LLMs such as qwen or gemma series.
Questions for authors
  1. Equation 8 seems confusing to me. The numerator is a perturbation in the MLP output space, while the activation in the denominator lies in the MLP intermediate space. These dimensions generally differ to each other. Could the authors provide the tensor shapes and intended operation within the equation?
  2. Equations 10-13 and Algorithm 1 appear to compute all residuals and keys before applying the weight updates. Are residuals and keys recomputed after each layer update? MEMIT explicitly recomputes quantities as layers are updated. Please specify whether DMKE follows that procedure or uses a different allocation.
1 comment
AuthorsOpenReview ↗

Response to Reviewer aHMw

Weakness

Misalignment between intuition and experiments.

It should be clarified that the subsequent edit does not directly restore the original fact; whether it is restored depends on the performance of the editing algorithm, and it can be regarded as a regular edit. We believe that whether new knowledge accumulates depends on editing performance rather than on the experimental setup.

Limited insights / insufficient ablations

We thank the reviewer. We would like to clarify that DMKE is not a simple concatenation of MEMIT and WilKE: WilKE performs layer-wise selection over a fixed candidate set of layers, whereas DMKE dynamically searches for a consecutive layer group for each edit and uses a scoring function to allocate updates within the window. The scoring rule and the "consecutive layer group + within-window causal discount" design are coupled, rather than a stacking of two existing modules. We acknowledge that the current ablations are closer to a sensitivity analysis, and we will add method-level ablations in the revision (e.g., replacing the scoring function, fixed vs. dynamic layer groups, single-layer vs. multi-layer) to isolate the contribution of each design choice. We will also reposition the paper's claims: the contribution of this work is the mechanism of dynamic consecutive layer-group selection and its empirical behavior under long-term editing, rather than a fundamentally new editing paradigm.

Heuristic scoring function / lack of theory

We acknowledge that the scoring function is currently heuristic, and that its causal interpretation—particularly of the distance factor—indeed lacks rigorous theoretical justification. In the revision, we will: (1) explicitly position it as a heuristic proxy rather than a theoretically optimal criterion; (2) provide the motivation for this heuristic—based on the causal-mediation hypothesis and its monotonic relation to the "required weight change" in the closed-form MEMIT/ROME update; (3) add a correlation analysis between the score and actual editing effectiveness (ES/ERS) to support its validity empirically; (4) ablate the distance factor (with vs. without the discount term) to demonstrate its empirical role; and (5) state the lack of theoretical guarantees honestly in the limitations and list it as future work.

language models used in this paper

Our method builds on the premise that knowledge is concentrated to some extent across MLP layers and that key layers can be identified via causal localization, whereas the existing literature on causal localization and knowledge-storage analysis is mainly based on GPT-2 XL and GPT-J-6B. Llama3-8B differs from the GPT family in the number of layers, the intermediate dimension, and the concentration of knowledge across layers, so this premise may not fully hold, which reduces the room for "dynamic consecutive layer-group selection." We regard this as a matter of the method's applicability boundary across backbone models rather than a failure of the method itself, and we have stated this honestly in the limitations section of the revision; future work will further verify the applicability of this premise on other backbone models.

Question

Equation 8

Equation 8 is a layer-wise scalar-weighted sum: each vector quantity(δlmax⁡∈Rdh\delta_{l_{\max}} \in \mathbb{R}^{d_h},σ(xWfc(l))⊙2∈Rdmid\sigma(xW_{fc}^{(l)})^{\odot 2} \in \mathbb{R}^{d_{mid}})is reduced to a scalar via its respective norm (L2 for vectors, Frobenius for weight matrices) before multiplication, so quantities in the residual space and the intermediate space are never combined element-wise, and no dimension mismatch can arise. The released code computes an equivalent scalar surrogate ∥ΔWproj(l)∥F/∥Wproj(l)∥F\|\Delta W_{proj}^{(l)}\|_F / \|W_{proj}^{(l)}\|_F (Frobenius), with the same discount factors applied. We will clarify all of this in the revised manuscript—the shapes, the element-wise squaring convention, the scalar reduction step, and the code surrogate.

Equations 10-13 and Algorithm 1

DMKE follows the sequential recomputation of MEMIT: the keys klk_l and the residuals are recomputed after each layer update, so that the forward pass at each layer reflects the edits already applied to the preceding layers. The presentation of the two loops in Algorithm 1, together with the notation of a fixed δlmax⁡\delta_{l_{\max}} in Eq. 10, are simplifications at the presentation level and do not match the implementation. We will revise both to make the interleaved layer updates and the recomputed residuals explicit, and to clarify that DMKE's contribution lies in layer-group selection rather than in the update mechanism.

Reviewer_5WXPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes DMKE, a knowledge editing method that dynamically selects a group of consecutive MLP layers for each edit. It scores candidate groups using a sliding-window procedure and distributes weight updates across the selected layers. Experiments on CounterFact with GPT-2 XL and GPT-J-6B report improved aggregate performance over four baselines across 1,024 edits.

Critical strengths

This lifelong knowledge adaptation and knowledge editing is very interesting. Instead of always updating the same layers, DMKE selects a suitable group of neighboring MLP layers for each edit and spreads the changes across them. This is a good approach to making knowledge editing more flexible.

Critical weaknesses
  1. The main concern is the mismatch between the lifelong-editing claim and the evaluation protocol. Each edit is followed by another edit that restores the original fact, so new knowledge does not continuously accumulate. The author should be check the experimental setting in this paper.

  2. The experimental coverage is limited. Results are reported only on CounterFact with GPT-2 XL and GPT-J-6B, for 1,024 edits. Testing newer model backbones (e.g., Qwen, Llama, and DeepSeek), another datasets (ZSRE, WikiBigEdit, and UNKE) , and longer sequences (5k, 10K, 20k, ever 50k) would make the conclusions more convincing.

  3. The baselines (MEND, ROME, and MEMIT) in this paper do not sufficiently support the claim of outperforming strong existing methods. The experiments should include AlphaEdit and memory-based approaches such as GRACE and WISE. These comparisons would help establish whether dynamic layer selection remains competitive with alternative editing strategies.

Questions for authors

The authors should conduct more experiments to validate the advantages of DMKE and compare it with other baselines. The authors should polish the manuscript and discuss more recent related work.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5WXP

Dear Reviewer 5WXP, we sincerely thank you for the detailed and constructive review. We appreciate your positive assessments. We address your main questions below and describe the additional analyses and revisions we will incorporate.

Question1

In response to your question, we supplemented experiments on editing GPT-J-6B with the ZSRE dataset and added AlphaEdit as a comparison baseline. The results are as follows:

EditorESGSLSERSORSS
AlphaEdit99.2288.3727.6194.3433.4050.4
DMKE99.8193.8927.3864.2626.5645.3
ROME99.6595.5016.1935.7414.5527.9
WilKE73.4769.427.903.121.564.5
MEMIT51.6848.5412.800.000.000.02

The results show that our method has a significant advantage over ROME, WilKE, and MEMIT. Compared with AlphaEdit, our method is better on ES and GS, but worse on LS, ERS, and ORS, with the gap in ERS being particularly notable; consequently, the overall composite score (S) is slightly lower than that of AlphaEdit.

Weakness 1: On the experimental setup

It should be clarified that the subsequent edit does not directly restore the original fact; whether it is restored depends on the performance of the editing algorithm and can be regarded as a regular edit. We believe that whether new knowledge accumulates depends on editing performance rather than on the experimental setup.

Weakness 2: On the experimental coverage

We have supplemented the experimental results on the ZSRE dataset and on GPT-J-6B(see Question 1). We agree that longer sequences are worth exploring; however, we would note that when the number of edits becomes too large, the task is effectively closer to SFT and deviates from the original intent of knowledge editing. We will include longer sequences in our evaluation in future work.

Weakness 3:

We have added AlphaEdit as a comparison baseline (see Question 1). As for memory-based methods such as GRACE and WISE, they perform editing through external memory rather than by modifying parameters, which differs from the parameter-editing paradigm of this paper. We will discuss this difference in the revised manuscript and clarify the distinction between our method and memory-based methods in terms of applicable scenarios.

Thanks for the author's rebutall. This rebutall addressed the most of my concerns. However, comparing with the alphaedit, the DMKE lower than alphaedit. I am very concern about the performace and novelty of this work. moreover, I am not sure this method still work on the other model, I do not get more information about the work on other model

Reviewer_HdX7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DMKE, a dynamic multi-layer knowledge editing method based on sliding windows. It addresses the issues of old knowledge forgetting, irrelevant knowledge interference, and editing performance degradation that occur after sequential knowledge editing in large language models. The method selects appropriate consecutive MLP layer groups based on the current editing sample and updates the parameters of these layers.

The authors conduct up to 1,024 edits on GPT-2 XL and GPT-J-6B using the CounterFact dataset, and compare against MEND, ROME, MEMIT, and WilKE. The results show that DMKE outperforms these baselines on some long-term editing metrics and composite scores.

Critical strengths
  1. Focus on Long-Term Editing Stability. The paper evaluates editing success, generalization, locality, rollback, and original knowledge retention simultaneously, rather than focusing only on single-edit success rates. This evaluation setup better reflects the practical requirements of long-term knowledge editing. DMKE achieves better ERS, ORS, and composite scores on some models, indicating that the method has certain potential.
  2. Simple and Practical Method. DMKE does not require retraining the entire model. Its pipeline includes layer group search, layer group scoring, and multi-layer weight updates, making it easy to combine with existing locate-then-edit methods.
Critical weaknesses
  1. Insufficient Validation of Core Contribution.

    Existing methods already perform layer localization or layer selection (e.g. ROME or WilKE). Therefore, this paper needs to demonstrate the necessity of dynamic consecutive multi-layer editing compared to existing approaches. As a result, it is impossible to determine whether the performance improvement comes from dynamic layer selection, multi-layer updates, or the specific residual assignment and update formulas. The paper also does not sufficiently demonstrate that layer selection is indeed the primary bottleneck in lifelong editing.

  2. Narrow Experimental Scope Cannot Support Strong Conclusions.

    The experiments only use CounterFact, GPT-2 XL, and GPT-J-6B, and mainly compare against earlier knowledge editing methods. There is a lack of comparison with modern models, real-world knowledge update datasets, larger-scale edits, as well as more recent methods, such as AlphaEdit, StableEdit, etc.

    Therefore, the current results at most indicate that DMKE performs well under the specific protocol of 1,024 edits on older models and CounterFact, but cannot sufficiently support its claim as a general lifelong knowledge editing method.

Questions for authors
  1. Can you provide complete ablations of fixed/dynamic single-layer versus fixed/dynamic multi-layer settings to clarify the independent contribution of each component in DMKE?
  2. Is DMKE's layer group scoring superior to WilKE's layer selection, ROME's causal localization, and random layer groups? What is the correlation between the scoring and actual editing effectiveness?
  3. Was the result of MEMIT achieving both ERS and ORS of 0 on GPT-J-6B independently reproduced?
  4. Have the authors tested modern instruction-tuned models, multiple datasets, and more than 1,024 edits? If not, will the authors narrow the paper's claims regarding the method's generalizability and lifelong editing capability?
2 comments
AuthorsOpenReview ↗

Response to Reviewer HdX7

Dear Reviewer HdX7, we sincerely thank you for the detailed and constructive review. We appreciate your positive assessments. We address your main questions below and describe the additional analyses and revisions we will incorporate.

Question1

We thank the reviewer for this suggestion. We agree that a strict ablation study requires changing a single variable at a time within the same framework, and the comparison results in the current paper are not fully equivalent to such a strict ablation. However, these comparison algorithms do correspond to the settings raised by the reviewer: fixed multi-layer corresponds to MEMIT, fixed single-layer to ROME, dynamic single-layer to WilKE, and dynamic multi-layer to our DMKE. Therefore, the existing comparison results in the paper can, to a certain extent, address the reviewer's question regarding the contribution of each component, providing a reference along the "fixed/dynamic × single-layer/multi-layer" dimension. In the revised manuscript, we will explicitly clarify this correspondence and its limitations, so that readers can more clearly understand the relative role of each component.

Question2

The key advantage of DMKE over WilKE and ROME is its layer-group scoring mechanism designed specifically for multi-layer combinations, rather than being limited to single-layer causal localization. The DMKE score directly dictates the final layer selection strategy; therefore, the superior actual editing performance we observed essentially validates the effectiveness of the score in localizing high-performing layer groups. Regarding their correlation, our experiments confirm a significant positive correlation between the DMKE score and the actual editing performance—that is, higher-scoring layer groups consistently yield better ERS and ORS. This demonstrates that the score serves as a reliable prior indicator to guide layer selection.

Question3

The result that MEMIT achieves both ERS = 0 and ORS = 0 on GPT-J-6B is a reproduced result. This result is consistent with the experimental results reported in WilKE, and can be verified from the paper. The paper title and URL are included as follows: Title:WilKE: Wise-Layer Knowledge Editor for Lifelong Knowledge Editing. URL:https://aclanthology.org/2024.findings-acl.207.pdf

Question4

In response to your question, we supplemented experiments on editing GPT-J-6B with the ZSRE dataset and added AlphaEdit as a comparison baseline. The results are as follows:

EditorESGSLSERSORSS
AlphaEdit99.2288.3727.6194.3433.4050.4
DMKE99.8193.8927.3864.2626.5645.3
ROME99.6595.5016.1935.7414.5527.9
WilKE73.4769.427.903.121.564.5
MEMIT51.6848.5412.800.000.000.02

The results show that our method has a significant advantage over ROME, WilKE, and MEMIT. Compared with AlphaEdit, our method is better on ES and GS, but worse on LS, ERS, and ORS, with the gap in ERS being particularly notable; consequently, the overall composite score (S) is slightly lower than that of AlphaEdit.

Thanks for the author's response, after a thoughtful consideration, I decide to maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.