- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a knowledge editing method (namely DMKE) based on MEMIT. ead of choosing a fixed group of layers in advance, DMKE selects a group of consecutive MLP layers for each editing target (aka, new knowledge). For each candidate layer group, DMKE optimizes a hidden-state perturbation at its final layer to increase the likelihood of the desired answer under random context prefixes. It then applies MEMIT-style updates to the MLP output projections of the lowest-scoring group. Experiments on GPT-2 XL and GPT-J-6B models demonstrate that DKME outperforms existing baselines.
Critical strengths
- DMKE incorporates edits into existing MLP weights without introducing external memory module.
- The reported aggregate improvements are substantial within the evaluated setting. Especially consider that DKME does not introduce additional editable layer numbers compared to MEMIT.
- The paper writing is generally easy to follow.
Critical weaknesses
- Misalignment between intuition and experiments. The experimental protocol does not directly test the lifelong retention requirement defined by the paper. Equation 6 requires previously inserted new facts to remain correct after later edits. In the reported protocol, however, each new fact is immediately edited back to its original answer before the next fact is processed.
- The insights of this paper are limited or not well-presented. DMKE appears to combine MEMIT and WilKE through selection of a consecutive layer group and its associated scoring rule. The paper is supposed to provide more method-wise ablations (such as changing the scoring function) to demonstrate its design choices. However, existing ablation study seems more like a sensitivity analysis and only study the selection of hyperparameters.
- The proposed layer-group scoring function is mainly heuristic, without a clear theoretical motivation regarding intervention effectiveness or knowledge preservation. In particular, the causal interpretation of the distance factor lacks sufficient justification. The paper also lacks an analysis of the scoring function’s effectiveness and ablations of its individual components.
- The language models used in this paper are kinda out-of-date. The author should include results on modern open-source LLMs such as qwen or gemma series.
Questions for authors
- Equation 8 seems confusing to me. The numerator is a perturbation in the MLP output space, while the activation in the denominator lies in the MLP intermediate space. These dimensions generally differ to each other. Could the authors provide the tensor shapes and intended operation within the equation?
- Equations 10-13 and Algorithm 1 appear to compute all residuals and keys before applying the weight updates. Are residuals and keys recomputed after each layer update? MEMIT explicitly recomputes quantities as layers are updated. Please specify whether DMKE follows that procedure or uses a different allocation.
Response to Reviewer aHMw
Weakness
Misalignment between intuition and experiments.
It should be clarified that the subsequent edit does not directly restore the original fact; whether it is restored depends on the performance of the editing algorithm, and it can be regarded as a regular edit. We believe that whether new knowledge accumulates depends on editing performance rather than on the experimental setup.
Limited insights / insufficient ablations
We thank the reviewer. We would like to clarify that DMKE is not a simple concatenation of MEMIT and WilKE: WilKE performs layer-wise selection over a fixed candidate set of layers, whereas DMKE dynamically searches for a consecutive layer group for each edit and uses a scoring function to allocate updates within the window. The scoring rule and the "consecutive layer group + within-window causal discount" design are coupled, rather than a stacking of two existing modules. We acknowledge that the current ablations are closer to a sensitivity analysis, and we will add method-level ablations in the revision (e.g., replacing the scoring function, fixed vs. dynamic layer groups, single-layer vs. multi-layer) to isolate the contribution of each design choice. We will also reposition the paper's claims: the contribution of this work is the mechanism of dynamic consecutive layer-group selection and its empirical behavior under long-term editing, rather than a fundamentally new editing paradigm.
Heuristic scoring function / lack of theory
We acknowledge that the scoring function is currently heuristic, and that its causal interpretation—particularly of the distance factor—indeed lacks rigorous theoretical justification. In the revision, we will: (1) explicitly position it as a heuristic proxy rather than a theoretically optimal criterion; (2) provide the motivation for this heuristic—based on the causal-mediation hypothesis and its monotonic relation to the "required weight change" in the closed-form MEMIT/ROME update; (3) add a correlation analysis between the score and actual editing effectiveness (ES/ERS) to support its validity empirically; (4) ablate the distance factor (with vs. without the discount term) to demonstrate its empirical role; and (5) state the lack of theoretical guarantees honestly in the limitations and list it as future work.
language models used in this paper
Our method builds on the premise that knowledge is concentrated to some extent across MLP layers and that key layers can be identified via causal localization, whereas the existing literature on causal localization and knowledge-storage analysis is mainly based on GPT-2 XL and GPT-J-6B. Llama3-8B differs from the GPT family in the number of layers, the intermediate dimension, and the concentration of knowledge across layers, so this premise may not fully hold, which reduces the room for "dynamic consecutive layer-group selection." We regard this as a matter of the method's applicability boundary across backbone models rather than a failure of the method itself, and we have stated this honestly in the limitations section of the revision; future work will further verify the applicability of this premise on other backbone models.
Question
Equation 8
Equation 8 is a layer-wise scalar-weighted sum: each vector quantity(δlmax∈Rdh,σ(xWfc(l))⊙2∈Rdmid)is reduced to a scalar via its respective norm (L2 for vectors, Frobenius for weight matrices) before multiplication, so quantities in the residual space and the intermediate space are never combined element-wise, and no dimension mismatch can arise. The released code computes an equivalent scalar surrogate ∥ΔWproj(l)∥F/∥Wproj(l)∥F (Frobenius), with the same discount factors applied. We will clarify all of this in the revised manuscript—the shapes, the element-wise squaring convention, the scalar reduction step, and the code surrogate.
Equations 10-13 and Algorithm 1
DMKE follows the sequential recomputation of MEMIT: the keys kl and the residuals are recomputed after each layer update, so that the forward pass at each layer reflects the edits already applied to the preceding layers. The presentation of the two loops in Algorithm 1, together with the notation of a fixed δlmax in Eq. 10, are simplifications at the presentation level and do not match the implementation. We will revise both to make the interleaved layer updates and the recomputed residuals explicit, and to clarify that DMKE's contribution lies in layer-group selection rather than in the update mechanism.