- Rating
- 3(+1)
- Confidence
- 1(−3)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes Gradient Drift Regularization (GDR) to mitigate the "catastrophic forgetting" of previously acquired knowledge during sequential knowledge editing in language models. The method preserves the residual-space gradients from historical edits (captured at the time of insertion) and recomputes gradients for a subset of historical samples prior to subsequent edits. The squared coordinate-wise difference between these two sets of gradients is then used as a quadratic regularization weight for the current residual optimization. A top-k mechanism, based on covariance-preconditioned key interactions, is employed to constrain the computational cost of calculating historical gradients. The authors derive a Taylor bound for the centered local response difference and demonstrate strong performance in both knowledge editing and the retention of previously learned information across three different models and benchmarks such as CounterFact and ZsRE.
Critical strengths
- Clear design addressing a practical problem. The issue of cumulative interference during sequential editing is a significant area of study. By leveraging the discrepancy between historical reference states and the current state, GDR provides a regularization signal that differs from standard single-state importance estimation (Section 5).
- Ease of integration and well-defined computational bounds. The weights defined in Equation 15 remain fixed during residual optimization, and the top-k mechanism limits the overhead of computing historical gradients. Appendix D.6 transparently acknowledges that both the scanning of historical samples and the storage requirements scale with the number of edits performed.
- Targeted experimental comparisons. Tables 1 and 2 cover three models and two editing datasets, including comparisons against baselines such as AlphaEdit and HSE. Tables 4 and 6 provide further analysis of selection strategies, response signals, and weight transformations. Notably, for the LLaMA model on the CounterFact generalization task, the performance improves from 78.15 (AlphaEdit) to 93.95—a significant gain, provided it can be verified.
Critical weaknesses
- Inconsistencies in statistical reporting. In Appendix C.3, error is defined as the instance-level Standard Error of the Mean (SEM), and binomial standard errors are provided for binary metrics; however, there are discrepancies between these definitions and some of the values presented in the tables. If the "CounterFact efficacy" values in Table 1 are derived from binary judgments for each of 1,000 individual edits, then the 97.50% reported for LLaMA GDR should correspond to a margin of error of approximately 0.494 percentage points (rather than the reported 0.28), and the 50.90% for MEMIT should correspond to approximately 1.581 (rather than 0.76). Appendix C.1 specifies that the RTE evaluation uses 277 samples; however, the figures of 78.30% and 68.12% in Table 7 cannot be obtained by rounding the result of an integer number of correct predictions divided by 277. While differences in sample units or averaging across multiple runs might account for some of these discrepancies, the text provides no clarification. This directly impacts the reproducibility of the results; furthermore, instance-level standard error of the mean (SEM) cannot serve as a substitute for accounting for the order of edits or the variance across experimental runs.
- The independent role of the gradient difference across states has not been sufficiently established. The perturbation term (representing shared historical samples) in the theorem is not the same entity as the residual (representing the current edit) in Equation 15; the mechanism by which parameter updates bridge these two remains undemonstrated. The "anchor" is derived from the model prior to the update plus an optimized residual, while the "query" is derived from the model after the update—the difference between them may also incorporate the initial update error. When the anchor gradient is small, the squared drift approximates the squared current gradient; the "anchor Fisher" data in Table 6 does not rule out this alternative explanation. Employing controls such as "query-gradient-only," using a post-update anchor, or implementing residual regularization (e.g., by matching update norms or shuffling coordinates) would provide a more direct test of the proposed mechanism. While the current results support an empirical correlation, they are insufficient to definitively distinguish between these potential explanations. Admittedly, this is a higher-level critique and not grounds for rejection; however, the paper would be strengthened if the authors could design experiments to further substantiate the robustness or scope of this empirical correlation.
- There is a correctable mathematical inconsistency in the formulation of the closed-form update. The exact interpolation condition in Eq. 4 generally does not follow from Eq. 5. With C=K₀K₀ᵀ≻0, r=v*−Wk, and t=kᵀC⁻¹k, Eq. 5 gives Δk=r·t/(1+t), rather than r. Deriving Eq. 29 from Eq. 27 also requires the background residual term to vanish, for example when V₀=WK₀; approximate satisfaction alone does not yield the stated equality. The paper should specify whether background values are refreshed and how the covariance scaling in C.3 enters these equations. Equation 61 also reuses the edited target for unrelated locality questions without defining their own correct answers.
- The scope of certain diagnostic interpretations lacks clarity. In Figures 2 and 6, the "earliest 20%" cohort varies with ; keeping this early cohort fixed would be more appropriate for directly illustrating the trajectory of "forgetting." For Table 3, Figure 5, and the timing analysis in C.3, the specific datasets and the criteria used for defining the temporal boundaries should be specified. Furthermore, the "Edit Score" in Table 3 lacks an explicit formula.
Questions for authors
see weakness 1,3,4.
If you can prove the experience relationship in weakness 2 by experiments, I will raise my score.
AFTER REBUTTAL
Essentially, I would have liked to give the author a score of 2.5, as they partially substantiated some of the weaknesses I identified; however, given the limited strength of this substantiation, I have opted to assign a final score of 3 with a confidence level of 1.
6 comments
Rebuttal by Authors (Part 2 of 2)
Part 1 explains how the anchor references differ. We continue W2 with the control experiments and then address W3 and W4.
W2(2) Anchor reference and the independent role of gradient drift (continued).
Table 7 (Appendix D.3.2) reports the requested controls after 1,000 sequential edits on LLaMA-3.1-8B-Instruct.
| Variant | CounterFact Hist. (%) | ZsRE Hist. (%) |
|---|---|---|
| GDR | 90.00 | 88.37 |
| Post-update anchor | 87.00 | 85.60 |
| Query-only | 85.00 | 84.50 |
| Shuffled GDR weights | 76.50 | 75.90 |
The post-update anchor preserves most of the gain. The original reference further improves retention. GDR exceeds query-only by 5.00 and 3.87 percentage points, respectively. Shuffling the same aggregate weight values reduces retention, supporting the importance of coordinate assignment.
Across these variants, Table 8 reports immediate-efficacy differences of at most 0.10/0.05 percentage points on CounterFact/ZsRE, with mean parameter-update norms within 1.8% of GDR.
Table 9 pairs writes from identical checkpoints and matches update norms per request. Mean historical displacement stays within 2% of GDR, and new-edit efficacy is comparable. GDR yields lower mean prediction damage on both datasets. On ZsRE, damage is 0.014 nats per target token for GDR, 0.024 for query-only, and 0.040 for shuffled weights. This supports coordinate-specific protection at matched update scales.
Table 10 (W1(3) in Part 1) also includes query-only. GDR achieves higher Hist. in every tested order, with means of 91.17% versus 87.50%. Sections 5.1 and 5.3 now link to these additional ablations and diagnostics.
W3 (Mathematical formulation).
We apologize for the imprecise formulation and thank the reviewer for identifying these issues. In Section 3, Eq. (4) now states a regularized least-squares objective, whose closed-form solution is Eq. (5), allowing a trade-off between target fitting and background preservation.
Appendix B.1 now defines relative to the current pre-write weights. Consequently, holds exactly, justifying the cancellation when deriving Eq. (29) from Eq. (27). This reference need not be explicitly constructed. Appendix C.3 specifies the scaled background matrix and its use in the update (Eq. (67)). We cache and reuse the second-moment estimates and define the output reference using the current weights.
We also corrected ZsRE specificity to use the locality question’s own reference answer (Eq. (66), Appendix C.1). Eq. (5) gives the same closed-form update under the clarified assumptions.
W4 (Scope of diagnostic interpretations).
(1) Fixed-cohort retention. Section 6.3 now explicitly clarifies that the earliest-20% cohort in Figs. 2 and 7 grows with . We retain this analysis and add Table 2, which tracks the same first 200 ZsRE edits on LLaMA-3.1-8B-Instruct throughout a shared 1,000-edit sequence. Drop is each method’s accuracy at minus its accuracy at , in percentage points.
| Method | Drop (pp) ↓ | |||
|---|---|---|---|---|
| GDR | 97.20 | 93.72 | 88.37 | 8.83 |
| AlphaEdit [2] | 96.90 | 87.45 | 76.25 | 20.65 |
| HSE [3] | 97.05 | 89.16 | 80.42 | 16.63 |
Starting accuracies differ by at most 0.30 percentage points, while GDR loses 8.83 points over the subsequent 800 edits, compared with 20.65 and 16.63 for AlphaEdit and HSE. This directly tracks forgetting without changing cohort membership.
(2) Datasets and timing boundaries. Table 4 and Fig. 4 (Appendix D.2) use 1,000 CounterFact edits on LLaMA-3.1-8B-Instruct. Table 4’s caption now states this setup. Fig. 6 also uses CounterFact (Appendix D.6).
Appendix C.3 times each request from before processing through the write and state updates, including GDR anchor computation and history updates. CUDA synchronization occurs at both boundaries. Timing excludes model loading, one-time covariance preparation, evaluation, and checkpointing. Table 4 and Fig. 6(a) average all 1,000 steps. Fig. 6(b) times the next write at the stated history length (Appendix D.6).
(3) Edit Score [1]. Appendix D.2 now explicitly defines the metric used in Table 4 and Fig. 4 as , with all three components expressed on the same percentage scale. This makes the reported editing-quality criterion explicit.
References
[1] Mass-Editing Memory in a Transformer.
[2] AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models.
[3] Hippocampal-like Sequential Editing for Continual Knowledge Updates in Large Language Models.
We have carefully considered your feedback and provided responses and additional details where relevant, and we look forward to hearing your thoughts and continuing the discussion.
As you can see, weaknesses 1, 3, and 4 are minor issues; I pointed them out simply to facilitate your revisions. Regarding weakness 2, while I acknowledge the empirical correlation, the underlying theoretical issues remain difficult to resolve. Therefore, I would like to see the authors design more rigorous boundary experiments to substantiate this correlation. The current manuscript is insufficient to support this point; consequently, I am maintaining my original rating.
Rebuttal by Authors (Part 1 of 2)
Thank you very much for your careful review and constructive comments.
W1 (Inconsistencies in statistical reporting).
(1) Efficacy and SEM. CounterFact Eff. assigns one binary outcome per edit, with percentage-point SEM 100p^(1−p^)/N. The two LLaMA entries in Table 1 highlighted by the reviewer correspond to 975 and 509 successes out of 1,000 edits, respectively.
We recomputed the CounterFact Eff. SEMs in Tables 1 and 3 (GPT-2-XL in Appendix D.1). Fractional or continuous scores, including ZsRE and the other CounterFact metrics, use s/N, where s is the sample SD across edit-level scores. ZsRE first averages next-token correctness given gold prefixes within each edit, then across edits. Section 6.1 and Appendices C.1 and C.3 now clarify these aggregation rules and SEM definitions.
(2) Downstream evaluation denominators and results. Generation or answer-parsing failures caused some examples to be skipped during aggregation, reducing the denominator. For GDR on RTE, the reported 68.12% came from 188 correct predictions among 276 scored examples after one example was omitted. Similar omissions affected GSM8K and SAMSum.
We added generation retries and parsing fallbacks, reran evaluation, and verified that every example was scored. On all 277 RTE examples, the corrected accuracies are 78.34% (217/277) before editing and 68.23% (189/277) after GDR. We updated Table 11 in Appendix D.4 (original Table 7) and clarified the evaluation splits in Appendix C.1. We apologize for these reporting errors and appreciate the reviewer’s careful checks.
(3) Variability across edit orders. To assess edit-order variability, we evaluated three shared random permutations of the same 1,000 CounterFact edits on LLaMA-3.1-8B-Instruct, resetting to the same initial checkpoint for each run. Table 10 summarizes the comparison.
These are means ± sample SD across the three orders, distinct from instance-level SEMs within one run. Hist. evaluates the first 200 edits of each permutation under the final model.
GDR scores higher on all four metrics in every tested order. Its ranges are 0.40, 0.60, 0.60, and 1.50 percentage points for Eff., Gen., Spe., and Hist., respectively. See Appendix D.3.4, Table 10, and Fig. 5 for per-order results.
W2 (Independent role of cross-state gradient differences).
(1) Connecting editing residuals to historical perturbations. In Theorem 1, ϵ is a common local probe of historical representations. For a fixed historical key, Appendix B.2, Eqs. (40)–(42), gives Δvj=Δikj=αijri, where ri is the layerwise residual allocated through Eq. (21). For a single additive target-layer write with ri=δi, this yields ϵj=αijδi. The current residual induces the historical perturbation through δi→ΔWi→Δhj.
Under its smoothness assumptions, Theorem 1 bounds differences between the centered responses to this common probe using gradient drift and curvature. Appendix B.3 further establishes ∑ej∈Bi⟨Δgj,δ⟩2≤dLGDR(i)(δ) (Eq. (57)), where Δgj=gjquery−gjanc and d=dim(δ). This provides a local diagonal surrogate for controlling first-order response differences. The controls in Part 2 assess actual multilayer writes.
(2) Anchor reference and the independent role of gradient drift. The original anchor is evaluated at the optimized-target reference, whereas the post-update anchor is evaluated on Gj immediately after the actual write (Eq. (74)). Eq. (78) separates insertion-time discrepancy from subsequent changes as gtquery−ganc=(gtquery−gpost)+(gpost−ganc).
On the fixed first 200 edits at T=1000, the aggregate norm ratio of insertion-time discrepancy to subsequent drift is 0.04 on CounterFact and 0.05 on ZsRE, with corresponding aggregate weight differences of 6% and 7% (Eq. (79), Appendix D.3.3). The final signal on this cohort is dominated by subsequent drift.
References
[1] Mass-Editing Memory in a Transformer.
[2] AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models.
We apologize for splitting this response due to the character limit and address the remaining W2 controls, W3, and W4 in Part 2.