- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes MoKE for sequential knowledge editing. It keeps the base model frozen, stores optimized hidden values in an external memory, and retrieves them through cosine similarity with threshold gating. Multiple context keys improve paraphrase coverage. Experiments report improved specificity over weight-editing baselines and sustained efficacy after 10,000 edits.
Critical strengths
-
With a modest additional memory footprint (269 MB for GPT2-XL and 688 MB for Llama3-8B), MoKE improves CounterFact specificity over DeltaEdit by 35.0 and 49.8 percentage points, respectively, in the controlled 3,000-edit comparison.
-
The experimental evaluation is reasonably thorough, covering two models and two datasets, a 10,000-edit scaling study, component ablations, and measurements of memory use and inference overhead.
Critical weaknesses
-
The novelty relative to NeuralDB [1] is insufficiently established. NeuralDB already uses an external KV database of optimized editing representations, cosine nearest-neighbor retrieval, and threshold gating. The central advantages emphasized here—avoiding repeated weight updates, preserving general capabilities, and supporting larger edit counts—are also central motivations and reported benefits of NeuralDB. Although MoKE differs in its retrieval representation, value-replacement mechanism, and context-key construction, these differences do not by themselves establish a substantial methodological contribution. The paper neither discusses NeuralDB nor provides a direct comparison or targeted ablations demonstrating the benefits of these departures. The context-key ablation supports one component, but leaves the overall contribution beyond existing work unclear.
-
The capacity study has limited coverage. The 10,000-edit comparison is restricted to GPT2-XL/CounterFact and DeltaEdit under its adopted configuration. Without competitive external-memory baselines and broader settings, the results primarily demonstrate an advantage over this particular weight-editing configuration.
Questions for authors
-
What is the central contribution beyond NeuralDB, given the substantial overlap in both mechanism and claimed advantages? Could you compare against NeuralDB under the same protocol and isolate the effects of retrieval representation, value replacement versus residual addition, and context-key construction?
-
Can the long-sequence evaluation be extended to additional model–dataset settings and competitive external-memory methods?
-
Some citations in the introduction are awkwardly formatted, such as “DeltaEdit Cao et al. (2026).” Appendix F also incorrectly refers to Table 12 for latency measurements.
-
What causes the large bf16–fp16 performance gap in DeltaEdit? Can an ablation using FP32 weight accumulation distinguish update-rounding errors from precision effects in target optimization or update computation?
4 comments
Longer edit sequences
We also agree that the original capacity experiment needed a competitive external-memory baseline. At 10,000 edits on GPT2-XL CounterFact, the mean scores are
| Method | Mean at 10,000 edits |
|---|---|
| DeltaEdit | 7.86 |
| GRACE | 66.46 |
| NeuralDB | 84.63 |
| MoKE | 85.63 |
For this 10,000-edit experiment, the method settings are selected using 2,000 edits and evaluated on the remaining 8,000. MoKE is ahead of NeuralDB by 1.08 [0.89, 1.27] points and is ahead on efficacy, generalization and specificity.
We also extended the DeltaEdit comparison to the other three model and dataset settings. At 10,000 edits, MoKE reaches a mean of 79.15 against 28.86 for DeltaEdit on Llama3-8B CounterFact, 72.65 against 13.10 on GPT2-XL ZsRE, and 74.83 against 73.23 on Llama3-8B ZsRE. The last setting is useful because DeltaEdit remains stable there, and the difference between the two approaches is correspondingly small.
Cause of the bf16 and fp16 difference
The reviewer also asked whether the large bf16 and fp16 difference for DeltaEdit comes from target optimization or from repeatedly accumulating updates in low precision.
DeltaEdit writes each edit directly into the same weight matrices. We therefore repeated the Llama3-8B CounterFact experiment while separating the precision used for computation from the precision used to accumulate the updates. The table reports the mean of efficacy, generalization and specificity after 3,000 edits.
| Compute | Weight accumulation | Mean |
|---|---|---|
| bf16 | bf16 in place | 33.67 |
| bf16 | FP32 master weights | 64.48 |
| fp16 | fp16 in place | 63.70 |
| fp32 | fp32 | 63.73 |
| fp32 | rounded to bf16 after every update | 4.48 |
Keeping FP32 master weights recovers the bf16 gap even though the edit computation itself remains in bf16. Conversely, FP32 computation collapses when the weights are rounded to bf16 after every update. This shows that the gap comes from repeatedly rounding the accumulated weight updates, rather than from the precision used to optimize the target.
This also explains why MoKE changes much less between bf16 and fp16, since it does not repeatedly write updates into the model weights.
Presentation points
We also thank the reviewer for noting the citation and table-reference issues. The citation formatting will be corrected, and the latency reference in Appendix F will point to the correct table.
We have updated the manuscript to incorporate the changes and additional experiments discussed in our response, and have uploaded the revised version. We encourage the reviewer to consult the updated paper, where the new results, analyses, and clarifications are integrated directly into the main text and appendix.
Thank you for your detailed response as well as the updates and clarifications to the manuscript. Nevertheless, from the additional experimental results and the revised descriptions in the paper, I still cannot identify a clear advantage or distinctive difference of the proposed method over NeuralDB. Therefore, I will keep my original score.
Response to Reviewer Fxh8
Thank you for the detailed review. We agree that NeuralDB is the closest prior external-memory method and should have been discussed and compared directly. This was a useful omission to identify, and we ran the requested comparison and ablations.
The review also prompted us to revisit a few implementation choices within MoKE. In particular, we use the standard locate-then-edit target for the NeuralDB comparison, test residual addition as well as replacement, and add a second cosine check using the FFN representation. These are small changes within the same external-memory framework, and we evaluate each of them separately below.
Direct comparison with NeuralDB
We first checked our NeuralDB implementation in its published setting using 2,000 facts and its published threshold of 0.65. The reported metrics are ES, PS and NS.
We then ran NeuralDB under the same 3,000-edit protocol used for MoKE. Efficacy measures success on the edited prompt, generalization measures success on paraphrases, and specificity measures whether neighborhood prompts retain their original answer. Mean is the average of the three.
Using NeuralDB exactly as published, MoKE is ahead by 1.06 [0.75, 1.37], 1.04 [0.73, 1.36], 2.78 [2.42, 3.14] and 0.67 [0.46, 0.88] mean points respectively.
We also selected NeuralDB's threshold using the same 600 held-out edits used to choose MoKE's settings.
After selecting NeuralDB's threshold in the same way, MoKE remains ahead on three settings. On GPT2-XL ZsRE the difference is -0.05 points with a 95% interval of [-0.26, 0.16], so the two methods are statistically tied.
What accounts for the difference from NeuralDB
The reviewer specifically asked about the retrieval representation, replacement versus residual addition, and context-key construction. We therefore varied each choice while keeping the others fixed. The values below are changes in mean score on held-out CounterFact edits.
The residual-stream representation alone is not better than NeuralDB's FFN representation. The gain appears when the residual key is used for retrieval and the same candidate is checked in the FFN representation before being applied. Residual addition also performs better than replacement on both models, and context-derived keys consistently improve over using only the edit prompt.
We thank the reviewer for suggesting these ablations, since they make it much clearer which design choices actually matter.
The stored target also matters. The paper used DeltaEdit's target so that the controlled comparison changed only where the same edit was stored. For the NeuralDB comparison, we use the standard locate-then-edit target used by NeuralDB. This removes the DeltaEdit-specific target optimization from the comparison and lets us isolate the retrieval and injection choices more directly.