acceptodds
Withdrawn from ICLR 2027

HebbianEdit: A Closed-Form Hebbian Memory for Knowledge Editing in LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Knowledge editing aims to update specific facts in a deployed language model without retraining. Existing methods rewrite the model's internal weights, train an extra module for every edit, or keep new facts as text outside the model. However, these previous works suggest that knowledge is represented in the form of parametric memory, giving up at least one of precision, efficiency, or revocability. In this work, we rethink that knowledge editing should be interpreted as memory editing, and propose **HebbianEdit** which innovates knowledge editing as closed-form memory injection. Specifically, we write new facts as key-direction pairs into an external Hebbian memory. Then we hold the new knowledge, whose projections and readout are solved in closed form with a squared key similarity that suppresses cross-talk, implemented as a two-layer MLP. Finally, the frozen model reads the memory through hidden-state perturbation at inference time. Updating the deployed model's knowledge is thus achieved by operations on memory rather than surgery on each fact. Comprehensive experiments show that HebbianEdit scales up to ten thousand sequential edits with superior efficacy and generalization at 100% with nearly zero locality flips. More importantly, the proposed memory is built in a single closed-form pass over all edits in seconds compared to the previous methods that use 8.61-33.70 seconds per edit.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Knowledge Editing, Hebbians Memory, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3qrnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HebbianEdit, which implements knowledge editing through FactMLP, an associative memory external to a frozen model. The method constructs fact keys from shallow hidden states and values from target-token unembeddings. A similarity gate identifies the relevant edit, after which a bilinear feature map and a least-squares readout reconstruct its value for hidden-state injection after the final normalization layer. The authors evaluate up to 10,000 edits on CounterFact and zsRE across three models, reporting high editing efficacy and lower construction costs than the compared methods. The paper also analyzes bilinear feature capacity, sharded construction, and projection refinement.

Critical strengths
  1. The method provides an external editing mechanism that requires no backpropagation updates and achieves strong large-scale results. In Table 1, CounterFact efficacy at 10,000 edits is at least 98.2% across all three models, while zsRE efficacy is at least 96.5%. Table 6 compares HebbianEdit with GRACE on the same hardware and shows a reduction in construction time from hours to minutes. Keeping the base model frozen while supplying edits through an external module has practical value, and these results are the main evidence supporting acceptance.

  2. The authors provide substantial comparisons examining the role of the nonlinear readout. Appendix D compares linear and bilinear memory recovery at equal width and reports end-to-end results. On CounterFact, the bilinear readout improves efficacy by approximately 3.3–4.5 percentage points. The comparison between sharded and unsharded construction also demonstrates the role of sharding in the production configuration. These experiments support specific design choices rather than only reporting final performance.

Critical weaknesses
  1. The contribution of FactMLP relative to directly retrieving stored values has not been established. Section 3.3 states that the gate first selects an entry and then passes that entry’s stored key, rather than the query key, to FactMLP. Query generalization is therefore primarily handled by the gate, while FactMLP reconstructs a known value from a known entry. A baseline using the same gate, directly returning the corresponding stored value, and applying the same hidden-state injection would test the necessity of the core design, but this comparison is missing. The linear-versus-bilinear readout comparison does not address this question. Without this baseline, it is difficult to assess what the feature construction and projection solves provide over simple key–value retrieval. This directly affects my assessment of the method’s novelty and practical value.

  2. The experiments do not adequately establish sequential editing and per-entry update capabilities. The paper repeatedly emphasizes 10,000 sequential edits, but the construction procedure solves over the full edit set in shards of 5,000 entries. Each shard’s projections and readout depend jointly on its facts, and value centering also depends on the edit set. The paper does not fully explain how arriving facts are inserted, how an existing fact is revised, or whether these operations require re-solving a shard. Table 6 compares one-time construction with GRACE’s per-edit online updates without matching the update schedule. If each insertion requires rebuilding the memory, the results establish efficient batch construction rather than the claimed low-cost continual maintenance. This is a key gap in the practical claims.

Questions for authors
  1. With exactly the same keys, gate, and injection strength, how does directly returning the matched entry’s stored value compare in editing performance, memory usage, and inference cost? What advantages does FactMLP provide over this baseline?

  2. Are the 10,000 edits applied sequentially, with existing edits evaluated after each insertion, or is the complete memory constructed before evaluation? Does adding a fact to a partially filled shard require re-solving that shard? Please report cumulative construction costs under a matched online update schedule.

  3. How are individual facts revised and revoked? Does revocation only remove the key from the gate, or does it also update the projections, readout, and centering terms? Do these operations affect other entries’ outputs, and has retention of the remaining edits been evaluated?

  4. Appendix C acknowledges that edits sharing the same subject and the full prefix up to the end of its mention have identical keys. How does the method handle queries in which the relation follows the subject, such as “France’s president” and “France’s capital”? How common are these cases in the reported evaluations?

  5. The gate thresholds in Table 2 vary across models, datasets, and edit counts. How are these thresholds selected, and is an independent validation set used? How do generality and specificity change as the memory grows when the threshold is held fixed?

  6. I could not find the AI use statement required by ICLR 2027. Please include this statement and specify whether generative AI tools were used and, if so, for which tasks.

Reviewer_KyM3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HebbianEdit, which stores edited facts in an external memory called FactMLP rather than rewriting the weights of the language model. The key of each edit is the normalized mean hidden state of the first four layers at the last subject token. The value is the centered unembedding vector of the target token. FactMLP uses a bilinear feature that approximates a squared similarity kernel, and all of its weights are solved in closed form. At inference time, the key of every input token is compared against all stored keys. If the best match exceeds a threshold, the stored key of the winning entry is passed through FactMLP, and the output is added to the final hidden state with a strength of 256. Experiments on CounterFact and zsRE with three backbones report high efficacy for up to 10K sequential edits, with construction times on the order of minutes. However, the readout is only ever evaluated at stored keys, so the results are largely determined by an untrained retrieval gate. In addition, several metrics are left undefined, and the closest memory-based editors are missing from the main comparison.

Critical strengths
  • The method is training-free and efficient. The memory for 10K edits is built in 1.8 to 14.4 minutes, compared with several hours for GRACE. The inference overhead is only 4.6% to 7.8%. When the gate stays closed, the forward pass is identical to that of the frozen model.
  • Using shallow hidden states at the subject token as keys is a useful empirical finding. The ablation clearly shows that deeper layers hurt generalization, which drops to 0.415.
  • The theoretical analysis is carefully checked against measurements. The kernel behavior, capacity frontier, sharding, and refinement are all measured and reported honestly, including cases where the measurements deviate from the theory.
Critical weaknesses
  • In the actual system, FactMLP is equivalent to a lookup table. Section 3.3 states that the readout receives the stored key of the winning entry, not the query key. The readout therefore only has to return the i-th value for the i-th stored key. A table indexed by the gate would do this exactly, using no more memory than the three FactMLP matrices. As a result, the squared kernel, the capacity bound, and the bilinear-versus-linear comparison in Appendix D all concern how well a network approximates a table that the method could simply store. This also contradicts the claim in Section 5 that the value is "computed rather than fetched."

  • The retrieval gate drives the results but is never optimized. It is a nearest-neighbor search over fixed shallow keys with a single threshold. Because the injection strength is 256, efficacy and generalization essentially measure the recall of the gate.

    • The threshold is tuned separately for each backbone, dataset, and stream length, ranging from 0.75 to 0.975 (Appendix Table 2).
    • The paper does not explain how these values were selected. This conflicts with the claim that a single hyperparameter set is frozen across all experiments.
    • If the thresholds were chosen on evaluation data, this would also amount to test-set leakage.
  • The gate keys on the subject and ignores the relation.

    • Because the key mainly encodes entity identity, a question about the same subject but with a different relation is likely to trigger the edit and receive the wrong answer at full strength.
    • Multiple edits on the same subject share a single key, and Appendix C acknowledges that they then collapse to their mean.
    • The design also requires the subject span at edit time and the exact subject name at query time, so aliases, pronouns, and unstructured edits are not covered.

    None of these cases are evaluated.

  • Specificity degrades as more edits are stored. On CounterFact with Qwen3-4B, it falls from 13.2 at 300 edits to 11.5 at 5K and 10.5 at 10K, compared with roughly 14 to 15 for the unedited model. The other backbones show the same trend. This is consistent with false triggers on look-alike neighborhood prompts. Such triggers become more likely as the gate scans every input token against an ever-growing key set. In contrast, GRACE stays within 0.4 points of the unedited model on the same metric (Appendix G). The trade-off between generalization and locality is controlled solely by the threshold.

  • The evaluation metrics are not defined.

    • Specificity is described as whether unrelated facts remain unchanged. Under that definition the unedited model should score 100, yet the table reports 14.0 to 22.3 on CounterFact.
    • The unedited model's efficacy is 0.3, which is also inconsistent with the standard probability-comparison metric.

    Without exact formulas, it is difficult to interpret these numbers or compare them with prior work.

  • The baselines are weak and incomplete. The baseline rows are copied verbatim from the BetaEdit paper rather than rerun, and many of them are 0.00 on every metric, including specificity. More importantly, the closest memory-based editors are left out of the main comparison:

    • GRACE [1] appears only in an appendix.
    • HoReN [3] is described as concurrent work but is not compared against.
    • WISE [4] is absent altogether.
  • The novelty is limited relative to the closest prior work.

    • GRACE [1] already keeps the backbone frozen and stores each edit as an entry in an external key-value codebook. It retrieves entries by thresholded similarity and revokes an edit by deleting its entry.
    • Larimar [2] already writes new facts into an external memory in closed form via least squares, without per-edit training, and supports selective forgetting.

    Since the readout is only evaluated at stored keys, HebbianEdit effectively reduces to a GRACE-style codebook with shallow subject keys and logit-direction values.

  • Some results are inconsistent.

    • Figure 3b shows that random projections at m = 2048 already reach 0.948 accuracy, and Appendix D shows a random linear readout recovering 0.96 of the keys. Yet Table 7 reports 0.000 efficacy for random features.
    • Figure 3b also shows alignment falling from 0.944 to 0.60 as refinement rounds increase. This suggests that the refinement objective is misaligned with the evaluation metric.
  • The evaluation scope is narrow. The paper does not measure fluency, general capability under false triggers, multi-hop or portability questions, or long-context behavior. Revoking individual edits is argued for but never tested. The claim that recall is transient is also unconvincing. The gate scans the entire input, and the subject remains in the prompt throughout generation.

Questions for authors
  1. If FactMLP is replaced by a table that returns the stored value of the entry selected by the gate, do the results change? If not, what role do the squared kernel and the capacity analysis play in the final system?
  2. How was the gate threshold selected for each cell in Appendix Table 2? How do the results change if a single threshold is used across all settings?
  3. What are the exact formulas for efficacy, generalization, and specificity? Could you also report locality as agreement with the unedited model's outputs?
  4. What is the specificity on questions that share the subject but differ in relation? How does the method handle multiple edits on the same subject?
  5. Could you rerun GRACE [1], WISE [4], and HoReN [3] under the same setup at 10K edits?
  6. How can the 0.000 results for random features in Table 7 be reconciled with Figure 3b and Appendix D?
  7. During generation, at which positions and decoding steps does the gate fire? This matters because the subject remains in the prompt and multi-token answers use a separate readout at each position.
  8. Could you provide an experiment that deletes individual entries? Does deletion require re-solving the readout for the affected shard?

References

[1] Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors, NeurIPS 2023

[2] Larimar: Large Language Models with Episodic Memory Control, ICML 2024

[3] HoReN: Normalized Hopfield Retrieval for Large-Scale Sequential Model Editing, arXiv 2026

[4] WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models, NeurIPS 2024

Reviewer_ApELOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes HebbianEdit, which constructs a key for each edit from shallow-layer hidden states and a value from the unembedding vectors of the target answer tokens. At inference time, an activation gate selects the best-matching entry from an external memory. The selected entry’s stored key is then passed through a quadratic layer and a readout layer to produce a hidden-state perturbation, which is added to the model’s final hidden state. The authors evaluate up to 10,000 edits on CounterFact and zsRE using three backbones. Table 1 reports efficacy of at least 96.5% across all settings, together with strong generalization to paraphrased queries in many large-scale settings.

Critical strengths
  1. Large-scale sequential editing, preserving the base model’s weights, and supporting revocability are all important requirements for practical knowledge-editing systems.
  2. The paper covers two datasets, three backbones, and multiple editing scales, and provides analyses of feature width, alternating refinement, and sharding.
Critical weaknesses
  1. The central FactMLP readout module lacks the most natural control experiment. In Section 3.3, the activation gate first scores a token’s key against all stored keys and selects the highest-scoring entry. The winning entry’s own stored key is then passed to the readout network. Thus, the critical decision of which fact to use has already been made by nearest-neighbor selection. A natural baseline is therefore to use exactly the same keys, gate threshold, and hidden-state injection mechanism, but directly return the target value stored in the selected entry. If this simple direct-lookup baseline performs comparably to HebbianEdit, the paper’s claims regarding the necessity of FactMLP, the squared key similarity, and the shared readout would need to be substantially weakened.

  2. Generalization degrades substantially at scale. In Table 1, the CounterFact generality scores at 10,000 edits are only 58.6%, 45.6%, and 80.1% for the three backbones. For two of the models, the method therefore still fails to produce the target answer for a large fraction of paraphrased queries. Meanwhile, CounterFact specificity decreases from the pre-edited values of 14.0, 14.4, and 21.8 to 10.5, 10.9, and 18.5, respectively. These results suggest that the method performs very well on the original query templates but suffers substantial scale-dependent degradation on paraphrased queries. For knowledge editing, generality is as important as efficacy. I recommend that the authors report the activation gate’s false-positive and false-negative rates across different query types and describe the claims that “existing knowledge remains intact” and that the method enables “precise editing” more cautiously.

  3. There remains a substantial gap between the theoretical results and the actual production system. The theory in Appendix C applies to Gaussian projections at initialization, whereas the production system uses projections after data-dependent refinement. Moreover, the geometric analysis defines ρ∗\rho^* as the largest correlation among non-duplicate pairs, whereas the experiments estimate it using the 99.9th percentile of pairwise correlations; these two quantities are not equivalent. The experimental margin therefore approximates, rather than exactly captures, the worst-case key geometry assumed by the theory. In addition, Figure 3(b) shows that further alternating refinement leads to substantial degradation. Consequently, the claim in Section 4.5 that “alternating refinement converges in one round” is inaccurate: the evidence supports empirical early stopping after one round, rather than algorithmic convergence.

Questions for authors
  1. Using exactly the same keys and activation gate, what are the efficacy, latency, and storage overhead of directly looking up and returning the selected entry’s target value?
  2. Does the “sequential editing” protocol in Table 1 update the memory after every edit arrives and re-evaluate all previously edited facts? If so, which matrices are updated upon each new edit?
  3. If the margin is recomputed using the true largest correlation among non-duplicate pairs rather than its 99.9th-percentile approximation, do the conclusions in Figure 3(c) still hold? Given the substantial degradation caused by further alternating refinement in Figure 3(b), what justifies describing the phenomenon in Section 4.5 as “convergence in one round” rather than empirical early stopping after one round?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.