acceptodds
Under review as a conference paper at ICLR 2027

EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory

Abstract

Conditional memory architectures such as DeepSeek Engram use input -grams to look up learned embeddings, expanding the capacity of large language models (LLMs) with limited additional computation. Beyond model scaling, this architecture has demonstrated the potential to decouple factual knowledge storage from general-purpose computation, offering a promising route to updating factual knowledge while keeping the Transformer backbone fixed. However, updating shared -gram embeddings can affect other edits and unrelated knowledge, while different expressions of the same fact may activate embeddings that were not updated. We propose **EngramEdit** for decoupled knowledge updates through conditional memory. EngramEdit first computes target memory representations that make the model predict the updated fact across multiple expressions. It then jointly updates the shared -gram embeddings to match these targets across expressions and edits, penalizing updates to frequently reused embeddings more strongly to preserve unrelated knowledge. Experiments show that EngramEdit enables independent factual knowledge updates through conditional memory, achieving near-perfect editing success. Revised knowledge is usable across unseen expressions and in multi-hop reasoning, with nearly the strongest baseline's accuracy under chain-of-thought (CoT) prompting. Unrelated knowledge and general capabilities are largely preserved even as factual updates accumulate. These findings show that EngramEdit turns conditional memory into an editable knowledge interface, extending its role beyond model scaling to support decoupled knowledge updates. The code is available at [https://anonymous.4open.science/r/EngramEdit](https://anonymous.4open.science/r/EngramEdit).

Primary area:
foundation or frontier models, including LLMs
Keywords:
Knowledge Editing, Conditional Memory, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2vm1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper extends Rome/Memit-style fact editing methods to the recent Engram architecture from Deepseek (and similar related architectures/models). The Engram architecture augments LLMs with sparse parameters in the form of n-gram embedding lookups (with hashing). This work follows a similar approach to prior fact editing methods and aims to learn some minimal parameter update that will “update” the model to produce the target edit with some additional regularization. Unlike earlier methods which often target FFN networks, this network applies the update to the sparse n-gram embeddings in Engram models. All experiments in this work use the LongCat-Flash-Lite model.

The learning procedure largely adapts the Rome/Memit recipe. It first learns some shared representation via gradient descent that minimizes the edit loss (and regularizers) across some paraphrases of the target edit. The authors then introduce a closed form mapping to map this optimized residual back to an embedding table update. The update is spread across the relevant n-gram embeddings. In practice, the updates are tied to specific n-grams instead of being applied to the hash embeddings themselves.

On the multihop MQuAKE eval, all methods have limited success under normal generation while EngramEdit has some success in a CoT setting. For evaluating edit success, this work adopts the standard CounterFact and ZsRE datasets. They show their method is generally more effective than various baselines across these datasets One exception here is worse specificity on ZsRE compared to all external baselines, indicating some amount of leakage. Their general capabilities evaluation shows limited capability loss.

Critical strengths

This work is a reasonable extension of the memory editing line of work to a new model architecture. The Engram architecture may be inherently more amenable to non-interfering parameter updates. The n-gram embedding updates, by construction, do not interfere with other sequences when the relevant n-gram embeddings are not activated.

The evaluation protocol reasonably follows the standards in this area, using the standard editing benchmarks/metrics along with 6 capabilities benchmarks and the multihop MQuake eval.

Critical weaknesses

The impact of this work is quite narrow and incremental given that it is primarily adapting existing techniques to a new sparse memory architecture that is not universally adopted in the field. It can only apply to Engram-style models.

The decision to update sparse n-gram embeddings both has the potential to reduce interference, but also limits generalizability. Training essentially has to cover all relevant n-grams. During inference, if none of the training n-grams are included in the prefix, it is impossible for the learned edits to generalize.

A number of limitations with these kinds of techniques have been raised by various recent work and this work seems to inherit those limitations.

Some of these limitations are:

  • The CounterFact metrics are pairwise (i.e. p(o*) > p(o)) which is much less strict than requiring producing the correct edit.
  • Prior work (Hoelscher-Obermaier et al., 2023) has found that the neighborhood prompts underestimated leakage and introduce a stricter Counterfact+ benchmark
  • Teacher forcing has been shown to inflate performance (Yang et al. 2025) and introduce a stricter free-form generation setup that would be more informative.
  • Prior work has shown with the MQuake benchmark, that these edits often fail to generalize to multi-hop settings. This is largely the case in this work as well in the standard setting. CoT improves this, likely by verbalizing the relevant n-grams to activate the information. This is a nice finding but also highlights the sensitivity of the method to generating the “correct” n-grams that were used during training.

The capabilities evaluation uses small sample sizes (100) from six datasets. Much of the reported variation looks like it could just be noise. For example, UnKE appears to decline, then improve over the base model, before finishing around the same level as the base model. Given the trend, it raises the question on whether these fluctuations are largely noise.

The work does not actually update the hash embeddings but instead stores specific n-gram vectors. This actually introduces parameters and thus increases overhead compared to true parameter editing methods. Applying it to the true embedding table makes it more scalable, but also reduces the interference benefits.

Questions for authors

What does performance look like on the Counterfact+ benchmark from Hoelscher-Obermaier et al., 2023 and using the WILD evaluation protocol from Yang et al., 2025 (“The Mirage of Model Editing…”)?

With a stricter top-1 definition of edit success, how does the method perform on CounterFact?

Does this work address any of the fact editing limitations outlined in recent work and mentioned above?

Can you expand the sample sizes of the capabilities evaluations?

Can you apply the updates to the actual hashed embedding values? How does this change results?

Reviewer_XyANOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates factual knowledge editing through conditional memory without modifying the Transformer backbone. The results on editing efficacy and generalization are impressive, but I have several concerns.

  1. A major motivation of EngramEdit is to address interference between edits that share n-gram embeddings. However, Appendix Figure 12 shows that only 8.66% of the edited facts involve cross-edit sharing, yet they account for 48 of the 49 editing failures. Efficacy drops from 99.98% for non-sharing facts to 88.91% for sharing facts. This is precisely the setting in which the proposed joint optimization is expected to be most useful. I would like to see a more focused evaluation under varying degrees of n-gram overlap, particularly when different edits require conflicting changes to shared embeddings.

  2. The generalization results deserve closer examination. EngramEdit generates multiple expressions to cover more n-grams, and Appendix Figure 11 shows that 98.05% of held-out paraphrases activate fact-related updated embeddings. Performance is considerably worse when this overlap is absent. How much of the reported improvement comes from broader lexical coverage rather than generalization across genuinely different expressions? Testing paraphrases with little or no n-gram overlap, including alternative entity references and substantially different sentence structures, would be informative.

  3. The technical contribution relative to existing knowledge editing methods is not entirely clear. Computing target representations and solving for regularized parameter updates largely follows the locate-then-edit paradigm. More importantly, Appendix B.4.3 indicates that EngramEdit stores separate update vectors indexed by exact n-grams instead of modifying the pretrained hashed memory tables. This implementation appears closer to maintaining an auxiliary set of memory updates than directly editing the original memory parameters. The distinction from existing memory-based editing approaches deserves a more careful discussion and experimental comparison.

  4. The claims concerning knowledge preservation and multi-hop reasoning seem somewhat overstated. Keeping the backbone frozen does not necessarily preserve its original behavior. In Figure 4, MRPC performance decreases by 9.2 F1 points after 5,000 edits. Similarly, although EngramEdit substantially outperforms the baselines on MQuAKE, its overall CoT accuracy is only 25.2%, dropping to 15.1% for four-hop questions. The relative gains are encouraging, but these absolute results provide limited evidence of reliable multi-hop reasoning with revised knowledge.

  5. The experimental evaluation relies entirely on LongCat-Flash-Lite. Since conditional-memory architectures can differ considerably in their memory lookup and fusion mechanisms, it is difficult to judge how broadly the method applies. The computational cost also deserves attention: each edit involves expression generation, target optimization, and a joint linear solve, but editing time and additional memory requirements are not adequately compared with the baselines. Finally, the sequential editing experiments mainly accumulate different facts, whereas repeated updates or contradictory revisions to the same fact would provide a more demanding test of continual knowledge editing.

Critical strengths

None

Critical weaknesses

None

Questions for authors

None

Reviewer_6PSCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies factual updates through n-gram conditional memory while keeping the Transformer backbone fixed. EngramEdit first optimizes a shared representation perturbation across expressions of each fact, then jointly allocates updates to the underlying n-grams, with stronger penalties on short and frequent sequences. Experiments show improvements on paraphrases and multi-hop questions with CoT. The implementation stores cumulative increments indexed by exact n-grams.

Critical strengths
  1. The improvement persists with the same expression augmentation. In Table 6, EngramEdit and MFT-S achieve CounterFact generalization scores of 97.0% and 90.7% using the same four generated expressions, with specificity of 85.2% and 85.0%.
  2. There is evidence that the model uses the updates. In C.3.1, disabling fact-related increments turns 5,228 of 5,850 previously successful prompts into failures, whereas disabling random increments matched in number and length causes none of these failures.
Critical weaknesses
  1. In C.3.2, 98.05% of test paraphrases activate updates for the current fact, while the remaining 1.95% account for 50.36% of failures. The evidence therefore primarily concerns expressions that trigger relevant updates. Please report results by subject token length and add a small, independently constructed set that changes the wording immediately before the subject. This would help establish how much coverage comes from unchanged n-grams within the subject.
  2. Table 6 provides this control on CounterFact and ZsRE, but Fig. 3 has no corresponding MQuAKE comparison. Additional expressions may improve the coverage of updated n-grams during CoT, so the 25.2% versus 8.5% result does not separate augmentation from the update algorithm.
Questions for authors
  1. B.2 evaluates CounterFact by relative answer likelihood and ZsRE by token accuracy with reference prefixes. Could you report answer accuracy under free generation for EngramEdit and MFT-S on the same paraphrases, to check whether the high scores translate into directly generated revised answers?
  2. In B.3, several backbone editors use layer 7. How were this layer, learning rates, and constraint strengths selected? Please specify the validation data and search budget, particularly for MoEEdit and MFT-S.
  3. What is the dimension of each increment in B.4.3, and is it added before or after projection and averaging? Are increments constrained to the original projection space? This determines the method’s actual editable parameter space.
  4. Which MQuAKE data version was used? Were results evaluated after all batches or aggregated from evaluations after individual batches? Please also give the total number of atomic edits and the policy for duplicate or conflicting requests.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.