- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper extends Rome/Memit-style fact editing methods to the recent Engram architecture from Deepseek (and similar related architectures/models). The Engram architecture augments LLMs with sparse parameters in the form of n-gram embedding lookups (with hashing). This work follows a similar approach to prior fact editing methods and aims to learn some minimal parameter update that will “update” the model to produce the target edit with some additional regularization. Unlike earlier methods which often target FFN networks, this network applies the update to the sparse n-gram embeddings in Engram models. All experiments in this work use the LongCat-Flash-Lite model.
The learning procedure largely adapts the Rome/Memit recipe. It first learns some shared representation via gradient descent that minimizes the edit loss (and regularizers) across some paraphrases of the target edit. The authors then introduce a closed form mapping to map this optimized residual back to an embedding table update. The update is spread across the relevant n-gram embeddings. In practice, the updates are tied to specific n-grams instead of being applied to the hash embeddings themselves.
On the multihop MQuAKE eval, all methods have limited success under normal generation while EngramEdit has some success in a CoT setting. For evaluating edit success, this work adopts the standard CounterFact and ZsRE datasets. They show their method is generally more effective than various baselines across these datasets One exception here is worse specificity on ZsRE compared to all external baselines, indicating some amount of leakage. Their general capabilities evaluation shows limited capability loss.
Critical strengths
This work is a reasonable extension of the memory editing line of work to a new model architecture. The Engram architecture may be inherently more amenable to non-interfering parameter updates. The n-gram embedding updates, by construction, do not interfere with other sequences when the relevant n-gram embeddings are not activated.
The evaluation protocol reasonably follows the standards in this area, using the standard editing benchmarks/metrics along with 6 capabilities benchmarks and the multihop MQuake eval.
Critical weaknesses
The impact of this work is quite narrow and incremental given that it is primarily adapting existing techniques to a new sparse memory architecture that is not universally adopted in the field. It can only apply to Engram-style models.
The decision to update sparse n-gram embeddings both has the potential to reduce interference, but also limits generalizability. Training essentially has to cover all relevant n-grams. During inference, if none of the training n-grams are included in the prefix, it is impossible for the learned edits to generalize.
A number of limitations with these kinds of techniques have been raised by various recent work and this work seems to inherit those limitations.
Some of these limitations are:
- The CounterFact metrics are pairwise (i.e. p(o*) > p(o)) which is much less strict than requiring producing the correct edit.
- Prior work (Hoelscher-Obermaier et al., 2023) has found that the neighborhood prompts underestimated leakage and introduce a stricter Counterfact+ benchmark
- Teacher forcing has been shown to inflate performance (Yang et al. 2025) and introduce a stricter free-form generation setup that would be more informative.
- Prior work has shown with the MQuake benchmark, that these edits often fail to generalize to multi-hop settings. This is largely the case in this work as well in the standard setting. CoT improves this, likely by verbalizing the relevant n-grams to activate the information. This is a nice finding but also highlights the sensitivity of the method to generating the “correct” n-grams that were used during training.
The capabilities evaluation uses small sample sizes (100) from six datasets. Much of the reported variation looks like it could just be noise. For example, UnKE appears to decline, then improve over the base model, before finishing around the same level as the base model. Given the trend, it raises the question on whether these fluctuations are largely noise.
The work does not actually update the hash embeddings but instead stores specific n-gram vectors. This actually introduces parameters and thus increases overhead compared to true parameter editing methods. Applying it to the true embedding table makes it more scalable, but also reduces the interference benefits.
Questions for authors
What does performance look like on the Counterfact+ benchmark from Hoelscher-Obermaier et al., 2023 and using the WILD evaluation protocol from Yang et al., 2025 (“The Mirage of Model Editing…”)?
With a stricter top-1 definition of edit success, how does the method perform on CounterFact?
Does this work address any of the fact editing limitations outlined in recent work and mentioned above?
Can you expand the sample sizes of the capabilities evaluations?
Can you apply the updates to the actual hashed embedding values? How does this change results?