- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper argues that LLMs handle specialized terminology poorly because 1) domain terms are split into many subtokens and 2) domain vocabularies are large and evolving. It proposes Lexical Conditional Memory (LCM), a retrofit module that keeps the native tokenizer and backbone architecture. LCM uses an Aho–Corasick matching to match complete domain terms over native token IDs and maps each match to one entry in an external lexical memory built from curated resources. The retrieved term vectors are injected into the hidden states at an early and a late layer through gated residual connections with independent parameters. Training follows a staged curriculum that aligns the memory, forces its use, and then calibrates the gates before downstream fine-tuning. The method is evaluated on six benchmarks in medicine, law, and chemistry using Qwen2.5-0.5B and 1.5B. LCM outperforms Engram baseline by about 2 points. Also, combining LCM with RAG gives the best overall performance showing that the method is complementary to RAG. Ablations on injection depth, routing design, and memory content, together with latency measurements, support the design choices.
Critical strengths
-
The paper address timely and well-motivated problem. Conditional memory is an active direction for separating stored knowledge from computation. The paper asks a natural, underexplored question, how to adapt such memory to specialized domains, where terms are long, rare, and evolving.
-
The retrofit setting is practical and has benefits over the existing methods that require pretraining with memory from scratch. Also, the native tokenizer is unchanged, so the method can be attached to existing models without vocabulary surgery.
-
The experiment section is well-controlled, with comprehensive evaluation. The baselines are matched to separate the source of the gains. DAPT+SFT controls for extra domain exposure, and Engram variant shares the backbone, injection interface, and training schedule. Gains are consistent across multiple domains and backbone families.
Critical weaknesses
- Although the comparison with Engram is highlighted over multiple sections, the comparison with Engram is limited in scale. Engram’s main results are at 27B, but this paper tests only 0.5B and 1.5B. The margins over SFT already narrow slightly from 0.5B to 1.5B (+3.0 to +2.7).
- Engram+RAG is not reported, so LCM+RAG’s advantage cannot be attributed to LCM specifically.
- The general capability is not evaluated. The capability outside the target domains is not evaluated, so catastrophic forgetting cannot be ruled out.
- The central framing claims are not directly tested. Although the "evolving vocabularies" are one of the limitations that paper mentioned, no experiment adds terms after the training. Also, changing facts can be beyond any lexical representation, and the scope relative to knowledge updating should be stated explicitly.
- The optimization pipeline brittleness. Naive end-to-end retrofitting (48.8) falls below plain SFT (49.5). The method seems to depend on a fine-grained five-phase schedule with many phase-specific choices
Questions for authors
- Can you report results with a trainable Engram memory, and Engram+RAG? Also, since Engram is a sparse-memory method, both total and active counts are needed for a fair comparison.
- I understand that the resource can be limited, but 0.5B and 1.5B scale is too limited for the reliability of the method compared to modern model scale. Are there results at 7B or larger (e.g., Qwen2.5-7B), even for a single domain?
- How does LCM affect performance on general-domain benchmarks relative to the base model and SFT?
- From my understanding, the “Engram” described in the paper is not Engram as originally proposed. Original Engram learns its embedding tables jointly with the model during pretraining. Here it is reduced to a hashed-routing variant of LCM’s own frozen inventory. The comparison therefore tests routing designs within LCM’s framework, not LCM against Engram.
8 comments
Response to Question 2: LCM on Larger and Newer Backbones
Thank you for this important suggestion. We appreciate your concern about whether the gains observed at 0.5B–1.5B remain reliable at a more modern model scale. We therefore extended the evaluation to Qwen3-4B and Qwen3-8B on the same medical-domain setting using MedQA and MedMCQA.
| Backbone | Method | MedQA | MedMCQA |
|---|---|---|---|
| Qwen3-4B | SFT | ||
| Controlled Engram | |||
| LCM | |||
| Qwen3-8B | SFT | ||
| Controlled Engram | |||
| LCM |
On Qwen3-4B, LCM improves over SFT by +3.1 points on MedQA and +2.3 points on MedMCQA. It also improves over Controlled Engram by +2.8 and +1.5 points, respectively.
The same pattern remains on Qwen3-8B. LCM improves over SFT by +3.2 points on MedQA and +2.7 points on MedMCQA, while exceeding Controlled Engram by +2.3 and +1.3 points.
These gains are comparable in magnitude to the original Qwen2.5-1.5B setting, where LCM improves over SFT by +2.7/+2.6 points on MedQA/MedMCQA. The small reduction in the original macro-average gain from 0.5B to 1.5B therefore does not develop into a disappearance of the benefit at larger scale. The absolute performance of the backbone rises substantially, while the additional contribution from LCM remains clearly present at both 4B and 8B.
For controlled adaptation, all methods use the same training regime within each backbone. Qwen3-4B uses full-parameter adaptation for SFT, Controlled Engram, and LCM. Qwen3-8B uses the same matched LoRA configuration across all three methods. We therefore interpret these experiments through within-backbone comparisons.
These results show that the usefulness of explicit specialized lexical memory persists on substantially larger and newer backbones. Stronger models provide more capable shared reasoning representations, while on-demand access to external complete-term lexical representations continues to offer additional support for knowledge-intensive tasks.
Response to Question 3: General-Domain Capability Retention
Thank you for raising this important question. We agree that target-domain improvements should be considered together with the model's capability outside the adaptation domain. We therefore evaluated the native pretrained model, matched SFT, and LCM on a suite of general-domain benchmarks at both Qwen3-4B and Qwen2.5-1.5B scales.
In addition to the standard MMLU and AGIEval aggregates, we report two subsets that reduce direct overlap with the medical adaptation domain. MMLU-General excludes the 11 medical subjects from MMLU. AGIEval-OOD further removes the biology task from the AGIEval aggregate used in our evaluation. These subsets help distinguish general-capability retention from improvements that may arise from direct overlap with medical knowledge.
Qwen3-4B
| Benchmark | Native | SFT | LCM |
|---|---|---|---|
| MMLU | 66.78 | 68.02 | 67.99 |
| MMLU-General | 68.93 | 70.40 | 70.57 |
| BBH | 51.42 | 51.00 | 51.83 |
| GSM8K | 79.98 | 80.36 | 81.50 |
| AGIEval | 24.14 | 26.93 | 27.64 |
| AGIEval-OOD | 24.31 | 27.35 | 28.12 |
On Qwen3-4B, LCM is at least as strong as matched SFT on five of the six evaluations. The only exception is MMLU, where the difference is −0.03 points. On the two subsets with direct medical overlap reduced, LCM improves over SFT by +0.17 points on MMLU-General and +0.77 points on AGIEval-OOD. We therefore do not observe a systematic general-capability penalty from adding the lexical-memory pathway.
We observe the same pattern on the original Qwen2.5-1.5B scale.
Qwen2.5-1.5B
| Benchmark | Native | SFT | LCM |
|---|---|---|---|
| MMLU | 59.59 | 60.89 | 61.12 |
| MMLU-General | 60.80 | 62.21 | 62.38 |
| BBH | 42.76 | 41.92 | 42.86 |
| GSM8K | 56.41 | 54.81 | 56.10 |
| AGIEval | 37.27 | 27.32 | 28.22 |
| AGIEval-OOD | 36.58 | 27.27 | 28.21 |
This second scale also illustrates why both the native model and matched SFT are useful controls. Domain adaptation itself changes some general-domain capabilities. For example, SFT decreases from 37.27 to 27.32 on AGIEval and from 36.58 to 27.27 on AGIEval-OOD relative to the native model. LCM does not deepen these reductions. It improves over matched SFT on all six evaluations, with gains ranging from +0.17 to +1.29 points. The same conclusion holds on the medical-overlap-reduced subsets, where LCM improves by +0.17 on MMLU-General and +0.94 on AGIEval-OOD.
Taken together, the results separate the effect of domain adaptation from the effect of introducing LCM. General-capability changes relative to the native model depend on the backbone and benchmark, while LCM does not introduce a systematic additional degradation beyond matched SFT across the evaluated suite. This is consistent with the central design of LCM: specialized lexical support is externalized into an on-demand pathway while the pretrained backbone continues to provide the shared representation and reasoning capacity.
Response to Question 1: Trainable Engram, Engram+RAG, and Memory Activity
Thank you very much for taking the time to carefully review our work and for the thoughtful and constructive feedback. We greatly appreciate your positive assessment of the problem motivation, the practical retrofit setting, and the controlled empirical evaluation. Your comments were very encouraging to us. They also highlighted several important directions for strengthening the evaluation, and we have taken these suggestions seriously in the additional experiments below.
We first distinguish two complementary Engram comparisons. On Qwen, Controlled Engram uses the same backbone, retrofit interface, training schedule, and source lexical inventory as LCM, providing a controlled comparison of the memory and routing design. We additionally evaluate Native Engram on Argon, where Engram is an original trainable component of the model architecture. This second setting directly tests LCM against a native trainable Engram memory.
To address the missing Engram+RAG comparison in the original results, we first add RAG to Controlled Engram under exactly the same Qwen2.5-1.5B setting:
Giving Controlled Engram the same retrieval support improves it over SFT + RAG by +0.9/+0.2 points. LCM + RAG remains higher by +1.7 points on MedQA and +0.8 points on MedMCQA under the same retrieval setting.
We also constructed a stronger RAG baseline to test this complementarity under a more competitive retrieval pipeline. The original RAG uses BGE-M3 dense retrieval. Strong RAG combines BM25 and BGE-M3 retrieval, fuses their candidates with reciprocal rank fusion, and applies Qwen3-Reranker-0.6B before selecting the same top-3 evidence under the same 512-token context budget. The retrieval corpus and base model remain fixed. On Qwen2.5-1.5B, this strengthens SFT + RAG from 44.0/49.0 to 45.9/49.6, while LCM + Strong RAG reaches 52.3/51.8.
More importantly for the trainable-Engram comparison, we apply the same Strong-RAG pipeline to Native Engram and LCM on Argon:
LCM exceeds Native Engram by +2.8/+0.8 points without RAG. Under the matched Strong-RAG pipeline, the corresponding advantage is +3.4/+1.9 points. Thus, the additional lexical-memory benefit remains present when a native trainable Engram receives the same strengthened retrieval support.
Following your suggestion, we also audited both total memory capacity and actual memory activity. For the Qwen2.5-1.5B comparison, all 1,273 MedQA and 4,183 MedMCQA prompts and token IDs are exactly matched between Controlled Engram and LCM.
Controlled Engram has 279,634 occupied buckets among its 295,872 allocated buckets. Native Engram on Argon contains 640,000 slots and performs 16 sparse lookups per input token. The access units are structurally different across the methods. Controlled Engram counts occupied hash-bucket accesses, Native Engram counts native sparse memory-slot lookups, and LCM counts final routed lexical entries. We therefore report their native total and active counts directly without converting them into a cross-method sparsity multiplier.
These results highlight the different memory-access patterns. Engram performs sparse lookup over hashed or native memory slots, while LCM activates its external memory at routed complete-term lexical endpoints. Across both the controlled Qwen setting and the native trainable-Engram setting, LCM retains its gains when retrieval support is provided to both methods. This is consistent with the central design of LCM, where specialized lexical representations are externalized and accessed conditionally as support for the shared reasoning backbone.