acceptodds
Under review as a conference paper at ICLR 2027

Lexical Conditional Memory: Decoupling Specialized Memory from Reasoning for Knowledge-Intensive Tasks

Abstract

LLMs couple specialized lexical memory and general reasoning within a shared parametric backbone, creating two challenges for knowledge-intensive tasks. First, domain-specific terminology is often fragmented into multiple native subtokens, potentially requiring repeated composition across layers before higher-level reasoning. Second, the specialized vocabularies required by knowledge-intensive tasks are often large, long-tailed, and continuously evolving. Encoding and maintaining such lexical knowledge entirely within shared model parameters is therefore inefficient. To address these challenges, we introduce Lexical Conditional Memory (LCM), a retrofit architecture that externalizes specialized lexical memory while preserving the native tokenizer and pretrained Transformer backbone. LCM comprises two complementary mechanisms. Deterministic variable-length lexical routing recognizes complete domain terms across variable-length subtoken spans and maps them directly to external memory entries, providing direct access to complete-term lexical representations. Hierarchical conditional memory injection integrates the retrieved representations through depth-specific interfaces, enabling specialized lexical knowledge to be accessed on demand rather than stored uniformly in the shared parameters. Together, these mechanisms provide an explicit external pathway for specialized lexical representations, reducing reliance on their implicit reconstruction and storage within the shared backbone. Across six benchmarks in medicine, law, and chemistry, LCM outperforms Engram by 2.1–2.2 macro-average points across two controlled Qwen2.5 scales and by 2.4 points in the native-Engram Argon setting. LCM also remains complementary to factual retrieval, with LCM+RAG achieving the highest overall macro-average performance. These results establish external lexical memory as an effective approach for adapting LLMs to knowledge-intensive tasks across architectures and domains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, external memory, conditional memory, lexical memory, domain adaptation, knowledge-intensive tasks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tV9oOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper argues that LLMs handle specialized terminology poorly because 1) domain terms are split into many subtokens and 2) domain vocabularies are large and evolving. It proposes Lexical Conditional Memory (LCM), a retrofit module that keeps the native tokenizer and backbone architecture. LCM uses an Aho–Corasick matching to match complete domain terms over native token IDs and maps each match to one entry in an external lexical memory built from curated resources. The retrieved term vectors are injected into the hidden states at an early and a late layer through gated residual connections with independent parameters. Training follows a staged curriculum that aligns the memory, forces its use, and then calibrates the gates before downstream fine-tuning. The method is evaluated on six benchmarks in medicine, law, and chemistry using Qwen2.5-0.5B and 1.5B. LCM outperforms Engram baseline by about 2 points. Also, combining LCM with RAG gives the best overall performance showing that the method is complementary to RAG. Ablations on injection depth, routing design, and memory content, together with latency measurements, support the design choices.

Critical strengths
  • The paper address timely and well-motivated problem. Conditional memory is an active direction for separating stored knowledge from computation. The paper asks a natural, underexplored question, how to adapt such memory to specialized domains, where terms are long, rare, and evolving.

  • The retrofit setting is practical and has benefits over the existing methods that require pretraining with memory from scratch. Also, the native tokenizer is unchanged, so the method can be attached to existing models without vocabulary surgery.

  • The experiment section is well-controlled, with comprehensive evaluation. The baselines are matched to separate the source of the gains. DAPT+SFT controls for extra domain exposure, and Engram variant shares the backbone, injection interface, and training schedule. Gains are consistent across multiple domains and backbone families.

Critical weaknesses
  • Although the comparison with Engram is highlighted over multiple sections, the comparison with Engram is limited in scale. Engram’s main results are at 27B, but this paper tests only 0.5B and 1.5B. The margins over SFT already narrow slightly from 0.5B to 1.5B (+3.0 to +2.7).
  • Engram+RAG is not reported, so LCM+RAG’s advantage cannot be attributed to LCM specifically.
  • The general capability is not evaluated. The capability outside the target domains is not evaluated, so catastrophic forgetting cannot be ruled out.
  • The central framing claims are not directly tested. Although the "evolving vocabularies" are one of the limitations that paper mentioned, no experiment adds terms after the training. Also, changing facts can be beyond any lexical representation, and the scope relative to knowledge updating should be stated explicitly.
  • The optimization pipeline brittleness. Naive end-to-end retrofitting (48.8) falls below plain SFT (49.5). The method seems to depend on a fine-grained five-phase schedule with many phase-specific choices
Questions for authors
  • Can you report results with a trainable Engram memory, and Engram+RAG? Also, since Engram is a sparse-memory method, both total and active counts are needed for a fair comparison.
  • I understand that the resource can be limited, but 0.5B and 1.5B scale is too limited for the reliability of the method compared to modern model scale. Are there results at 7B or larger (e.g., Qwen2.5-7B), even for a single domain?
  • How does LCM affect performance on general-domain benchmarks relative to the base model and SFT?
  • From my understanding, the “Engram” described in the paper is not Engram as originally proposed. Original Engram learns its embedding tables jointly with the model during pretraining. Here it is reduced to a hashed-routing variant of LCM’s own frozen inventory. The comparison therefore tests routing designs within LCM’s framework, not LCM against Engram.
8 comments
AuthorsOpenReview ↗

Response to Question 1: Trainable Engram, Engram+RAG, and Memory Activity

Thank you very much for taking the time to carefully review our work and for the thoughtful and constructive feedback. We greatly appreciate your positive assessment of the problem motivation, the practical retrofit setting, and the controlled empirical evaluation. Your comments were very encouraging to us. They also highlighted several important directions for strengthening the evaluation, and we have taken these suggestions seriously in the additional experiments below.

We first distinguish two complementary Engram comparisons. On Qwen, Controlled Engram uses the same backbone, retrofit interface, training schedule, and source lexical inventory as LCM, providing a controlled comparison of the memory and routing design. We additionally evaluate Native Engram on Argon, where Engram is an original trainable component of the model architecture. This second setting directly tests LCM against a native trainable Engram memory.

To address the missing Engram+RAG comparison in the original results, we first add RAG to Controlled Engram under exactly the same Qwen2.5-1.5B setting:

MethodMedQAMedMCQA
SFT + Original RAG44.0±.7544.0_{\pm .75}49.0±.3849.0_{\pm .38}
Controlled Engram + Original RAG44.9±.5144.9_{\pm .51}49.2±.2349.2_{\pm .23}
LCM + Original RAG46.6±.2446.6_{\pm .24}50.0±.3850.0_{\pm .38}

Giving Controlled Engram the same retrieval support improves it over SFT + RAG by +0.9/+0.2 points. LCM + RAG remains higher by +1.7 points on MedQA and +0.8 points on MedMCQA under the same retrieval setting.

We also constructed a stronger RAG baseline to test this complementarity under a more competitive retrieval pipeline. The original RAG uses BGE-M3 dense retrieval. Strong RAG combines BM25 and BGE-M3 retrieval, fuses their candidates with reciprocal rank fusion, and applies Qwen3-Reranker-0.6B before selecting the same top-3 evidence under the same 512-token context budget. The retrieval corpus and base model remain fixed. On Qwen2.5-1.5B, this strengthens SFT + RAG from 44.0/49.0 to 45.9/49.6, while LCM + Strong RAG reaches 52.3/51.8.

More importantly for the trainable-Engram comparison, we apply the same Strong-RAG pipeline to Native Engram and LCM on Argon:

Argon-0.5BMedQAMedMCQA
Native Engram27.0±.3927.0_{\pm .39}33.6±.2433.6_{\pm .24}
LCM29.8±.3129.8_{\pm .31}34.4±.2634.4_{\pm .26}
Native Engram + Strong RAG29.8±.5129.8_{\pm .51}34.5±.4334.5_{\pm .43}
LCM + Strong RAG33.2±.7333.2_{\pm .73}36.4±.2036.4_{\pm .20}

LCM exceeds Native Engram by +2.8/+0.8 points without RAG. Under the matched Strong-RAG pipeline, the corresponding advantage is +3.4/+1.9 points. Thus, the additional lexical-memory benefit remains present when a native trainable Engram receives the same strengthened retrieval support.

Following your suggestion, we also audited both total memory capacity and actual memory activity. For the Qwen2.5-1.5B comparison, all 1,273 MedQA and 4,183 MedMCQA prompts and token IDs are exactly matched between Controlled Engram and LCM.

MethodTotal memory unitsMedQA active/tokenMedMCQA active/tokenUnique active/example
Controlled Engram295,872 hash buckets0.39730.267987.99 / 21.65
LCM295,871 lexical entries0.04780.030211.25 / 2.48

Controlled Engram has 279,634 occupied buckets among its 295,872 allocated buckets. Native Engram on Argon contains 640,000 slots and performs 16 sparse lookups per input token. The access units are structurally different across the methods. Controlled Engram counts occupied hash-bucket accesses, Native Engram counts native sparse memory-slot lookups, and LCM counts final routed lexical entries. We therefore report their native total and active counts directly without converting them into a cross-method sparsity multiplier.

These results highlight the different memory-access patterns. Engram performs sparse lookup over hashed or native memory slots, while LCM activates its external memory at routed complete-term lexical endpoints. Across both the controlled Qwen setting and the native trainable-Engram setting, LCM retains its gains when retrieval support is provided to both methods. This is consistent with the central design of LCM, where specialized lexical representations are externalized and accessed conditionally as support for the shared reasoning backbone.

AuthorsOpenReview ↗

Response to Question 2: LCM on Larger and Newer Backbones

Thank you for this important suggestion. We appreciate your concern about whether the gains observed at 0.5B–1.5B remain reliable at a more modern model scale. We therefore extended the evaluation to Qwen3-4B and Qwen3-8B on the same medical-domain setting using MedQA and MedMCQA.

BackboneMethodMedQAMedMCQA
Qwen3-4BSFT61.6±.1661.6_{\pm .16}57.2±.3357.2_{\pm .33}
Controlled Engram61.9±.2561.9_{\pm .25}58.0±.7058.0_{\pm .70}
LCM64.7±.4464.7_{\pm .44}59.5±.5859.5_{\pm .58}
Qwen3-8BSFT65.1±.6565.1_{\pm .65}60.7±.4160.7_{\pm .41}
Controlled Engram66.0±.3666.0_{\pm .36}62.1±.6262.1_{\pm .62}
LCM68.3±.5768.3_{\pm .57}63.4±.2963.4_{\pm .29}

On Qwen3-4B, LCM improves over SFT by +3.1 points on MedQA and +2.3 points on MedMCQA. It also improves over Controlled Engram by +2.8 and +1.5 points, respectively.

The same pattern remains on Qwen3-8B. LCM improves over SFT by +3.2 points on MedQA and +2.7 points on MedMCQA, while exceeding Controlled Engram by +2.3 and +1.3 points.

These gains are comparable in magnitude to the original Qwen2.5-1.5B setting, where LCM improves over SFT by +2.7/+2.6 points on MedQA/MedMCQA. The small reduction in the original macro-average gain from 0.5B to 1.5B therefore does not develop into a disappearance of the benefit at larger scale. The absolute performance of the backbone rises substantially, while the additional contribution from LCM remains clearly present at both 4B and 8B.

For controlled adaptation, all methods use the same training regime within each backbone. Qwen3-4B uses full-parameter adaptation for SFT, Controlled Engram, and LCM. Qwen3-8B uses the same matched LoRA configuration across all three methods. We therefore interpret these experiments through within-backbone comparisons.

These results show that the usefulness of explicit specialized lexical memory persists on substantially larger and newer backbones. Stronger models provide more capable shared reasoning representations, while on-demand access to external complete-term lexical representations continues to offer additional support for knowledge-intensive tasks.

AuthorsOpenReview ↗

Response to Question 3: General-Domain Capability Retention

Thank you for raising this important question. We agree that target-domain improvements should be considered together with the model's capability outside the adaptation domain. We therefore evaluated the native pretrained model, matched SFT, and LCM on a suite of general-domain benchmarks at both Qwen3-4B and Qwen2.5-1.5B scales.

In addition to the standard MMLU and AGIEval aggregates, we report two subsets that reduce direct overlap with the medical adaptation domain. MMLU-General excludes the 11 medical subjects from MMLU. AGIEval-OOD further removes the biology task from the AGIEval aggregate used in our evaluation. These subsets help distinguish general-capability retention from improvements that may arise from direct overlap with medical knowledge.

Qwen3-4B

BenchmarkNativeSFTLCM
MMLU66.7868.0267.99
MMLU-General68.9370.4070.57
BBH51.4251.0051.83
GSM8K79.9880.3681.50
AGIEval24.1426.9327.64
AGIEval-OOD24.3127.3528.12

On Qwen3-4B, LCM is at least as strong as matched SFT on five of the six evaluations. The only exception is MMLU, where the difference is −0.03 points. On the two subsets with direct medical overlap reduced, LCM improves over SFT by +0.17 points on MMLU-General and +0.77 points on AGIEval-OOD. We therefore do not observe a systematic general-capability penalty from adding the lexical-memory pathway.

We observe the same pattern on the original Qwen2.5-1.5B scale.

Qwen2.5-1.5B

BenchmarkNativeSFTLCM
MMLU59.5960.8961.12
MMLU-General60.8062.2162.38
BBH42.7641.9242.86
GSM8K56.4154.8156.10
AGIEval37.2727.3228.22
AGIEval-OOD36.5827.2728.21

This second scale also illustrates why both the native model and matched SFT are useful controls. Domain adaptation itself changes some general-domain capabilities. For example, SFT decreases from 37.27 to 27.32 on AGIEval and from 36.58 to 27.27 on AGIEval-OOD relative to the native model. LCM does not deepen these reductions. It improves over matched SFT on all six evaluations, with gains ranging from +0.17 to +1.29 points. The same conclusion holds on the medical-overlap-reduced subsets, where LCM improves by +0.17 on MMLU-General and +0.94 on AGIEval-OOD.

Taken together, the results separate the effect of domain adaptation from the effect of introducing LCM. General-capability changes relative to the native model depend on the backbone and benchmark, while LCM does not introduce a systematic additional degradation beyond matched SFT across the evaluated suite. This is consistent with the central design of LCM: specialized lexical support is externalized into an on-demand pathway while the pretrained backbone continues to provide the shared representation and reasoning capacity.

Reviewer_9iaUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework that externalizes specialized lexical representations while preserving the native tokenizer and pre-trained backbone. The goal is to address the fragmentation of domain-specific terminology into subtokens, which may impair reasoning. It first creates a deterministic variable-length lexical routing mechanism to recognize domain terms, then uses hierarchical conditional memory injection to integrate the new term representations with LLM hidden states. The authors mainly evaluate the framework’s performance on datasets from medicine, law, and chemistry using Qwen2.5 models.

Critical strengths

The paper clearly presents its motivation and provides an empirical evaluation of the proposed method on 0.5B-1.5B models.

Critical weaknesses

A core assumption of this work is that fragmenting domain-specific terminology into subtokens harms LLM reasoning. However, I am unsure whether this assumption holds. Some prior work suggests that the early layers of LLMs may already combine subtoken representations as listed below, but I did not find experiments or analysis in the paper that directly address this possibility or validate the assumption.

In addition, the evaluation is limited to relatively small models (0.5B–1.5B parameters), and the paper does not explain whether the framework can be applied to larger models. This leaves the broader applicability of the approach unclear.

Related work:
1. Kaplan et al., From Tokens to Words: On the Inner Lexicon of LLMs (ICLR 2025).
2. Feucht et al., Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs (EMNLP 2024)

Questions for authors
  1. It would be helpful to clarify how widespread the effect of terminology fragmentation on LLM reasoning is, including which model sizes or types are affected and whether it persists in larger models, such as 3B, 7B or larger models.
  2. The framework includes several components, such as alignment, gating, and joint task adaptation, but their individual effects are not examined through ablation studies. Those analyses would help clarify their relative contributions and whether each is necessary.
  3. The paper claims that the framework enables the “efficient incorporation of evolving domain vocabularies,” but all experiments use static, predefined vocabularies. Additional evidence or discussion would help clarify how the framework supports evolving vocabularies and how its efficiency is assessed in that setting.
5 comments
AuthorsOpenReview ↗

Response to Question 1: Fragmentation and Larger Backbones

Thank you very much for the careful review and for your encouraging assessment of the motivation and empirical evaluation. We especially appreciate you pointing us to Kaplan et al. and Feucht et al. These studies provide an important perspective for interpreting our results: LLMs can internally consolidate multi-subtoken words into higher-level lexical representations in their early and middle layers. This insight helped us sharpen the central question behind LCM.

LCM is motivated by separating specialized lexical memory from the shared parametric pathway used for general reasoning. The relevant question is therefore not whether a model can internally compose subtokens, but whether specialized domain terms must rely entirely on this implicit reconstruction process, or whether direct access to complete-term lexical representations still provides additional value. Our results indicate that this value becomes larger as native-token fragmentation increases.

We analyze this directly by pooling all six benchmarks on Qwen2.5-0.5B over the same three runs as the main results. Fragmentation is defined by the maximum native-subtoken length Lmax⁡L_{\max} among matched terms, with all bins fixed before evaluating model performance.

Subset#ExamplesSFTEngramLCMLCM−SFTLCM−Engram
No hit1,78537.336.838.1+0.8+1.3
Hit13,92157.258.260.0+2.8+1.8
Lmax⁡=2L_{\max}=22,22842.344.543.9+1.6−0.6
Lmax⁡=3–4L_{\max}=3\text{--}46,27260.361.463.0+2.7+1.6
Lmax⁡≥5L_{\max}\geq 55,42159.760.163.3+3.6+3.2

The pattern is pronounced. LCM improves over SFT by +0.8 points on examples without a lexical hit and by +2.8 points on examples where lexical memory can intervene. More importantly, the gain over SFT increases from +1.6 to +2.7 to +3.6 points as fragmentation grows. Relative to Engram, the difference changes from −0.6 at Lmax⁡=2L_{\max}=2 to +1.6 at Lmax⁡=3–4L_{\max}=3\text{--}4 and +3.2 at Lmax⁡≥5L_{\max}\geq 5. Thus, explicit complete-term access becomes most useful precisely when the native tokenizer decomposes a specialized term into more pieces.

Your question also raises an important second possibility: stronger models may become better at performing this implicit lexical composition internally. We therefore extended the evaluation to Qwen3-4B and Qwen3-8B on the same medical setting.

BackboneMethodMedQAMedMCQA
Qwen3-4BSFT61.6±.1661.6_{\pm .16}57.2±.3357.2_{\pm .33}
Engram61.9±.2561.9_{\pm .25}58.0±.7058.0_{\pm .70}
LCM64.7±.4464.7_{\pm .44}59.5±.5859.5_{\pm .58}
Qwen3-8BSFT65.1±.6565.1_{\pm .65}60.7±.4160.7_{\pm .41}
Engram66.0±.3666.0_{\pm .36}62.1±.6262.1_{\pm .62}
LCM68.3±.5768.3_{\pm .57}63.4±.2963.4_{\pm .29}

On Qwen3-4B, LCM improves over SFT by +3.1/+2.3 points on MedQA/MedMCQA and over Controlled Engram by +2.8/+1.5 points. On Qwen3-8B, the corresponding gains are +3.2/+2.7 points over SFT and +2.3/+1.3 points over Controlled Engram. All methods use the same adaptation regime within each backbone, with full-parameter adaptation at 4B and a matched LoRA configuration at 8B.

These two analyses address complementary parts of the question. The fragmentation-conditioned result directly shows that the incremental value of explicit lexical memory grows with native-token fragmentation on the pooled six-benchmark analysis. The larger-backbone experiments then show that the overall benefit of this lexical pathway remains present on substantially stronger Qwen3 models.

Taken together, the results support the core design principle of LCM: stronger backbones can form increasingly capable internal representations, while specialized complete-term lexical memory remains valuable as an external, on-demand source of lexical support, reducing reliance on implicit reconstruction and storage within the shared parametric backbone.

AuthorsOpenReview ↗

Response to Question 2: Component-wise Analysis of the Staged Retrofit

Thank you for this helpful suggestion. We conducted a component-wise ablation of the staged retrofit on Qwen2.5-0.5B using MedQA and MedMCQA, with the same three-run evaluation protocol as the main experiments. We remove one training stage at a time while keeping the remaining procedure unchanged. We also include a deterministic hard-gate variant and the naive end-to-end retrofit from the original evaluation.

VariantMedQAMedMCQAMean
Decoupled staged retrofit38.3±.3138.3_{\pm .31}41.6±.4341.6_{\pm .43}39.95
w/o Phase-I Alignment36.9±.2336.9_{\pm .23}40.8±.3840.8_{\pm .38}38.85
w/o Phase-II Forced Utilization34.2±.3434.2_{\pm .34}39.8±.6539.8_{\pm .65}37.00
w/o Phase-III Gate Calibration36.9±.2136.9_{\pm .21}39.2±.5639.2_{\pm .56}38.05
w/o Phase-IV Joint Adaptation25.3±.7925.3_{\pm .79}20.8±.9620.8_{\pm .96}23.05
Hard Gate37.5±.5937.5_{\pm .59}39.9±.7039.9_{\pm .70}38.70
Naive E2E retrofit33.9±.3933.9_{\pm .39}38.5±.2838.5_{\pm .28}36.20

The stages make different contributions. Removing Phase-I Alignment lowers the two-task mean by 1.10 points, indicating a modest but consistent benefit from first aligning the external lexical representations with the model interface. Removing Phase-II Forced Utilization causes a larger 2.95-point drop. On MedQA, performance falls from 38.3 to 34.2, close to the SFT result of 34.3. This suggests that explicitly establishing use of the newly introduced lexical pathway is important before later adaptation.

Phase-III Gate Calibration also contributes substantially. Removing it reduces the mean by 1.90 points. The Hard Gate control provides a complementary comparison. Deterministic hit/non-hit gating reaches 38.70 mean accuracy, compared with 39.95 for the fully calibrated learned gate. At the same time, Hard Gate performs slightly above the learned gate without dedicated Phase-III calibration. This indicates that the benefit comes from learned gating together with dedicated calibration, which enables selective lexical access to be integrated more effectively.

The largest degradation occurs when Phase-IV Joint Adaptation is removed. The mean falls from 39.95 to 23.05. The preceding stages prepare the lexical pathway through alignment, utilization, and gate calibration, while Phase IV completes task-level integration with the pretrained model. The naive end-to-end control further supports the importance of this staged process. It reaches only 36.20 mean accuracy, 3.75 points below the full staged retrofit and slightly below the SFT mean of 36.75.

Taken together, the stages form a functionally coupled integration process. Phase I establishes representational compatibility. Phase II ensures that the new lexical pathway is actively utilized. Phase III calibrates selective access, and Phase IV integrates the resulting lexical support with downstream task reasoning. Their effects are therefore complementary and sequential. This reflects the central design principle of LCM: externalizing specialized lexical memory becomes most useful when the pretrained reasoning backbone can access and incorporate that memory through a controlled integration process.

AuthorsOpenReview ↗

Response to Question 3: Incremental Incorporation of Evolving Vocabulary

Thank you for raising this important point. We conducted an additional experiment to directly evaluate post-training incremental lexical-inventory expansion. The goal is to test whether newly introduced domain terminology can be incorporated after LCM has already completed its retrofit training, and to measure the cost of doing so.

We first train LCM using only an initial 40% lexical inventory (V0). The remaining vocabulary is withheld during training and divided into three disjoint 20% batches. After training is complete, all model parameters are frozen. We then expand the external lexical inventory sequentially from 40% to 60%, 80%, and finally 100% by appending the corresponding lexical entries and updating the external routing structure. No gradient update or model retraining is performed at any expansion stage. The vocabulary partitions are fixed before performance evaluation using frequency and native-fragmentation stratification.

Lexical inventoryMedQAMedMCQAMean
V0 40%37.4±.3037.4_{\pm .30}39.7±.1439.7_{\pm .14}38.55
+dV1 60%38.0±.4038.0_{\pm .40}40.6±.3040.6_{\pm .30}39.30
+dV2 80%37.7±.1637.7_{\pm .16}41.2±.5541.2_{\pm .55}39.45
Full 100%38.1±.3338.1_{\pm .33}41.2±.5141.2_{\pm .51}39.65

Expanding the inventory from 40% to 100% after training yields a net improvement of +0.7 points on MedQA and +1.5 points on MedMCQA, corresponding to a +1.10-point increase in the two-task mean. The intermediate MedQA stages show small fluctuations, while the overall 40% to 100% expansion produces a clear net gain.

The final post-training expanded model also approaches the performance of the original LCM trained with the complete lexical inventory from the outset. The original full-inventory LCM obtains 38.3±.3138.3_{\pm .31} on MedQA and 41.6±.4341.6_{\pm .43} on MedMCQA, compared with 38.1±.3338.1_{\pm .33} and 41.2±.5141.2_{\pm .51} after incremental expansion. The remaining gaps are only 0.2 and 0.4 points, respectively.

We further measured the cost of each vocabulary update:

TransitionAdded termsTotal wall timeAdded storageGradient steps
40% → 60%59,17022.658 s305.25 MiB0
60% → 80%59,17021.662 s305.24 MiB0
80% → 100%59,18628.460 s305.33 MiB0

Each update incorporates approximately 59K new lexical entries in 21.7–28.5 seconds, with no parameter optimization. The external lexical store grows approximately linearly from 610.42 MiB at 40% to 1,526.23 MiB at 100%.

These results illustrate an operational benefit of externalizing specialized lexical memory. Once the pretrained model has learned how to access the lexical pathway, newly introduced terminology can be incorporated by extending the external lexical store and routing index while keeping the shared parametric backbone fixed. This allows the specialized vocabulary to evolve independently of model retraining and directly supports the long-term maintainability motivation of LCM.

Reviewer_f965OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LCM augments pretrained language models with exact matching of complete domain terms and a frozen lexical table of pooled native-token embeddings. Two depth-specific gated interfaces and a staged retrofit integrate these representations while preserving the tokenizer and backbone architecture. Six benchmarks, two Qwen scales, native-Engram Argon validation, content and routing controls, RAG combinations, and off-device measurements evaluate its domain-adaptation value.

Critical strengths
  1. The evaluation covers medicine, law and chemistry, with consistent results across the tested settings. The controlled Engram retrofit, the native-Engram comparison on Argon, and the combined LCM+RAG setting provide useful evidence for lexical adaptation and its compatibility with retrieval (Tables 1–2, §4.2).

  2. The main-text ablations help assess the roles of the injection interfaces and staged retrofitting. The content and routing controls provide further support for the design choices, while latency and lexical-store measurements clarify the implementation costs (Tables 3–4, §§4.3–4.5).

Critical weaknesses
  1. The conceptual advance remains modest. As the related work acknowledges (§2.1), exact longest-match memory access already appears in Memory Grafting. LCM combines full domain-term lookup, gated multi-depth injection and staged adaptation (§§3.2–3.4). The gains and controls in §§4.2–4.4 support this combination, but establish a useful domain adaptation more clearly than a substantially new memory mechanism.

  2. Model coverage is limited to small backbones. Tables 1–2 evaluate Qwen2.5-0.5B, Qwen2.5-1.5B and Argon-0.5B; the Argon result adds architectural diversity but not evidence beyond 1.5B. There is no validation on larger backbones or newer families such as Qwen3 or Qwen3.5. It therefore remains unclear whether explicit full-term memory offers comparable benefits when the backbone already has stronger lexical and domain representations. A representative larger, newer open-weight model would materially strengthen the broader applicability claim; otherwise, the conclusions should be explicitly limited to the evaluated small-model setting.

Questions for authors
  1. From the main-text results in §§4.2–4.4, what is the strongest new insight beyond combining exact lexical lookup and gated memory injection for domain adaptation?

  2. How does LCM’s benefit change with backbone capability? A controlled comparison on a representative larger and newer open-weight model, for example Qwen3-8B or Qwen3.5-9B, would help distinguish broadly useful lexical support from benefits particular to small models.

  3. The gains over the current RAG baseline in Table 1 appear modest. Would LCM remain competitive against a stronger, well-tuned RAG pipeline, such as hybrid retrieval with a recent reranker? In particular, does adding LCM still help when the RAG pipeline is strengthened? Please keep the base model, retrieval corpus and context budget fixed, and report accuracy alongside inference cost.

4 comments
AuthorsOpenReview ↗

Response to Question 1: Strongest Insight Beyond Exact Lexical Lookup

Thank you very much for taking the time to carefully review our work and for the thoughtful and constructive feedback. We are also grateful for your encouraging assessment of the paper’s empirical breadth and analysis. Your comments helped us sharpen the central insight behind LCM.

At a high level, LCM is motivated by a simple separation: specialized lexical memory need not be stored and repeatedly reconstructed inside the same parametric pathway used for general reasoning. By externalizing complete domain-term representations while preserving the native tokenizer and pretrained backbone, LCM provides an explicit lexical pathway that can be accessed on demand. The strongest empirical insight from our results is that the value of this pathway increases with native-token fragmentation.

Our fragmentation-conditioned analysis pools all six benchmarks on Qwen2.5-0.5B and averages the same three runs as the main results. Fragmentation is measured by the maximum native-subtoken length Lmax⁡L_{\max} among matched lexical terms, with bins fixed before model evaluation.

Subset#ExamplesSFTEngramLCMLCM−SFTLCM−Engram
No hit1,78537.336.838.1+0.8+1.3
Hit13,92157.258.260.0+2.8+1.8
Lmax⁡=2L_{\max}=22,22842.344.543.9+1.6−0.6
Lmax⁡=3–4L_{\max}=3\text{--}46,27260.361.463.0+2.7+1.6
Lmax⁡≥5L_{\max}\geq 55,42159.760.163.3+3.6+3.2

Two patterns stand out. First, LCM improves over SFT by only +0.8 points on examples without a lexical hit, compared with +2.8 points when lexical memory can intervene. More importantly, the LCM gain over SFT grows from +1.6 to +2.7 to +3.6 points as fragmentation increases. The comparison with Engram is even more informative: LCM changes from −0.6 at Lmax⁡=2L_{\max}=2 to +1.6 at Lmax⁡=3–4L_{\max}=3\text{--}4 and +3.2 at Lmax⁡≥5L_{\max}\geq 5. The advantage therefore becomes strongest precisely where complete domain terms are most fragmented by the native tokenizer.

The routing factorial provides a complementary view of where this gain comes from:

RoutingBounded 2–4Variable-length
Hashed50.351.7
Exact50.552.5

Variable-length complete-term routing improves accuracy by +1.4 points under hashed addressing and +2.0 points under exact addressing. By comparison, switching from hashed to exact addressing yields only +0.2 points with bounded 2–4 routing, while its gain increases to +0.8 points once variable-length complete-term routing is available. This shows that recovering the complete lexical unit is the larger standalone contributor, while deterministic exact addressing provides an additional benefit once that representation is available.

The training side shows the same overall picture. On MedQA and MedMCQA, the full decoupled staged retrofit reaches a two-task mean of 39.95, whereas a naive end-to-end retrofit reaches 36.20. Thus, providing an external lexical pathway is not sufficient by itself; the pretrained model must also learn to incorporate that specialized information without collapsing it back into the shared reasoning pathway.

Taken together, we see the central contribution as a fragmentation-conditioned role for complete lexical memory: as native tokenization increasingly breaks specialized terms into smaller pieces, explicitly externalizing and retrieving complete-term representations becomes increasingly useful. The routing controls identify complete-term lexicalization as the main contributor, while the staged retrofit enables this external lexical support to be integrated effectively into the pretrained reasoning backbone.

AuthorsOpenReview ↗

Response to Question 2: LCM on Larger and Newer Backbones

Thank you for this important suggestion. We agree that evaluating stronger and newer backbones is essential for understanding whether explicit lexical support remains useful as the underlying model becomes more capable. We therefore extended the evaluation to Qwen3-4B and Qwen3-8B on MedQA and MedMCQA, using the same medical domain setting.

At a high level, this experiment also tests a central motivation of LCM. Stronger backbones provide substantially better general reasoning and domain performance, but specialized lexical knowledge can still be maintained through an explicit external pathway rather than relying entirely on the shared parametric backbone. The question is whether this separation remains useful once the backbone itself becomes much stronger.

BackboneMethodMedQAMedMCQA
Qwen3-4BSFT61.6±.1661.6_{\pm .16}57.2±.3357.2_{\pm .33}
Engram61.9±.2561.9_{\pm .25}58.0±.7058.0_{\pm .70}
LCM64.7±.4464.7_{\pm .44}59.5±.5859.5_{\pm .58}
Qwen3-8BSFT65.1±.6565.1_{\pm .65}60.7±.4160.7_{\pm .41}
Engram66.0±.3666.0_{\pm .36}62.1±.6262.1_{\pm .62}
LCM68.3±.5768.3_{\pm .57}63.4±.2963.4_{\pm .29}

For Qwen3-4B, LCM improves over SFT by +3.1 points on MedQA and +2.3 points on MedMCQA. It also improves over Controlled Engram by +2.8 and +1.5 points, respectively.

The same pattern holds on Qwen3-8B. LCM improves over SFT by +3.2 points on MedQA and +2.7 points on MedMCQA, while remaining ahead of Controlled Engram by +2.3 and +1.3 points.

These gains are comparable in magnitude to the original Qwen2.5-1.5B setting, where LCM improves over SFT by +2.7/+2.6 points on MedQA/MedMCQA. At the same time, the absolute performance of the Qwen3 backbones is substantially higher. For example, SFT increases from 42.7/47.1 on Qwen2.5-1.5B to 65.1/60.7 on Qwen3-8B, yet LCM still provides an additional 3.2/2.7-point improvement.

For controlled adaptation, all methods within each backbone use the same training regime. Qwen3-4B uses full-parameter adaptation for SFT, Controlled Engram, and LCM. Qwen3-8B uses the same matched LoRA configuration across all three methods.

Overall, we do not observe the benefit of explicit lexical memory diminishing as backbone capability increases. Across both newer Qwen3 scales, LCM consistently improves over both the matched SFT and Controlled Engram baselines. This suggests that stronger parametric reasoning and explicit specialized lexical memory remain complementary: increasing backbone capability raises the overall performance level, while on-demand access to external complete-term representations continues to provide additional support.

AuthorsOpenReview ↗

Response to Question 3: LCM with a Stronger RAG Pipeline

Thank you for this thoughtful suggestion. We followed the proposed controlled setting and strengthened only the retrieval pipeline while keeping the base model, retrieval corpus, and context budget fixed. Specifically, we use the same Qwen2.5-1.5B backbone and the same medical retrieval corpus as in Table 1. The final retrieved evidence remains top-3 passages within the same 512-token document-context budget.

The stronger pipeline combines BM25 and BGE-M3 dense retrieval, fuses their top candidates with reciprocal rank fusion, and reranks the resulting candidate set using Qwen3-Reranker-0.6B before selecting the final top-3 passages. This gives us a substantially stronger hybrid-and-reranked RAG baseline without changing the information source or allowing additional context.

MethodMedQAMedMCQA
SFT + Original RAG44.0±.7544.0_{\pm .75}49.0±.3849.0_{\pm .38}
SFT + Strong RAG45.9±.3345.9_{\pm .33}49.6±.6249.6_{\pm .62}
LCM + Strong RAG52.3±.4752.3_{\pm .47}51.8±.4951.8_{\pm .49}

The strengthened RAG pipeline itself improves over the original dense-only RAG by +1.9 points on MedQA and +0.6 points on MedMCQA, confirming that it provides a stronger retrieval baseline. More importantly, adding LCM on top of this stronger pipeline yields a further +6.4 points on MedQA and +2.2 points on MedMCQA, corresponding to a +4.3-point two-task mean improvement over the matched Strong-RAG baseline.

We also measured inference cost under a matched online setting on the same A100 GPU, using batch size 1 and the same fixed 400-query subset over three timed rounds.

SettingRetrieval (ms/query)Generation (ms/query)E2E (ms/query)
SFT + Original RAG13.706164.907178.613
SFT + Strong RAG1592.983169.9531762.936
LCM + Strong RAG1589.067308.6301897.697

As expected, hybrid retrieval with neural reranking is substantially more expensive than the original dense-only pipeline. Within the matched Strong-RAG setting, LCM increases generation latency from 170.0 to 308.6 ms/query. Because retrieval dominates the overall pipeline, this corresponds to an end-to-end increase of 134.8 ms/query, or 7.64%, from 1762.9 to 1897.7 ms/query. Peak allocated GPU memory is 32,961.6 MiB for both SFT + Strong RAG and LCM + Strong RAG, so we observe no measurable additional peak GPU allocation in this setting.

These results also connect directly to the central motivation of LCM. Stronger RAG improves the factual evidence supplied through the context, while LCM provides on-demand access to specialized complete-term lexical representations outside the shared parametric backbone. Strengthening factual retrieval therefore does not remove the utility of the lexical pathway. Under the stronger hybrid-and-reranked pipeline, the two forms of support remain complementary, with LCM providing additional accuracy at a modest incremental end-to-end system cost.

AuthorsOpenReview ↗

Summary of Additional Evidence Addressing the Main Concerns

Thank you again for the thoughtful and constructive review. To make the additional evidence easier to assess, we briefly summarize how the new experiments address the three main concerns raised in the review.

1. Mechanistic insight.
The clearest new finding is that the value of explicit complete-term lexical memory increases with native-token fragmentation. Across the pooled six-benchmark analysis, LCM's gain over SFT rises from +1.6 to +2.7 to +3.6 points as matched terms increase from 2 subtokens to 3–4 and then at least 5 subtokens. The routing factorial further separates the contributions of span construction and addressing. Variable-length complete-term routing contributes +1.4 points under hashed addressing and +2.0 points under exact addressing, while exact addressing adds a further +0.2/+0.8 points under bounded/variable-length routing. A full staged-training ablation also shows that the decoupled retrofit improves the Medicine average from 36.2 to 40.0 over naive end-to-end optimization. Together, these results identify both when complete lexical memory is most useful and how it is effectively integrated into the pretrained backbone.

2. Larger and stronger backbones.
We extended the evaluation to Qwen3-4B and Qwen3-8B on MedQA and MedMCQA under matched adaptation settings. On Qwen3-4B, LCM improves over SFT by +3.1/+2.3 points and over Controlled Engram by +2.8/+1.5 points. On Qwen3-8B, the corresponding gains are +3.2/+2.7 over SFT and +2.3/+1.3 over Controlled Engram. The absolute backbone performance increases substantially, yet the benefit of explicit lexical support remains. This extends the original Qwen2.5 results to larger and newer models.

3. Stronger retrieval.
Following the suggested controlled setting, we strengthened only the retrieval pipeline using hybrid BM25 and BGE-M3 retrieval, reciprocal-rank fusion, and Qwen3-Reranker-0.6B, while keeping the base model, retrieval corpus, and 512-token document-context budget fixed. LCM + Strong RAG improves over the matched SFT + Strong RAG baseline by +6.4 points on MedQA and +2.2 points on MedMCQA, corresponding to a +4.3-point two-task mean gain. The incremental end-to-end latency of adding LCM within this Strong-RAG pipeline is 7.64%, with no measurable increase in peak allocated GPU memory.

We further performed an evidence-level lexical-routing audit on the exact retrieved evidence used by these predictions. Under Strong RAG, the pooled lexical token hit rate increases from 5.38% to 5.82% (+0.44 pp, 95% CI [+0.36, +0.53]), corresponding to an approximately 8.2% relative increase. This provides additional evidence that stronger retrieval supplies evidence with a higher density of specialized content addressable by LCM's lexical pathway.

Taken together, the additional experiments strengthen a consistent picture. Specialized lexical memory can be externalized from the shared parametric pathway used for general reasoning, and its value becomes especially clear when complete domain terms are strongly fragmented by native tokenization. The same benefit persists as backbone capability increases and remains useful under substantially stronger retrieval.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Additional Evidence and Manuscript Revisions (Part 1/2)

We sincerely thank all reviewers for their careful reading, constructive comments, and detailed suggestions. We appreciate that the reviews recognized the motivation of separating specialized knowledge from general computation, the practicality of retrofitting existing pretrained models without modifying the native tokenizer, and the breadth of our evaluation across medicine, law, and chemistry. Reviewers also recognized the careful experimental controls, including matched baselines, routing and memory-content ablations, staged-training analysis, and efficiency measurements. Their feedback helped us clarify the central motivation of LCM and motivated several additional experiments that directly test its key claims.

The revised manuscript is organized around one central idea: specialized lexical memory does not need to be stored and repeatedly reconstructed within the same shared parametric pathway used for general reasoning. LCM provides an explicit external pathway for complete-term lexical representations while preserving the native tokenizer and pretrained backbone. The original experiments established the effectiveness of this design across six knowledge-intensive benchmarks. The new evidence complements those results by examining why the lexical pathway helps, whether its benefit persists on stronger models and different architectures, whether the external memory can evolve after training, and how it can be integrated effectively.

1. The benefit of complete lexical access increases with native-token fragmentation

The original multi-domain experiments showed consistent gains over SFT and Controlled Engram. We now directly examine when these gains become most pronounced.

Across pooled lexical-hit examples, the LCM gain over SFT increases from +1.6 points for terms represented by two native subtokens to +2.7 points for three to four subtokens and +3.6 points for five or more subtokens. Relative to Controlled Engram, the corresponding difference grows from -0.6 to +1.6 and +3.2 points.

A routing-factorial analysis provides complementary evidence. Variable-length complete-term routing improves the macro-average by +1.4 points under hashed addressing and +2.0 points under exact addressing. Exact addressing contributes an additional +0.2 and +0.8 points under bounded and variable-length routing, respectively.

Together, these results refine the interpretation of LCM. Internal subtoken composition can occur in pretrained LLMs, while explicit complete-term lexical support becomes increasingly useful as specialized terminology is fragmented into more native subtokens. The routing analysis further shows that recovering the complete lexical unit provides the larger standalone contribution, with exact addressing adding further benefit.

2. The benefit persists on stronger backbones and across memory architectures

We extend the original Qwen2.5 evaluation to Qwen3-4B and Qwen3-8B.

On Qwen3-4B, LCM improves over SFT by +3.1/+2.3 points on MedQA/MedMCQA and over Controlled Engram by +2.8/+1.5 points. On Qwen3-8B, the corresponding gains are +3.2/+2.7 points over SFT and +2.3/+1.3 points over Controlled Engram.

These larger-backbone results complement the original Argon experiment. On Argon-0.5B, LCM improves the six-benchmark macro-average from 44.1 with Native Engram to 46.5. The two settings therefore provide complementary evidence. Qwen3 evaluates whether explicit lexical support remains useful as the shared backbone becomes substantially stronger, while Argon evaluates LCM against a natively integrated Engram architecture.

The consistent pattern across Qwen2.5, Qwen3, and Argon indicates that stronger parametric representations and explicit specialized lexical memory remain complementary.

In Part 2 below, we summarize the additional evidence on post-training vocabulary evolution, general-capability retention, staged integration, and stronger retrieval.

AuthorsOpenReview ↗

Summary of Additional Evidence and Manuscript Revisions (Part 2/2)

3. External lexical memory can evolve after training while general capabilities are preserved relative to matched adaptation

A second motivation of LCM concerns large and evolving specialized vocabularies. We now test this property directly.

We train LCM using only the initial 40% of a medical lexical inventory. The remaining 60% is unavailable during training. After training, all model parameters are frozen, and the external lexical store is expanded sequentially to 60%, 80%, and 100% with zero gradient updates.

Expanding the inventory from 40% to 100% improves MedQA by +0.7 points and MedMCQA by +1.5 points. The final expanded model remains only 0.2/0.4 points below an LCM model trained with the complete inventory from the outset. Thus, newly introduced terminology can be incorporated by updating the external lexical store and routing index without retraining the shared model.

We also evaluate general capabilities on MMLU, MMLU-General, BBH, GSM8K, AGIEval, and AGIEval-OOD. On Qwen2.5-1.5B, LCM exceeds matched domain SFT on all six evaluations. On Qwen3-4B, it exceeds SFT on five of six, with only a 0.03-point difference on MMLU.

These two analyses support complementary aspects of the same design principle. Specialized lexical knowledge can be extended outside the backbone after training, while the resulting model preserves general capabilities at a level comparable to or stronger than matched domain SFT.

4. Staged retrofitting enables effective integration of the external lexical pathway

We further add a component-wise study of the staged retrofit on MedQA and MedMCQA.

The complete staged procedure reaches a Medicine average of 40.0, compared with 36.2 for naive end-to-end retrofitting. Removing Phase I alignment, Phase II forced utilization, Phase III gate calibration, and Phase IV joint adaptation reduces the average to 38.9, 37.0, 38.1, and 23.1, respectively.

The results clarify the functional role of the training procedure. Alignment establishes compatibility between lexical representations and the pretrained model. Forced utilization encourages the model to make effective use of the new pathway. Gate calibration supports selective access, and joint adaptation integrates lexical support with downstream reasoning.

This analysis shows that externalizing lexical memory and learning to use that memory are complementary parts of the retrofit.

Additional retrieval evidence

We also strengthen the RAG comparison using hybrid BM25 and dense retrieval followed by neural reranking, while keeping the retrieval corpus and context budget fixed. The stronger pipeline improves the RAG baseline. Adding LCM on top of it still provides an additional +6.4/+2.2 points on MedQA/MedMCQA.

This result further separates the roles of the two mechanisms. Retrieval provides task-relevant factual evidence through the context, while LCM provides on-demand access to specialized complete-term lexical representations. Their benefits remain complementary under a substantially stronger retrieval pipeline.

Overall picture

The new evidence extends the original multi-domain evaluation along a coherent progression:

effectiveness → mechanism → larger-backbone generalization → architectural generalization → post-training evolvability → capability retention → controlled integration.

Taken together, the results strengthen the central message of LCM. Explicit complete-term lexical memory is especially useful when specialized terminology is strongly fragmented by native tokenization. Its benefit remains present as backbone capability increases, and the external lexical inventory can continue to evolve without modifying the shared model. This supports the broader design principle of decoupling specialized lexical memory from the shared parametric pathway responsible for general representation and reasoning.

We have incorporated these analyses into the revised manuscript, including the larger Qwen3 validation, fragmentation-conditioned analysis, post-training lexical-inventory expansion, general-capability evaluation, and full staged-training ablation.

Finally, we corrected an accidental row swap in the original routing-factorial table. The corrected macro-averages are 50.3/51.7 for hashed bounded/variable-length routing and 50.5/52.5 for exact bounded/variable-length routing. The corrected results support the interpretation above: variable-length complete-term construction provides the larger standalone contribution, and exact addressing provides an additional gain once complete-term routing is available.

We again thank all reviewers for the thoughtful feedback. Their comments helped us strengthen the empirical validation, clarify the scope of the claims, and present the central motivation and contribution of LCM more clearly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.