acceptodds
Under review as a conference paper at ICLR 2027

Co-LMLM: Continuous-Query Limited Memory Language Models

Abstract

Externalizing knowledge in LLM pre-training is a promising avenue to achieve higher performance at smaller scales, control knowledge use, and overall increase model transparency. We propose continuous-query limited memory language models (Co-LMLM), an LLM that interleaves flexible vector retrieval queries with next-token predictions, and is pre-trained to copy knowledge returned from the KB, rather than memorize it. Co-LMLM is pre-trained with a scalable approach that jointly trains a knowledge-externalizing LLM, induces its knowledge base, and learns an expressive continuous retrieval mechanism. Across pre-training at multiple model scales, Co-LMLM outperforms prior knowledge-externalizing and vanilla LLMs in both perplexity and factual precision. At 360M scale, this includes lower perplexity than models pre-trained on 40 more data, and SimpleQA-verified performance that is in line with gpt-4o-mini and higher than Claude Sonnet 4.5.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Pretrained Large Language Models, Knowledge Offloading

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qRekOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Co-LMLM, a method that stores factual knowledge in an external knowledge base while training a language model to generate text conditioned on retrieved information during pretraining. The approach annotates factual spans and their corresponding questions, then applies bidirectional contrastive learning to align the hidden states at the <FACT> position in the text and the <FACT-q> position in the question. At inference time, the model uses these hidden states to query an external index, retrieves free-text factual fragments, and continues generation based on the retrieved content. Extensive experiments demonstrate substantial improvements in factual question answering while maintaining language understanding performance comparable to that of general-purpose language models in certain settings.

Critical strengths
  1. The paper addresses an important problem, and the motivation and methodological design are clear. Free-text factual fragments overcome the representational limitations of relational triples, while queries derived directly from hidden states reduce the computational overhead of explicit query generation. In addition, the training format encourages the model to use external information during generation. Together, these design choices form a coherent and well-motivated approach.

  2. The gains in factual accuracy over the baseline are substantial. Co-LMLM notably outperforms REL-LMLM in both short- and long-form generation under the multitask setting, providing strong evidence of the method’s effectiveness.

Critical weaknesses
  1. Its applicability to larger models remains to be verified. The paper demonstrates the effectiveness of Co-LMLM on 135M and 360M models, but these results are insufficient to determine whether its benefits extend to larger models. Although the authors acknowledge this limitation, further validation at larger scales would strengthen the case for the method’s scalability.

  2. The overall cost assessment and comparison with recent memory-based methods remain insufficient. Co-LMLM incurs additional costs for corpus annotation, question generation, and training, making it difficult to assess the trade-off between its overall performance gains and resource requirements. Furthermore, the experimental comparisons focus primarily on REL-LMLM and LMLM-ASKER. Comparisons with recent representative methods, such as an adapted Engram, under comparable resource budgets—or further discussion of their comparability—would help clarify the advantages of the proposed method.

Questions for authors
  1. Based on the results for the 135M and 360M models, are there any indications that the relative benefits of knowledge externalization vary with model size? If preliminary experiments with larger models are available, could the authors provide further details?

  2. Could the authors further quantify the overhead associated with corpus annotation and question generation, and report inference costs including retrieval, to help assess the trade-off between overall performance gains and resource requirements?

  3. Have the authors considered experimental comparisons with recent methods such as Engram under comparable resource budgets? If such comparisons pose substantial adaptation challenges, could the authors explain these challenges and clarify which comparative conclusions the current experiments support?

2 comments

We thank the reviewer for their thoughtful comments. We would like to address the reviewer’s questions:

Q1: Based on the results for the 135M and 360M models, are there any indications that the relative benefits of knowledge externalization vary with model size? If preliminary experiments with larger models are available, could the authors provide further details?

We agree that building scaling laws is important, and this is an ongoing research direction for us. In addition to the complexities of building scaling ladders in conventional LLMs, CoLMLM also grows along the KB. A new method of building scaling ladders is needed. Borgeaud et al., 2022 and Shao et al., 2024 provide a base on which we are building. Overall, this is the right way to go, and it’s a new project on its own. In our current results, Co-LMLM’s advantage over the Standard LM is stable from 135M to 360M (Table 1), e.g. +22.0 / +23.9 on TriviaQA, +32.1 / +30.3 on PopQA, +18.1 / +20.9 on SimpleQA.

Q2: Could the authors further quantify the overhead associated with corpus annotation and question generation, and report inference costs including retrieval, to help assess the trade-off between overall performance gains and resource requirements?

We will extend the discussion of this in the paper. As you review the numbers below, please keep in mind that our implementation and cluster are not optimized. Annotation and training measured on one NVIDIA B200 GPU (tokens are SmolLM2 tokens):

StageThroughputCost
Corpus annotation (fact spans and questions), Wikipedia / FineWeb-Edu16.4K / 34.9K tokens/s per GPU16.9 / 8.0 GPU-hours per 1B tokens, once per corpus
Training, 135M: Standard / Co-LMLM457K / 254K tokens/s per GPU0.61 / 1.09 GPU-hours per 1B tokens (Co-LMLM at 0.56 of the Standard LM's throughput)
Training, 360M: Standard / Co-LMLM243K / 138K tokens/s per GPU1.14 / 2.02 GPU-hours per 1B tokens (0.57)
Inference, 360M, batch 1: Standard / Co-LMLM1.05 / 1.27 ms per tokenCo-LMLM at 1.21 times the latency
Inference, 360M, 256 concurrent sequences: Standard / Co-LMLM71K / 46K tokens/sCo-LMLM at 0.65 of the throughput

Annotation is paid once per corpus. Question generation is 80-90% of it, and its cost per token grows with fact density (59 facts per 1K tokens on Wikipedia, 19 on FineWeb-Edu). The compute share of the annotation process relative to the training process will decrease as we scale our models.

Training currently runs at 1.8 times the GPU time per token compared to a standard LLM. Each fact’s paired question goes forward and backward through the model (overall, 39% more FLOPs), and the contrastive objective lowers utilization.

Inference is measured with the entire Wikipedia index searched on the same GPU using Faiss. At batch 1, Co-LMLM-360M takes 1.27 ms per token against 1.05 ms for the Standard LM. This overhead is driven by the extra tokens CoLMLM processes in the retrieval block, and by the index search.

While Co-LMLM is more expensive than standard LMs when comparing compute per token, it allows for entirely new capabilities and opens up a new avenue for developing language models where the user has direct control and observability over the model’s factual knowledge.

Q3: Have the authors considered experimental comparisons with recent methods such as Engram under comparable resource budgets? If such comparisons pose substantial adaptation challenges, could the authors explain these challenges and clarify which comparative conclusions the current experiments support?

Engram is fundamentally different from Co-LMLM in that it still stores knowledge in the LM’s weights, while Co-LMLM keeps it externalized as readable text. Engram’s entries are learned vectors, which cannot be read, attributed to a source, or edited directly. The two methods therefore serve substantially different purposes: Engram raises parametric capacity at fixed compute, while Co-LMLM externalizes knowledge in a controllable manner. Engram could in principle serve as a stronger parametric baseline than our Standard LM. However, no Engram model has been released at any size, and the smallest model reported in the paper is a 3B MoE. Furthermore, training code was not released. The official Engram repo contains only a demonstration script missing the implementations of several components and mechanisms. A comparison would therefore require re-implementing Engram from the paper at a scale it was never tested at, and would measure our reimplementation rather than Engram itself.

AuthorsOpenReview ↗

Rebuttal follow-up

Dear Reviewer,
Thank you again for your thoughtful review. As the discussion period nears its end, a brief reminder of our response:

  1. Building scaling laws is important, and this is an ongoing research direction for us. Current results show Co-LMLM's advantage over the Standard LM is stable from 135M to 360M.
  2. We reported annotation, training and inference costs.
  3. We explained why Engram, which stores knowledge in the model's weights and has no released models or training code, is not directly comparable.

We would be glad to address any remaining concerns. If you feel our response addressed them, we would be grateful if you would consider updating your assessment.
Thank you for your time.

Reviewer_pmbQOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates knowledge externalization during language model pretraining to reduce reliance on parametric memory. It proposes Co-LMLM, which triggers retrieval by generating a special token and directly uses the hidden state at that position as a continuous query. Retrieved factual text is inserted into the context before generation resumes. Training combines next-token prediction with a bidirectional contrastive objective, using data constructed through factual-span identification and paired-question generation. Compared with Rel-LMLM, which relies on explicit relational queries, the method supports free-form factual spans and reduces the overhead of generating textual queries.

Critical strengths

The method integrates continuous queries into an implementable pretraining pipeline.

The factuality experiments show substantial improvements across multiple model sizes.

The appendix includes analyses that help interpret the results.

Critical weaknesses

Appendix A.6 claims that actual perplexity lies between Dynamic and Dynamic-normalized perplexity, but the lower-bound interpretation of Dynamic perplexity is not sufficiently justified, since omitting retrieval-trigger probabilities does not rule out alternative trajectories with higher target-text probability. The paper should clarify the generative distribution over retrieval operations, distinguish trajectory scores from marginal text likelihood, and ensure consistent scoring positions and truncation across models.

Section 3.2 can be read as directly bracketing the original factual span and masking its loss, whereas Figure 2 inserts a paraphrased-answer block before that span and then predicts the original span.

Full-annotation precision is reported as 76.1%, but the assessment covers only the opening portions of 25 Wikipedia documents, relies primarily on model-based judgments, and lacks an independent human evaluation and a corresponding analysis on FineWeb.

The URL provided for OneGen in the references is incorrect.

Questions for authors

Refer to the above text.

2 comments

We thank the reviewer for their thoughtful comments. We address the reviewer’s concerns:

Concern 1: lower-bound interpretation of Dynamic perplexity. The point about Dynamic perplexity is correct, and we will fix Appendix A.6 accordingly. The most important observation when discussing dynamic perplexity is that the retrievals are latent actions. This creates a weird artifact in PPL, because PPL is by definition based on having access to the complete gold sequence, and each decision is conditioned on a gold history. This is not possible in LMLM models.
An accurate retrieval-conditioned perplexity computation is intractable, since it would require computing likelihoods for all possible trajectories. Our dynamic perplexity computation is therefore conditioned only on a single chosen trajectory. We will clarify this in the paper.
Regarding the reviewer’s note to ensure consistent scoring positions and truncation across models: all models are already scored on the same documents and the same original text.

Concern 2: Section 3.2 can be read as directly bracketing the original factual span and masking its loss, whereas Figure 2 inserts a paraphrased-answer block before that span and then predicts the original span. We thank the reviewer for pointing out this lack of clarity. Figure 2 is the correct reading. We will improve the wording in Section 3.2. In training, each fact becomes <FACT> answer </FACT> span: the annotated answer sits inside the block, and the original span follows it unchanged. The NTP loss masks the answer tokens and </FACT>. It trains <FACT> and the original span, which the model predicts based on the inserted answer. The answer paraphrases the span by design (Section 3.3, guideline (c)). At inference, the answer span is retrieved from the KB, which comes from another document and is usually worded differently from the current context. Paraphrased answers expose the model to this mismatch during training.
We will revise Section 3.2 to make this clearer.

Concern 3: Annotation quality assessment lacks an independent human evaluation and a corresponding analysis on FineWeb. We manually evaluated 10 annotated Wikipedia documents and 10 annotated FineWeb-Edu documents. Scoring follows Appendix B.1: full annotation (span, question, and answer) vs. detection only (real fact, span may be imperfect). Brackets indicate 95% CIs.

Documents (judge)Full annotation precisionFull annotation recallDetection only precisionDetection only recall
Wikipedia, 10 (human)77.7 [70.3, 83.8]79.5 [71.3, 86.0]96.0 [92.3, 99.4]98.2 [95.4, 100.0]
FineWeb-Edu, 10 (human)83.6 [79.2, 88.2]75.7 [72.0, 81.7]97.0 [91.9, 100.0]87.8 [84.2, 93.3]
All 20 (human)79.3 [74.1, 83.9]78.4 [72.8, 83.5]96.3 [93.3, 98.8]95.1 [92.4, 97.7]
Wikipedia, 25 (LLM, Table 9)76.183.189.197.4

On Wikipedia, human full-annotation precision is in line with the LLM-judge. FineWeb-Edu is similar, so the annotators transfer to web text, although they miss more facts there. We will add this evaluation to Appendix B.1.

Concern 4: The URL provided for OneGen in the references is incorrect. We thank the reviewer for pointing this out. We will correct it.

AuthorsOpenReview ↗

Rebuttal follow-up

Dear Reviewer,
Thank you again for your detailed review. As the discussion period nears its end, a brief reminder of our response:

  1. We addressed the reviewer’s concerns about the framing of dynamic perplexity and will update Appendix A.6 accordingly.
  2. Figure 2 is the correct reading of Section 3.2, and we will revise Section 3.2 to make that clear.
  3. We added a human evaluation of annotation quality on Wikipedia and FineWeb-Edu.

We would be glad to address any remaining concerns. If you feel our response addressed them, we would be grateful if you would consider updating your assessment.
Thank you for your time.

Reviewer_d3x3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper propose CO-LMLM. It provides an improvement over the previous paper "Pre-training limited memory language models with internal and external knowledge, 2026" The key difference is the vector retrieval queries that CO-LMLM utilized. The older method used textual queries, and use a wiki-style subject-relation-obj tuples. The author claims this vector retrieval method is better. I will raise questions about this in later sections.

Critical strengths

The idea presentation is very clear. The difference comparing with previous method is easy to understand. The testing methodology is also easy to follow. And its benefits are clearly shown.

Critical weaknesses

In the experiments, the author mentioned: " We create a KB from Wikipedia only for a direct comparison with the REL-LMLM KB that is constructed from this same corpus. It also guarantees that the KB does not cover information unavailable to the baselines. This the default KB, unless another is specified. The CO-LMLM Wikipedia KB includes ∼240M items, compared to REL-LMLM’s ∼145M"

This seems to indicate the improvement might also come from the KB size.

Questions for authors

As I mentioned in the Critical Weaknesses section, I'm not sure how to rule out that the better perplexity isn't related to the KB size, aka, externalization step. It would be ideal to have an experiment to demonstrate that. If it is also part of the improvement, maybe discuss further about how much this helps.

And a side note, in the abstract, KB was directly used, maybe change to knowledge base (KB), as it is the first place mentioned it in the paper.

3 comments

We thank the reviewer for their thoughtful comments.

We would like to address the following concern raised by the reviewer: Co-LMLM's Wikipedia KB has about 240M items versus about 145M for Rel-LMLM, so the gains might come from KB size rather than from the method.

Co-LMLM differs from Rel-LMLM in two ways: (i) what counts as a fact (free-form spans instead of relational triplets) and (ii) how facts are queried (a continuous query read from the hidden state instead of a decoded text query). The larger KB is a direct consequence of (i): both KBs are built from the same Wikipedia corpus, and Co-LMLM externalizes more of it because its facts are not limited to relations. Both (i) and (ii) are contributions of Co-LMLM, and the increase of the KB size is a method outcome, not a mere confounder.

Nevertheless, we agree that it is important to separate the significance of each contribution. To separate the two, we evaluated LMLM-Asker (Section 4.6) trained on Wikipedia at both model sizes. Asker is trained on Co-LMLM's annotated corpus and retrieves the same fact spans, but queries in a way methodologically closer to Rel-LMLM: it decodes a text question and embeds it with the same sentence encoder. This allows an approximation of the difference between LMLM-Asker and Rel-LMLM as attributed to the richness of the KB. The difference between Co-LMLM and LMLM-Asker can be attributed to the querying mechanism. Factuality results are given below (parentheses give the change from the row above):

Model (Wikipedia)TriviaQAPopQAT-RExSimpleQAFactScore
Rel-LMLM-135M (relational facts, text query)13.427.242.96.223.1
LMLM-Asker-135M (Co-LMLM facts, text query)24.6 (+11.2)46.8 (+19.6)31.5 (−11.4)10.5 (+4.3)37.9 (+14.8)
Co-LMLM-135M (Co-LMLM facts, continuous query)28.4 (+3.8)47.3 (+0.5)40.5 (+9.0)18.2 (+7.7)35.0 (−2.9)
Rel-LMLM-360M (relational facts, text query)15.626.647.45.622.9
LMLM-Asker-360M (Co-LMLM facts, text query)23.1 (+7.5)43.5 (+16.9)39.1 (−8.3)8.1 (+2.5)30.7 (+7.8)
Co-LMLM-360M (Co-LMLM facts, continuous query)31.4 (+8.3)46.5 (+3.0)47.4 (+8.3)21.2 (+13.1)34.2 (+3.5)

Both contributions are important: on Factual QA, LMLM-Asker significantly improves over Rel-LMLM, and Co-LMLM improves significantly over LMLM-Asker. For T-REx, which is a knowledge completion task for knowledge triplets, Rel-LMLM is better suited than LMLM-Asker, but Co-LMLM’s querying mechanism proves to be comparable to Rel-LMLM even when the task is perfectly tailored for Rel-LMLM’s factual definition. The gains over Rel-LMLM therefore come from both changes, and the larger KB pays off fully only with the continuous query.

Co-LMLM not only improves on raw benchmark scores, it is also much more efficient at inference time (see Table 16).

Finally, a correction to the LMLM-Asker results in Table 16: while running these evaluations we found that the reported LMLM-Asker-360M-FW results were outdated. The correct scores are 32.5 / 50.1 / 47.1 / 18.6 / 33.7 on TriviaQA / PopQA / T-REx / SimpleQA / FactScore (Table 16: 15.4 / 43.2 / 38.5 / 7.5 / 25.0; Co-LMLM-360M-FW: 36.9 / 50.6 / 54.5 / 21.7 / 33.3). The updated results simply show a stronger LMLM-Asker baseline, which Co-LMLM still significantly improves upon, so no conclusions are affected.

As for the side note, we thank the reviewer for pointing that out and we will correct it to “knowledge base (KB)”.

AuthorsOpenReview ↗

Rebuttal follow-up

Dear Reviewer,

Thank you again for your review. As the discussion period nears its end, a brief reminder of our response to your question: the larger KB is a result of Co-LMLM's broader fact definition, and our new LMLM-Asker comparison shows that both this and Co-LMLM's continuous query mechanism contribute to the gains over Rel-LMLM.

We would be glad to address any remaining concerns. If you feel our response addressed them, we would be grateful if you would consider updating your assessment. Thank you for your time.

Thanks for the followup, I've read your response when it was posted. It helped me to better understand and I decided to keep my score for now.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.