- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Co-LMLM, a method that stores factual knowledge in an external knowledge base while training a language model to generate text conditioned on retrieved information during pretraining. The approach annotates factual spans and their corresponding questions, then applies bidirectional contrastive learning to align the hidden states at the <FACT> position in the text and the <FACT-q> position in the question. At inference time, the model uses these hidden states to query an external index, retrieves free-text factual fragments, and continues generation based on the retrieved content. Extensive experiments demonstrate substantial improvements in factual question answering while maintaining language understanding performance comparable to that of general-purpose language models in certain settings.
Critical strengths
-
The paper addresses an important problem, and the motivation and methodological design are clear. Free-text factual fragments overcome the representational limitations of relational triples, while queries derived directly from hidden states reduce the computational overhead of explicit query generation. In addition, the training format encourages the model to use external information during generation. Together, these design choices form a coherent and well-motivated approach.
-
The gains in factual accuracy over the baseline are substantial. Co-LMLM notably outperforms REL-LMLM in both short- and long-form generation under the multitask setting, providing strong evidence of the method’s effectiveness.
Critical weaknesses
-
Its applicability to larger models remains to be verified. The paper demonstrates the effectiveness of Co-LMLM on 135M and 360M models, but these results are insufficient to determine whether its benefits extend to larger models. Although the authors acknowledge this limitation, further validation at larger scales would strengthen the case for the method’s scalability.
-
The overall cost assessment and comparison with recent memory-based methods remain insufficient. Co-LMLM incurs additional costs for corpus annotation, question generation, and training, making it difficult to assess the trade-off between its overall performance gains and resource requirements. Furthermore, the experimental comparisons focus primarily on REL-LMLM and LMLM-ASKER. Comparisons with recent representative methods, such as an adapted Engram, under comparable resource budgets—or further discussion of their comparability—would help clarify the advantages of the proposed method.
Questions for authors
-
Based on the results for the 135M and 360M models, are there any indications that the relative benefits of knowledge externalization vary with model size? If preliminary experiments with larger models are available, could the authors provide further details?
-
Could the authors further quantify the overhead associated with corpus annotation and question generation, and report inference costs including retrieval, to help assess the trade-off between overall performance gains and resource requirements?
-
Have the authors considered experimental comparisons with recent methods such as Engram under comparable resource budgets? If such comparisons pose substantial adaptation challenges, could the authors explain these challenges and clarify which comparative conclusions the current experiments support?
2 comments
Rebuttal follow-up
Dear Reviewer,
Thank you again for your thoughtful review. As the discussion period nears its end, a brief reminder of our response:
- Building scaling laws is important, and this is an ongoing research direction for us. Current results show Co-LMLM's advantage over the Standard LM is stable from 135M to 360M.
- We reported annotation, training and inference costs.
- We explained why Engram, which stores knowledge in the model's weights and has no released models or training code, is not directly comparable.
We would be glad to address any remaining concerns. If you feel our response addressed them, we would be grateful if you would consider updating your assessment.
Thank you for your time.
We thank the reviewer for their thoughtful comments. We would like to address the reviewer’s questions:
Q1: Based on the results for the 135M and 360M models, are there any indications that the relative benefits of knowledge externalization vary with model size? If preliminary experiments with larger models are available, could the authors provide further details?
We agree that building scaling laws is important, and this is an ongoing research direction for us. In addition to the complexities of building scaling ladders in conventional LLMs, CoLMLM also grows along the KB. A new method of building scaling ladders is needed. Borgeaud et al., 2022 and Shao et al., 2024 provide a base on which we are building. Overall, this is the right way to go, and it’s a new project on its own. In our current results, Co-LMLM’s advantage over the Standard LM is stable from 135M to 360M (Table 1), e.g. +22.0 / +23.9 on TriviaQA, +32.1 / +30.3 on PopQA, +18.1 / +20.9 on SimpleQA.
Q2: Could the authors further quantify the overhead associated with corpus annotation and question generation, and report inference costs including retrieval, to help assess the trade-off between overall performance gains and resource requirements?
We will extend the discussion of this in the paper. As you review the numbers below, please keep in mind that our implementation and cluster are not optimized. Annotation and training measured on one NVIDIA B200 GPU (tokens are SmolLM2 tokens):
Annotation is paid once per corpus. Question generation is 80-90% of it, and its cost per token grows with fact density (59 facts per 1K tokens on Wikipedia, 19 on FineWeb-Edu). The compute share of the annotation process relative to the training process will decrease as we scale our models.
Training currently runs at 1.8 times the GPU time per token compared to a standard LLM. Each fact’s paired question goes forward and backward through the model (overall, 39% more FLOPs), and the contrastive objective lowers utilization.
Inference is measured with the entire Wikipedia index searched on the same GPU using Faiss. At batch 1, Co-LMLM-360M takes 1.27 ms per token against 1.05 ms for the Standard LM. This overhead is driven by the extra tokens CoLMLM processes in the retrieval block, and by the index search.
While Co-LMLM is more expensive than standard LMs when comparing compute per token, it allows for entirely new capabilities and opens up a new avenue for developing language models where the user has direct control and observability over the model’s factual knowledge.
Q3: Have the authors considered experimental comparisons with recent methods such as Engram under comparable resource budgets? If such comparisons pose substantial adaptation challenges, could the authors explain these challenges and clarify which comparative conclusions the current experiments support?
Engram is fundamentally different from Co-LMLM in that it still stores knowledge in the LM’s weights, while Co-LMLM keeps it externalized as readable text. Engram’s entries are learned vectors, which cannot be read, attributed to a source, or edited directly. The two methods therefore serve substantially different purposes: Engram raises parametric capacity at fixed compute, while Co-LMLM externalizes knowledge in a controllable manner. Engram could in principle serve as a stronger parametric baseline than our Standard LM. However, no Engram model has been released at any size, and the smallest model reported in the paper is a 3B MoE. Furthermore, training code was not released. The official Engram repo contains only a demonstration script missing the implementations of several components and mechanisms. A comparison would therefore require re-implementing Engram from the paper at a scale it was never tested at, and would measure our reimplementation rather than Engram itself.