- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes OBP, a pretraining framework that dynamically offloads tokens to an external knowledge base (KB) based on measured retrieval information gain, and trains a classifier to decide when to retrieve at inference. Experiments with GPT2-124M/355M and SmolLM2-135M/360M on KILT Wikipedia show lower open-book perplexity and higher QA scores than a standard LM + RAG baseline.
Critical strengths
The idea of learning at pretraining time which facts to memorize versus offload is conceptually appealing. The information-gain criterion is model-adaptive, and the framework does not require human fact-span annotations. The classifier enables dynamic retrieval at inference.
Critical weaknesses
-
KB is same-source with pretraining/evaluation, limiting external-validity claims. The KB is extracted with Qwen3-4B-Instruct from the same KILT Wikipedia snapshot used for training and for gold provenance in evaluation (§4.2). The KB covers both training and held-out articles. Although retrieval uses an external store, this setup cannot validate the paper’s stated motivations—updating time-sensitive knowledge, keeping private information out of parameters, or adapting to new domains. It also cannot show that OBP works with a genuinely disjoint or evolving KB. KB-update, domain-specific, or private-KB experiments are needed.
-
Limited analysis of forgetting/degradation. OBP masks offloaded tokens from the plain LM objective (§3.2). The paper reports overall closed-book perplexity within 0.6 of baseline (Table 2) and fact-token perplexity without retrieval (Table 10), but does not isolate offloaded tokens, measure reversibility, or test downstream degradation beyond QA. Since offloading is the core mechanism, this is a critical missing analysis.
-
Efficiency is not quantified. The Discussion states that offloading training is “several times slower” and reports inference retrievals per sequence, but provides no wall-clock time, FLOPs, memory, or KB construction cost. For a pretraining method, this is a major omission.
-
Limited scale and missing comparisons to closest methods/heuristics. Models are at most 360M parameters and trained for one epoch. The only baselines are plain LM and plain LM + RAG. The paper does not compare its learned retrieval classifier against simple heuristics (fixed interval, entropy threshold) or against the closest offloading/retrieval-training method, LMLM. Comparisons to inference-time dynamic retrieval methods (e.g., Self-RAG, FLARE) are not fully apples-to-apples, but should be discussed or controlled. Without such comparisons, the value of the learned policy remains unclear.
Questions for authors
I encourage the authors to further clarify the above concerns and discuss how they plan to address these limitations in future revisions.
4 comments
The authors have addressed some of my concerns. I have updated my score.
Thank you for replying to our previous rebuttal and adjusting your score. As our experimental results come out, we attempt to address these remaining concerns:
KB is same-source with pretraining/evaluation, limiting external-validity claims. The KB is extracted with Qwen3-4B-Instruct from the same KILT Wikipedia snapshot used for training and for gold provenance in evaluation (§4.2). The KB covers both training and held-out articles. Although retrieval uses an external store, this setup cannot validate the paper’s stated motivations—updating time-sensitive knowledge, keeping private information out of parameters, or adapting to new domains. It also cannot show that OBP works with a genuinely disjoint or evolving KB. KB-update, domain-specific, or private-KB experiments are needed.
Knowledge base and efficiency: Two broader concerns remain. Since the KB includes facts extracted from held-out articles, the current experiments do not establish generalization to genuinely new or evolving knowledge sources. The paper should clarify this limitation and narrow the corresponding claims.
We experimented with training our models on different KBs, and evaluating these on different held-out splits. Our new results show that our method can generalize to different knowledge bases and tasks.
(a) Evaluation on Simple Wikipedia and SciQ
We selected two different tasks to test our models trained with default KB. We randomly draw 1,000 pages from Simple Wikipedia as a held-out split, and use the official held-out split of SciQ with 1,000 questions. We report the evaluation results below.
Simple Wikipedia:
Since our default KB doesn't extract entries from Simple Wikipedia, top-5 retrieval returns little useful evidence. Thus, retrievals hurt performance across all models. However, our models maintain similar performance to the baseline, and are hurt less by irrelevant retrieved entries in three out of four models.
| Model | Setting | Closed-book PPL | Always retrieve PPL | Selective retrieval PPL |
|---|---|---|---|---|
| GPT2-124M | Baseline | 15.7 | 18.9 | - |
| OBP | 16.3 | 17.1 | 16.9 | |
| GPT2-355M | Baseline | 12.9 | 15.0 | - |
| OBP | 13.3 | 13.8 | 13.7 | |
| SmolLM2-135M | Baseline | 24.8 | 25.5 | - |
| OBP | 24.7 | 25.9 | 25.7 | |
| SmolLM2-360M | Baseline | 12.6 | 13.4 | - |
| OBP | 13.0 | 13.1 | 13.1 |
SciQ:
We use the released support paragraph for each question as retrieved evidence. OBP is better able to use the external evidence, i.e. it has larger improvement from closed- to open-book than the baseline models.
| Model | Setting | Closed-book accuracy (%) | Open-book accuracy (%) | Gain (Open - Closed) |
|---|---|---|---|---|
| GPT2-124M | Baseline | 34.3 | 66.1 | 31.8 |
| OBP | 30.7 | 70.4 | 39.7 | |
| GPT2-355M | Baseline | 36.6 | 69.7 | 33.1 |
| OBP | 38.4 | 75.4 | 37.0 | |
| SmolLM2-135M | Baseline | 25.4 | 41.1 | 15.7 |
| OBP | 24.6 | 48.4 | 23.8 | |
| SmolLM2-360M | Baseline | 34.8 | 54.0 | 19.2 |
| OBP | 31.7 | 57.1 | 25.4 |
(b) Training with a different KB
In addition, we trained GPT-124M (single seed) with a disjoint KB not extracted from the training corpus. We trained on Simple Wikipedia for 20 epochs, and used the original KILT wikipedia snapshot as the knowledge base. Our perplexity results below show that, even when trained on simpler text, OBP models have the capability to understand more difficult text from a disjoint KB. Note that the closed-book performance on baseline here is worse due to possible overfitting.
| Model | Closed-book PPL | Always Retrieve PPL | Selective Retrieval PPL |
|---|---|---|---|
| Baseline | 18.2 | 16.8 | – |
| OBP | 17.1 | 11.7 | 11.6 |
We hope these results address your remaining concerns, and we plan to include them in our next revision.
Thank you for the detailed review. Before we add new experiments, we would like to ask a few clarification questions.
Figure 2 reports perplexity separately on the tokens the model offloaded and on the remaining tokens. What additional analysis would you consider necessary to "isolate offloaded tokens"? Could you also clarify what you mean by "measure reversibility" and "test downstream degradation beyond QA"?
We'll be adding the fixed-interval and entropy-threshold comparisons, using the same trained model at matched numbers of retrievals. Table 4 already reports LMLM’s accuracy on our QA tasks. While we can add FLARE as a comparison, a direct comparison of retrieval decisions with LMLM and Self-RAG is harder. LMLM and Self-RAG are trained on corpora with annotated retrievals and trigger retrieval by generating special tokens during inference. What kind of discussion/control would match what you have in mind?