acceptodds
Under review as a conference paper at ICLR 2027

Open-Book Pretraining for Language Models: Selective Memorization with Dynamic Retrieval

24 views

Abstract

Language models use their limited parametric memory to store facts that could instead be retrieved from an external store. We ask whether a model can decide, during pretraining, which facts to memorize and which to leave to a knowledge base (KB). We introduce Open-Book Pretraining (OBP), a pretraining method that makes such decisions dynamically by identifying positions where retrieved context reduces next-token loss, according to the current state of the model. Along with the pretrained language model, we learn a lightweight classifier that decides when to retrieve during inference. Using this approach, we train small language models (0.5B parameters) from scratch on Wikipedia. Evaluated with retrieval on held-out pages, a 360M OBP model reaches a lower perplexity than a 1.7B model trained on the same data, while retaining competitive perplexity even without retrieval. Relative to standard pretraining plus retrieval-augmented generation, OBP models use retrieved evidence better while memorizing fewer facts in their parameters. Our learned classifier is able to recognize when to retrieve during inference, adapting to different sizes of knowledge base. To our knowledge, OBP is the first framework that dynamically selects between parametric memory and external retrieval during pretraining, based on model- and KB-specific retrieval gains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Retrieval-Augmented Language Models, Memorization in Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9mmkOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes OBP, a pretraining framework that dynamically offloads tokens to an external knowledge base (KB) based on measured retrieval information gain, and trains a classifier to decide when to retrieve at inference. Experiments with GPT2-124M/355M and SmolLM2-135M/360M on KILT Wikipedia show lower open-book perplexity and higher QA scores than a standard LM + RAG baseline.

Critical strengths

The idea of learning at pretraining time which facts to memorize versus offload is conceptually appealing. The information-gain criterion is model-adaptive, and the framework does not require human fact-span annotations. The classifier enables dynamic retrieval at inference.

Critical weaknesses
  1. KB is same-source with pretraining/evaluation, limiting external-validity claims. The KB is extracted with Qwen3-4B-Instruct from the same KILT Wikipedia snapshot used for training and for gold provenance in evaluation (§4.2). The KB covers both training and held-out articles. Although retrieval uses an external store, this setup cannot validate the paper’s stated motivations—updating time-sensitive knowledge, keeping private information out of parameters, or adapting to new domains. It also cannot show that OBP works with a genuinely disjoint or evolving KB. KB-update, domain-specific, or private-KB experiments are needed.

  2. Limited analysis of forgetting/degradation. OBP masks offloaded tokens from the plain LM objective (§3.2). The paper reports overall closed-book perplexity within 0.6 of baseline (Table 2) and fact-token perplexity without retrieval (Table 10), but does not isolate offloaded tokens, measure reversibility, or test downstream degradation beyond QA. Since offloading is the core mechanism, this is a critical missing analysis.

  3. Efficiency is not quantified. The Discussion states that offloading training is “several times slower” and reports inference retrievals per sequence, but provides no wall-clock time, FLOPs, memory, or KB construction cost. For a pretraining method, this is a major omission.

  4. Limited scale and missing comparisons to closest methods/heuristics. Models are at most 360M parameters and trained for one epoch. The only baselines are plain LM and plain LM + RAG. The paper does not compare its learned retrieval classifier against simple heuristics (fixed interval, entropy threshold) or against the closest offloading/retrieval-training method, LMLM. Comparisons to inference-time dynamic retrieval methods (e.g., Self-RAG, FLARE) are not fully apples-to-apples, but should be discussed or controlled. Without such comparisons, the value of the learned policy remains unclear.

Questions for authors

I encourage the authors to further clarify the above concerns and discuss how they plan to address these limitations in future revisions.

4 comments

Thank you for the detailed review. Before we add new experiments, we would like to ask a few clarification questions.

"Limited analysis of forgetting/degradation. OBP… does not isolate offloaded tokens, measure reversibility, or test downstream degradation beyond QA. Since offloading is the core mechanism, this is a critical missing analysis."

Figure 2 reports perplexity separately on the tokens the model offloaded and on the remaining tokens. What additional analysis would you consider necessary to "isolate offloaded tokens"? Could you also clarify what you mean by "measure reversibility" and "test downstream degradation beyond QA"?

"The paper does not compare its learned retrieval classifier against simple heuristics (fixed interval, entropy threshold) or against the closest offloading/retrieval-training method, LMLM. Comparisons to inference-time dynamic retrieval methods (e.g., Self-RAG, FLARE) are not fully apples-to-apples, but should be discussed or controlled."

We'll be adding the fixed-interval and entropy-threshold comparisons, using the same trained model at matched numbers of retrievals. Table 4 already reports LMLM’s accuracy on our QA tasks. While we can add FLARE as a comparison, a direct comparison of retrieval decisions with LMLM and Self-RAG is harder. LMLM and Self-RAG are trained on corpora with annotated retrievals and trigger retrieval by generating special tokens during inference. What kind of discussion/control would match what you have in mind?

The authors have addressed some of my concerns. I have updated my score.

Thank you for replying to our previous rebuttal and adjusting your score. As our experimental results come out, we attempt to address these remaining concerns:

KB is same-source with pretraining/evaluation, limiting external-validity claims. The KB is extracted with Qwen3-4B-Instruct from the same KILT Wikipedia snapshot used for training and for gold provenance in evaluation (§4.2). The KB covers both training and held-out articles. Although retrieval uses an external store, this setup cannot validate the paper’s stated motivations—updating time-sensitive knowledge, keeping private information out of parameters, or adapting to new domains. It also cannot show that OBP works with a genuinely disjoint or evolving KB. KB-update, domain-specific, or private-KB experiments are needed.

Knowledge base and efficiency: Two broader concerns remain. Since the KB includes facts extracted from held-out articles, the current experiments do not establish generalization to genuinely new or evolving knowledge sources. The paper should clarify this limitation and narrow the corresponding claims.

We experimented with training our models on different KBs, and evaluating these on different held-out splits. Our new results show that our method can generalize to different knowledge bases and tasks.

(a) Evaluation on Simple Wikipedia and SciQ

We selected two different tasks to test our models trained with default KB. We randomly draw 1,000 pages from Simple Wikipedia as a held-out split, and use the official held-out split of SciQ with 1,000 questions. We report the evaluation results below.

Simple Wikipedia:

Since our default KB doesn't extract entries from Simple Wikipedia, top-5 retrieval returns little useful evidence. Thus, retrievals hurt performance across all models. However, our models maintain similar performance to the baseline, and are hurt less by irrelevant retrieved entries in three out of four models.

ModelSettingClosed-book PPLAlways retrieve PPLSelective retrieval PPL
GPT2-124MBaseline15.718.9-
OBP16.317.116.9
GPT2-355MBaseline12.915.0-
OBP13.313.813.7
SmolLM2-135MBaseline24.825.5-
OBP24.725.925.7
SmolLM2-360MBaseline12.613.4-
OBP13.013.113.1

SciQ:

We use the released support paragraph for each question as retrieved evidence. OBP is better able to use the external evidence, i.e. it has larger improvement from closed- to open-book than the baseline models.

ModelSettingClosed-book accuracy (%)Open-book accuracy (%)Gain (Open - Closed)
GPT2-124MBaseline34.366.131.8
OBP30.770.439.7
GPT2-355MBaseline36.669.733.1
OBP38.475.437.0
SmolLM2-135MBaseline25.441.115.7
OBP24.648.423.8
SmolLM2-360MBaseline34.854.019.2
OBP31.757.125.4

(b) Training with a different KB

In addition, we trained GPT-124M (single seed) with a disjoint KB not extracted from the training corpus. We trained on Simple Wikipedia for 20 epochs, and used the original KILT wikipedia snapshot as the knowledge base. Our perplexity results below show that, even when trained on simpler text, OBP models have the capability to understand more difficult text from a disjoint KB. Note that the closed-book performance on baseline here is worse due to possible overfitting.

ModelClosed-book PPLAlways Retrieve PPLSelective Retrieval PPL
Baseline18.216.8–
OBP17.111.711.6

We hope these results address your remaining concerns, and we plan to include them in our next revision.

Reviewer_okraOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes open-book pretraining. For each token in a 10-token window it measures how much facts raise the token's log-probability. Over a threshold, they are removed from the language-modeling loss and trained only with the facts in context with 20 times the weight. A two-layer MLP on the hidden state learns to predict such positions and triggers retrieval at inference. The authors show that OBP reaches lower perplexity than the same architecture trained normally and then given retrieval, and higher zero-shot QA accuracy when evidence is in the prompt. Without retrieval it is slightly worse, which the authors interpret as reduced memorization.

Critical strengths

The core idea is nice. Instead of annotating fact spans up front, as LMLM and Co-LMLM do, the model itself decides what to offload, based on how much retrieval actually helps it at that point in training. If the claims hold up, I can see others picking this up, so this is the main thing that speaks for acceptance. The setup is also careful within its scope. OBP and the baseline share architecture and data, there are four models with three seeds each, and the authors are upfront about some limitations (e.g. the caption of Table 4, the training cost in Sec. 5). Most of my concerns below can be addressed with extra controls in this same setup, which is why I don't see this as a clear reject.

Critical weaknesses

My main concern is that the experiments do not show that offloading is what makes the difference. Compared to the baseline, OBP changes two things at once: the model sees retrieved facts in its context during training and the offloaded tokens are removed from the plain LM loss. Every result with retrieval (Fig. 2, Tables 3 and 4) could come from the first change alone. To me the obvious control is missing, i.e. training on the same retrieved contexts with the same loss weights but keeping all tokens in the plain loss.

The comparison in the headline doesn't help with the 360M vs. 1.7B result in the abstract, they compare a model with retrieval to one without, and the KB also contains facts extracted from the held-out test pages (l. 302-303). The "about 3x better" on masked tokens compares different token sets, since each model's masked set is defined by where retrieval helped that particular model (Fig. 2 caption). The big QA gains in Table 3 are zero-shot against a baseline that never saw that evidence in its input during training. Once both models are fine-tuned (Table 11 in the appendix), the gap is mostly 0-2 points. On the memorization side, only 0.4-0.5% of training tokens are ever offloaded, closed-book perplexity on fact tokens goes up by 4-6% (not at all for SmolLM2-135M) and no seed variance is reported. From this I can't tell whether OBP stores fewer facts or is simply a slightly weaker LM.

This is the main reason for my diminished score. The central claim of the paper, selective memorization through adaptive offloading, is not yet backed by the experiments.

Given that this technique is novel to me, I was questioning what we actually learn from the paper. The intro gives several reasons to memorize less (smaller models, updating knowledge, privacy, new domains), but the paper never commits to one of them and none is tested. For example, no fact is ever edited or deleted, although the conclusion talks about knowledge that "can be inspected and modified". So, a model trained open-book does better on open-book tests than a model that wasn't, which is pretty much what I would expect - maybe I miss a point here. The more interesting question is the one the authors leave open in Sec. 5: how little does a model need to memorize to keep comparable general performance? To me that is the question a strong paper on this topic has to answer, and OBP looks like a good tool for it.

Overall, I am struggling with the compositional manner of the approach. LMLM and Co-LMLM mask fact spans from the loss, Toolformer uses a loss-based criterion to decide where external information helps and Self-RAG learns when to retrieve. Combining them can be a contribution, but the intro and Table 1 make it look like more. The two columns where only OBP gets a tick are defined by OBP's own features and Co-LMLM and Self-RAG are missing from the table (Self-RAG's retrieval threshold can be changed at inference, which is exactly "dynamic retrieval"). The "first framework" claim only holds with all its qualifiers.

I also don't think "annotation free" is accurate. The KB is built by running Qwen3-4B over every paragraph of the training corpus, and the appendix itself calls this labelling and annotation. This means Qwen3-4B's extraction decides what can be offloaded at all. It also defines the fact tokens in Table 10 and the gold evidence in the QA experiments.

This is why I would stay below a clear accept even if the first point were fixed. The contribution would still be a combination of known parts that produces an expected result.

Questions for authors
  1. can you add the control from weakness (1), i.e. a model trained on the same retrieved contexts and loss weights, with no tokens removed from the plain loss? One model size would be enough. If OBP still clearly wins in Fig. 2 and Table 3 my main concern goes away.
  2. How does SmolLM2-1.7B do when it also gets retrieval? And what does Fig. 2 look like if the KB entries extracted from the held-out pages are removed?
  3. In what sense is the method annotation free, given that every paragraph of the training corpus goes through Qwen3-4B? And how much compute did building the KB take?
5 comments

Dear reviewer,

Thank you for the detailed and thoughtful questions, and for your positive assessment of our work. We try to address each of your concerns below:

My main concern is that the experiments do not show that offloading is what makes the difference. Compared to the baseline, OBP changes two things at once: the model sees retrieved facts in its context during training and the offloaded tokens are removed from the plain LM loss. Every result with retrieval (Fig. 2, Tables 3 and 4) could come from the first change alone. To me the obvious control is missing, i.e. training on the same retrieved contexts with the same loss weights but keeping all tokens in the plain loss.

Can you add the control from weakness (1), i.e. a model trained on the same retrieved contexts and loss weights, with no tokens removed from the plain loss? One model size would be enough. If OBP still clearly wins in Fig. 2 and Table 3 my main concern goes away.

We have added the no-offload control that you suggested. All results are on GPT2-124M with single seed. From the numbers, no-offload OBP memorizes more offloaded facts than OBP, and has similar or slightly worse performance with retrieval. We first report closed-book and open-book perplexity on held-out splits:

ModelClosed-book PPLAlways retrieve PPLSelective retrieval PPL
Baseline13.212.3–
OBP13.710.310.2
No-offload OBP13.710.410.2

For offloaded facts, we logged offloaded tokens every 200 steps from step 15k, amounting to about 63k offloaded tokens per seed. We report mean perplexity on these offloaded tokens versus bystander tokens and plain tokens. We also note that the high perplexity for offloaded tokens is expected, since these are the most fact-rich and therefore difficult tokens to predict.

Set of TokensAvg. PPL of BaselineAvg. PPL of OBPAvg. PPL of No-offload OBP
Offloaded tokens70.076.774.3
Bystander tokens in the same sampled spans13.413.813.8
Plain tokens in the same logged sequences14.314.914.9

The comparison in the headline doesn't help with the 360M vs. 1.7B result in the abstract, they compare a model with retrieval to one without, and the KB also contains facts extracted from the held-out test pages (l. 302-303). The "about 3x better" on masked tokens compares different token sets, since each model's masked set is defined by where retrieval helped that particular model (Fig. 2 caption). The big QA gains in Table 3 are zero-shot against a baseline that never saw that evidence in its input during training. Once both models are fine-tuned (Table 11 in the appendix), the gap is mostly 0-2 points.

How does SmolLM2-1.7B do when it also gets retrieval?

In RAG-related literature, it is commonly adopted in prior work (RETRO, Atlas, etc.) to report a model with retrieval to a larger baseline without retrieval. Nevertheless, to address your question, we list SmolLM2-1.7B results here, together with other results from Figure 2. These numbers show that SmolLM2-360M OBP is slightly worse than the SmolLM2-1.7B baseline, but has increased the gap between closed-book and open-book more:

ModelClosed-book PPLAlways retrieve PPLSelective retrieval PPL
GPT2-1.5B9.08.8-
GPT2-355M OBP11.28.58.4
SmolLM2-1.7B8.77.8-
SmolLM2-360M OBP11.38.58.4

We would also like to clarify about the fine-tuning results in the appendix. Fine-tuning on a QA training split does not only teach the model the answer format and how to use retrieved passages; it can also teach it facts that the evaluation split asks about. The training and evaluation splits of these benchmarks overlap substantially: Lewis et al. (2021) find that 60%-70% of test answers in NQ and TriviaQA also appear as answers in the training split. Fine-tuned accuracy therefore mixes retrieval use with answer memorization. For this reason, the main text reports zero-shot results following community standards, and the appendix gives fine-tuned results for completeness.

And what does Fig. 2 look like if the KB entries extracted from the held-out pages are removed?

If KB entries extracted from the held-out pages are simply removed, there may not be useful information to retrieve in the remaining entries, since there is insufficient factual overlap between Wikipedia pages on different topics. Thus, we instead address your question by testing on different downstream tasks that don't contain verbatim information from these KB entries on KILT held-out split. We test on both Simple Wikipedia (1,000 pages, selected randomly) and the SciQ held-out split.

Since our default KB doesn't extract entries from Simple Wikipedia, top-5 retrieval returns little useful evidence. Thus, retrievals hurt performance across all models. However, our models maintain similar performance to the baseline, and are hurt less by irrelevant retrieved entries in three out of four models.

Evaluation results on Simple Wikipedia:

ModelSettingClosed-book PPLAlways retrieve PPLSelective retrieval PPL
GPT2-124MBaseline15.718.9-
OBP16.317.116.9
GPT2-355MBaseline12.915.0-
OBP13.313.813.7
SmolLM2-135MBaseline24.825.5-
OBP24.725.925.7
SmolLM2-360MBaseline12.613.4-
OBP13.013.113.1

For SciQ, we use the released support paragraph for each question as retrieved evidence. OBP is better able to use the external evidence, i.e. it has larger improvement from closed- to open-book than the baseline models.

Evaluation results on SciQ:

ModelSettingClosed-book accuracy (%)Open-book accuracy (%)Gain (Open - Closed)
GPT2-124MBaseline34.366.131.8
OBP30.770.439.7
GPT2-355MBaseline36.669.733.1
OBP38.475.437.0
SmolLM2-135MBaseline25.441.115.7
OBP24.648.423.8
SmolLM2-360MBaseline34.854.019.2
OBP31.757.125.4
Reviewer_YLZZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Open-Book Pretraining (OBP), which learns during pretraining which knowledge to memorize and which to offload to an external knowledge base (KB). For each token, OBP measures how much retrieval reduces the model’s loss, and tokens with a large gain are trained only with the retrieved evidence in context. A small classifier is also trained to decide when to retrieve at test time. Experiments on GPT-2 and SmolLM2 show that OBP makes better use of retrieved documents than a RAG baseline, and that a small OBP model can outperform a much larger baseline when retrieval is available.

Critical strengths
  1. The idea of letting the model itself decide what to memorize and what to look up, based on how much retrieval actually helps the current model, is natural and well motivated. It removes the need for costly fact annotation and adapts automatically to model size, training stage, and KB, which I find more principled than the fixed offloading used in prior work such as LMLM.

  2. The empirical results on retrieval use are convincing. OBP makes clearly better use of the same retrieved documents than a standard model with RAG, and the learned classifier lets one adjust how often to retrieve at inference without retraining. That said, the evidence for reduced memorization itself is weaker than the retrieval gains.

Critical weaknesses
  1. Weak evidence for selective memorization. The retrieval gains are clear, but the drop in memorization is small and inconsistent across models. It also largely disappears after finetuning, so the paper’s central claim is not well supported.

  2. Many hand-crafted choices. Span selection, template positions, content-word filters, and loss weights are all fixed by hand. This weakens the claim that the model itself decides what to offload.

Questions for authors
  1. Is there more direct evidence that OBP memorizes less, beyond closed-book accuracy and fact-token perplexity? For example, a controlled test on facts that were offloaded versus not, or a knowledge-update experiment where only the KB is changed.

  2. How does OBP compare with LMLM or a RETRO-style model trained on the same data and KB? If those methods reach similar retrieval gains, the benefit of OBP’s dynamic, model-dependent offloading would be less clear.

6 comments

"Is there more direct evidence that OBP memorizes less, beyond closed-book accuracy and fact-token perplexity? For example, a controlled test on facts that were offloaded versus not, or a knowledge-update experiment where only the KB is changed."

Thank you for the detailed review. Before we add new experiments, we would like to understand what kind of controlled test on offloaded versus non-offloaded facts you have in mind, beyond the perplexity on masked and non-masked tokens that we report in Figure 2?

Dear reviewer,

Thank you for the detailed and thoughtful review. We attempt to address your questions and concerns below:

Weak evidence for selective memorization. The retrieval gains are clear, but the drop in memorization is small and inconsistent across models. It also largely disappears after finetuning, so the paper’s central claim is not well supported…

Is there more direct evidence that OBP memorizes less, beyond closed-book accuracy and fact-token perplexity? For example, a controlled test on facts that were offloaded versus not, or a knowledge-update experiment where only the KB is changed.

We agree that the differences between models in our results are limited by the current scale of our model and data. To better measure the reduced memorization in OBP, we ran additional experiments:

(a) Higher perplexity gap between offloaded tokens and non-offloaded tokens

During training, we logged offloaded tokens every 200 steps from step 15k, amounting to about 63k offloaded tokens per seed. We report mean perplexity on these offloaded tokens versus bystander tokens and plain tokens, for the final baseline and final OBP models of GPT2-124M:

Set of TokensAvg. PPL of BaselineAvg. PPL of OBP
Offloaded tokens70.076.7
Bystander tokens in the same sampled spans13.413.8
Plain tokens in the same logged sequences14.314.9

These closed-book results show that OBP models have a larger perplexity gap between offloaded tokens and other tokens, which is the expected behavior since our objective pushes models to memorize offloaded tokens less. We also note that the high perplexity for offloaded tokens is expected, since these are the most fact-rich and therefore difficult tokens to predict.

(b) OBP on Simple Wikipedia, with two off-the-shelf KBs

To test memorization in a setting where the baseline is able to memorize more, we trained on a smaller dataset (Simple Wikipedia) for 20 epochs, using either the same dataset or the original KILT snapshot as KB. The Simple Wikipedia KB contains all 2.29M sentences of the corpus, including those of the held-out pages. The KILT snapshot KB contains the 15.9M sentences of the 207k English Wikipedia articles whose titles match a Simple Wikipedia page.

Perplexity on held-out split

We randomly draw 1,000 pages from Simple Wikipedia as the held-out split.

KBModelClosed-book PPLAlways Retrieve PPLSelective Retrieval PPL
Simple WikipediaBaseline18.28.9–
OBP17.42.92.9
KILT snapshotBaseline18.216.8–
OBP17.111.711.6

Perplexity on offloaded tokens

KBModelOffloaded tokens PPLBystander tokens PPLPlain tokens PPL
Simple WikipediaBaseline16.93.55.0
OBP32.94.06.4
KILT snapshotBaseline12.64.55.0
OBP22.95.66.6

The overall perplexity results show that models are able to understand complex retrieved sentences from a plug-in KB, even when only trained on simple sentences. Our OBP models have ~2x higher perplexity on offloaded tokens than baseline models, showing that when the baseline starts to memorize, OBP's training objective pushes the model towards less memorization.

(c) Knowledge-update experiments, changing KB during inference

Building on our original training runs using atomic KB, and the added runs with Simple Wikipedia KB or KILT KB, we test our models on SciQ. We use the support paragraph from SciQ as evidence. Our results show again that OBP is worse at memorization and better at using retrieved evidence.

KBModelClosed-book accuracy (%)Open-book accuracy (%)Gain (Open - Closed)
DefaultKILT-Baseline34.366.131.8
KILT-OBP30.770.439.7
Simple WikipediaSW-Baseline33.057.424.4
SW-OBP28.960.131.2
KILT snapshotSW-OBP28.760.131.4

How does OBP compare with LMLM or a RETRO-style model trained on the same data and KB? If those methods reach similar retrieval gains, the benefit of OBP’s dynamic, model-dependent offloading would be less clear.

Both LMLM and RETRO are static approaches, since LMLM explicitly annotates facts and RETRO retrieves once every 64 tokens. RETRO does not explicitly control which information is memorized in model parameters, while LMLM and Co-LMLM are static offloading approaches. Therefore, we separate this question into two design choices for retrieval-augmented pretraining: (a) dynamic vs. static; (b) explicit offloading vs. no explicit control for memorization. We show different retrieval gains resulting from each of these choices below.

(a) Dynamic vs. Static

The biggest difference between dynamic and static offloading is that dynamic offloading don't control how much information and which information is being offloaded. In LMLM and Co-LMLM, the KB is constructed together with the training corpora to always contain the ground truth information their annotator wants to offload. LMLM offloaded 11.6% of their training corpus, calculated from their released data.

We compare dynamic and static offloading under the same offloading budget. In the static variant, each training sequence offloads one randomly chosen annotated fact span for default KB construction. Four other positions are selected randomly and offload nothing (and thus only contribute to bystander loss and plain loss). This matches OBP's offloading rate (about 4.5 tokens per sequence). Our results show that dynamic offloading outperforms this static, annotation-based variant in both selective retrieval PPL and open-book question answering EM scores:

ModelClosed-book PPLAlways retrieve PPLSelective retrieval PPL
Baseline13.312.2-
OBP13.710.310.1
Static-OBP13.89.711.0
ModelSettingTriviaQANQT-REx
BaselineClosed-book4.61.136.9
Open-book13.49.446.4
OBPClosed-book5.41.234.4
Open-book27.315.652.9
Static-OBPClosed-book5.11.333.2
Open-book19.412.747.4

Comments on the paper

3 comments

Dear reviewer,

Thank you for the quick reply. We attempt to address each of these questions below:

(1) Perplexity on offloaded tokens

During training, we logged offloaded tokens every 200 steps from step 15k, amounting to about 63k offloaded tokens per seed. We report mean perplexity on these offloaded tokens versus bystander tokens and plain tokens, for the final baseline and final OBP models of GPT2-124M:

Set of TokensAvg. PPL of BaselineAvg. PPL of OBP
Offloaded tokens70.076.7
Bystander tokens in the same sampled spans13.413.8
Plain tokens in the same logged sequences14.314.9

These closed-book results show that OBP models have a larger perplexity gap between offloaded tokens and other tokens, which is the expected behavior since our objective pushes models to memorize offloaded tokens less. We also note that the high perplexity for offloaded tokens is expected, since these are the most fact-rich and therefore difficult tokens to predict.

(2) Our learned retrieval classifier against simple heuristics

We compare our learned retrieval classifier against simple heuristics and FLARE below, on GPT2-124M and SmolLM2-135M. We use the same query format across all policies for a fair comparison. For FLARE, the setting is to retrieve when the predicted sentence continuation has a low-confidence token, and we predict 10 tokens at each position.

To match retrieval budgets with our classifier (i.e. # of retrievals per sequence, ~30 and ~19 for the two τinf\tau_\text{inf} values), we take tokens of different quantiles with highest entropy/lowest confidence in FLARE. Given matched retrieval budgets, as well as overall, our learned retrieval classifier shows the best average held-out perplexity.

GPT2-124M

PolicyModel# of Retrievals per sequenceAvg. PPL
No RetrievalBaseline013.24
AlwaysBaseline40.612.32
Entropy, top 30%Baseline29.711.85
No RetrievalOBP013.71
AlwaysOBP40.610.34
Our classifier, τinf\tau_\text{inf} = 0.5OBP29.310.17
Our classifier, τinf\tau_\text{inf} = 0.7OBP18.610.48
Entropy, top 30%OBP29.810.60
Entropy, top 10%OBP16.811.53
FLARE, bottom 50%OBP29.610.83
FLARE, bottom 20%OBP16.711.90
Fixed interval, every 10 tokensOBP20.211.96

SmolLM2-135M

PolicyModel# of Retrievals per sequenceAvg. PPL
No RetrievalBaseline021.18
AlwaysBaseline42.319.04
Entropy, top 30%Baseline30.318.90
No RetrievalOBP021.27
AlwaysOBP42.315.98
Classifier, τinf\tau_\text{inf} = 0.5OBP32.415.73
Classifier, τinf\tau_\text{inf} = 0.7OBP18.716.37
Entropy, top 30%OBP30.316.31
Entropy, top 10%OBP17.517.81
FLARE, bottom 50%OBP30.316.67
FLARE, bottom 20%OBP17.618.18
Fixed interval, every 10 tokensOBP21.118.69

We have drafted a discussion paragraph below, and will include it in our revision:

Other approaches for learning when to retrieve (Asai et al., 2023; Zhao et al., 2026) train on corpora where retrieval positions are annotated offline by a larger model. During inference, the models retrieve when they generate the corresponding special token. Moreover, their retrieval frequencies are different during training: Self-RAG decides once per sentence whether to retrieve, and LMLM uses 54.6M annotated knowledge triplets for ~3B tokens. Therefore, their retrieval policies are not directly comparable with our classifier. We compare our classifier with simple heuristics and FLARE (Jiang et al., 2023), and apply these methods at the same candidate positions, with the same query format and retrieved documents.

[1] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection, ICLR 2024.

[2] Linxi Zhao, Sofian Zalouk, Christian Belardi, Justin Lovelace, Jin Peng Zhou, Ryan Thomas Noo-nan, Dongyoung Go, Kilian Q Weinberger, Yoav Artzi, and Jennifer J. Sun. Pre-training limited memory language models with internal and external knowledge. ICLR 2026.

[3] Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active Retrieval Augmented Generation. EMNLP 2023.

(3) Efficiency Metrics

After the warmup phase, we sample 5 spans per training sequence and retrieve the top-5 documents for each span. We report training FLOPs per sequence:

ModelBaseline (TFLOPs)OBP (TFLOPs)OBP / baseline
GPT2-124M0.82.22.6×
GPT2-355M2.36.92.9×
SmolLM2-135M0.92.72.9×
SmolLM2-360M2.47.33.0×

We then report total training FLOPs and wall-clock time ratio per training step, on 4 RTX A6000 GPUs. Wall-clock time includes retrieval cost, which is an approximate nearest-neighbor search over our knowledge base for each span:

ModelBaseline (×1018\times 10^{18} FLOPs)OBP (×1018\times 10^{18} FLOPs)OBP / baseline, FLOPsOBP / baseline, s/step
GPT2-124M3.16.92.2×4.6×
GPT2-355M8.821.72.5×3.9×
SmolLM2-135M3.99.42.4×3.7×
SmolLM2-360M10.025.52.6×4.5×

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.