acceptodds
Under review as a conference paper at ICLR 2027

NewsCPT: News as a Substrate for Temporal Continued Pretraining

Abstract

Large language models (LLMs) become increasingly stale as the world changes beyond their pretraining cutoff. Continued pretraining (CPT) on newly produced text offers a natural way to update them, but it remains unclear which new facts such updates acquire, how far that knowledge propagates, and at what cost. We study these questions using news as a substrate for temporal CPT. We construct and release a globally deduplicated, timestamped corpus of 205M news articles from January 2024 to May 2026, along with dated question sets derived from it and Wikipedia's Current Events portal, and continue pretraining eight models from three families. Across three post-cutoff knowledge benchmarks, news CPT consistently outperforms both base models and matched-budget controls trained on post-cutoff FineWeb-Edu text. At the level of individual facts, accuracy improvement grows approximately log-linearly with how often the corpus states the fact, but those gains propagate only weakly to their consequences. The update degrades broader downstream performance, which replay with web data only partly mitigates. Mechanistically, retaining only the CPT-induced MLP updates preserves much of the new factual knowledge while improving several downstream losses, suggesting that acquisition and capability loss are partly separable. Our results provide a large-scale empirical account of how LLMs acquire new factual knowledge through CPT: factual acquisition is strongly associated with exposure in the update corpus, while propagation beyond the stated fact remains limited and acquisition and capability loss can be partially separated after training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Continued Pretraining, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8sDSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies temporal knowledge updating through continued pretraining on news. It introduces a timestamped, globally deduplicated corpus of 205 million news articles and dated factual evaluations. Eight models from three families are trained on 6 billion English news tokens and compared with base models and matched-token-budget FineWeb-Edu controls. The authors report improved post-cutoff factual accuracy, an association between factual exposure and acquisition, limited propagation to related questions, and degradation on broader capability benchmarks. Post-hoc weight-splicing experiments suggest that combining CPT-updated MLPs with the remaining base-model weights preserves much of the Wikipedia factual accuracy gain while partially reducing MMLU degradation.

Critical strengths

The paper provides a substantial empirical resource for studying temporal knowledge updates using real news. It constructs a timestamped, globally deduplicated corpus of 205 million articles and evaluates eight models from three families against base models and matched-token-budget FineWeb-Edu controls. The coverage of three factual benchmarks, including the independently collected Daily Oracle, strengthens the empirical breadth of the study. The consistent post-cutoff improvements across the tested models and benchmarks provide useful evidence about the benefits and costs of this update corpus (Section 2; Table 9).

Critical weaknesses
  1. The quantitative evidence supporting the central MLP-splicing result is internally inconsistent. Table 16 explicitly states that it uses the main-results protocol and the full Wikipedia-after set, yet its full-CPT gains differ from those implied by Table 9: 11.77 versus 12.19 percentage points for Llama-3.1-8B, and 2.79 versus 3.45 for OLMo-2-1B. Figure 25 introduces a further discrepancy, showing approximately 1.8 points for the latter model. Consequently, the aggregate gain retained by MLP splicing is approximately 91.2% using Table 16, compared with the reported 92.6%. These inconsistencies require reconciliation of checkpoint identities, evaluation settings, and reported results before the magnitude of this key finding can be assessed confidently.

  2. The MLP analysis lacks matched controls needed to establish its mechanistic interpretation and practical significance. Appendix H constructs the hybrid model by retaining base-model values for all non-MLP parameters, so the experiment does not isolate which removed updates account for the recovery. Furthermore, the interpolation baseline is reported on Daily Oracle (Figure 17), while the main splicing result uses Wikipedia-after and MMLU. A comparison on identical evaluations, at matched factual acquisition or capability retention, is needed to establish the benefit of module selection beyond simply reducing the overall update. This matters particularly because Appendix H.1 acknowledges larger acquisition losses on Daily Oracle and new downstream regressions for some models, without providing complete per-model results for these expanded evaluations.

Questions for authors
  1. Can you reconcile the numerical inconsistencies in the MLP analysis? For OLMo-2-1B, the full-CPT Wikipedia-after gain is 3.45 percentage points according to Table 9, 2.79 in Table 16, and approximately 1.8 in Figure 25. Please clarify the checkpoint identities, question sets, and scoring configurations used in these results, and provide consistent base/full-CPT/MLP-only accuracies together with the calculations underlying the reported 92.6% gain retention and 42.6% MMLU recovery. Resolving these discrepancies would materially increase confidence in the central quantitative finding.

  2. Does MLP-only splicing provide a better acquisition–retention tradeoff than uniform weight interpolation under matched evaluation conditions? A focused comparison using existing checkpoints on Wikipedia-after, Daily Oracle-after, and MMLU would be informative, particularly at comparable factual gains or comparable MMLU losses. Please also provide the complete per-model results underlying Appendix H.1, including the reported regressions. These results would help establish whether the benefit of module selection is robust across evaluations and exceeds that obtainable by simply scaling down the full update.

  3. Which reverted parameter groups account for the recovery observed with MLP-only splicing? Appendix H defines this intervention as keeping all non-MLP parameters at their base-model values. Could you specify the exact parameter groups involved and, for a small representative subset of models, compare it with an intervention that starts from the full CPT checkpoint and restores only the attention projections, leaving all other parameters at their CPT values? This would help distinguish the contribution of attention rollback from other parameter reversions and strengthen the interpretation of the module-wise results.

4 comments

We thank the reviewer for the careful review and the detailed, actionable questions, which helped us correct and strengthen the splicing analysis. We address each point below.

Response to Weakness 1 and Question 1

We are grateful to the reviewer for identifying these inconsistencies. For OLMo-2-1B, Table 16 contained a copy-paste error (+2.79 instead of +1.79 percentage points); the remaining discrepancy with Table 9 (+1.79 versus +3.45) arose from the OLMES scoring-protocol bug described in our general response. The Llama-3.1-8B discrepancy comes from re-scoring with different few-shot examples. We have aligned the scoring protocol with Table 9 and regenerated the affected results. Checkpoint identities and base/full-CPT/MLP-only accuracies are provided in Table 16 for Wikipedia-after and Daily Oracle; Figure 25 presents both evaluations. Corrected MLP-only gain retention is 88.7% on Wikipedia-after and 93.9% on Daily Oracle, while the revised MMLU recovery is 45.8%. Please see our general response for details of the evaluation corrections underlying these revised estimates.

The retention figures of 88.7% and 93.9% are calculated by dividing the summed accuracy gains over base of the MLP-only spliced models by the corresponding summed gains of the full-CPT models, on Wikipedia-after and Daily Oracle, respectively. The 45.8% MMLU recovery is calculated by dividing the summed accuracy improvement from full CPT to MLP-only splicing by the summed accuracy drop from base to full CPT. Each calculation uses the same checkpoint set in its numerator and denominator.

Response to Weakness 2

Matched controls (Weakness 2.1 and Question 3). Thank you for identifying this ambiguity. The MLP-only splice retains the CPT values of gate_proj, up_proj, and down_proj in every layer and restores all remaining parameters to their base-model values. The reverted groups are: (i) token embeddings and the output head (embed_tokens and lm_head, respecting weight tying where present); (ii) normalization parameters, including attention Q/K normalization where present; (iii) attention query/key projections (q_proj, k_proj); and (iv) attention value/output projections (v_proj, o_proj).

To distinguish attention rollback from other modules' reversions, we compare interventions that start from the full CPT checkpoint and restore only the selected group to base, leaving every other parameter at its CPT value. The table below reports means over the eight CC-News checkpoints; both metrics are changes relative to full CPT within this experiment.

Parameters restored to baseMMLU change (pp)Wikipedia-after change (pp)
Embeddings and output head+0.07−0.21
Normalization+0.01−0.30
Q, K projections+0.17−0.23
V, O projections+0.58−0.67
All attention projections (Q, K, V, O)+0.74−0.79

Restoring only the attention projections therefore roughly reproduces the MMLU recovery of the full non-MLP reset (0.74 vs 0.85 pp), while sacrificing slightly less post-cutoff Wikipedia accuracy (0.79 vs 0.92 pp). Among the individual groups, restoring V, O produces the largest average MMLU recovery. Restoring embeddings/output heads or normalization alone has a much smaller average effect.

These results support the interpretation that reverting attention-projection updates is sufficient to obtain comparable average MMLU recovery while retaining CPT values for embeddings, normalization, and MLP parameters. The effects vary across models and need not combine additively, so we do not interpret them as a unique decomposition of forgetting or as evidence that attention-frozen training would yield the same outcome.

Practical significance (Weakness 2.1). We distinguish the practical implications of our findings from the stronger mechanistic interpretation. Selective module splicing can recover capabilities affected by CPT while preserving much of the acquired knowledge, providing a way to adjust the acquisition–retention trade-off after training without repeating CPT. The new MLP-only training experiments summarized in our general response extend this evidence to a training-time intervention, with both factual acquisition and downstream capability retention evaluated. Together, these findings support post-hoc module selection and MLP-only training as promising interventions, without establishing a unique mechanistic explanation or replacing matched comparisons with whole-model interpolation on identical evaluations.

Splicing on common evaluations (Weakness 2.2). We have now evaluated all four splice configurations on after-cutoff Daily Oracle in Table 16. MLP-only splicing retains 93.9% of the aggregate full-CPT acquisition gain on Daily Oracle and 88.7% on Wikipedia-after, compared with 15.7% and 21.4% for attention-only splicing. This supports a consistent concentration of factual acquisition in MLP updates across both benchmarks.

Appendix H.1's earlier claim of larger acquisition losses on Daily Oracle arose from an answer-formatting artefact, described in our general response; the aggregate pattern reverses after correction. We have revised that discussion and provide complete per-model results in Table 16 for Daily Oracle and Table 18 for the expanded downstream suite. Splicing reduces the average downstream change from base from −1.38 to −0.85 points, although some model-specific regressions remain.

MLP-only splicing vs. weight interpolation (Weakness 2.2 and Question 2). Tables 16 and 17 now compare module splicing and uniform weight interpolation on Wikipedia-after, Daily Oracle-after, and MMLU under the same evaluation conditions. At comparable mean MMLU losses, MLP-only splicing and interpolation at α=0.3 produce changes of −1.01 and −0.89 percentage points, and splicing achieves larger after-cutoff gains on both Wikipedia (+7.28 versus +6.86 pp) and Daily Oracle (+5.14 versus +4.41 pp). The ordering also holds at matched factual gains, where reaching splicing's gains requires α<0.3 and a larger MMLU loss. Interpolation instead offers a tunable operating point that splicing does not: α=0.5 reduces the MMLU loss to −0.40 pp at smaller gains (+5.35 and +3.29 pp). Module selection therefore preserves more acquisition at a given capability loss, while interpolation provides continuous control over where on the trade-off a practitioner lands.

Thank you for the detailed rebuttal. The numerical corrections and additional experiments have addressed several of my original concerns.

I noticed that you added MLP-only training experiments in Appendix J, which seem to achieve comparable or even better results than post-hoc MLP splicing. This makes me wonder:

  1. What is the practical advantage of post-hoc MLP splicing? If training only the MLPs already achieves a better acquisition–retention trade-off, why would we prefer full CPT followed by weight splicing?

  2. Could standard PEFT methods achieve similar results? For example, would LoRA or adapters also allow the model to acquire new knowledge while preserving existing capabilities? Without such baselines, it is unclear whether the benefits are specific to MLP updates or simply arise from restricting the parameter updates.

One minor remaining concern is that Tables 16 and 17 use different few-shot examples. Could you clarify whether the advantage over weight interpolation holds under identical evaluation settings?

Q1. What is the practical advantage of post-hoc MLP splicing?

We agree with the reviewer that, given our new results, MLP-only training is the better-supported choice when planning a new training run. This finding emerged from additional experiments conducted during rebuttal, motivated by our post-hoc splicing results. Splicing showed that retaining only the MLP updates preserves most factual gains while reducing downstream degradation, which prompted us to test whether restricting training to MLPs could achieve an even better trade-off. We therefore view splicing's primary contribution as an experimental intervention that motivated and guided this training strategy, rather than as a preferred alternative to MLP-only training. Splicing also retains a practical advantage when a full-CPT checkpoint is already available: it can improve the acquisition–retention trade-off without further training, whereas adopting MLP-only training at that stage would require a new training run.

Q2. Could LoRA or adapters achieve a similar acquisition–retention trade-off?

Biderman et al. [1] found that low-rank LoRA learned less and forgot less than full CPT, with similar forgetting at comparable levels of acquisition. In a time-incremental setting closer to ours, Öncel et al. [2] found that LoRA on attention and MLP projections at ranks 128 and 256 recovered most of full CPT's after-cutoff Daily Oracle gain on Llama-3.2-3B, while before-cutoff accuracy stayed close to baseline. Given our limited rebuttal time, we are unable to empirically evaluate this. We will include such a comparison in our camera-ready version.

Minor concern. Does the advantage over interpolation hold with identical few-shot examples?

Yes, it does. We re-evaluated interpolation using the same pipeline, questions, and few-shot examples as splicing. The table reports changes from each base model in percentage points (pp), averaged across Llama-3.2-3B, OLMo-2-7B, and Qwen3-4B:

MethodMMLUWikipedia (after cutoff)Daily Oracle (after cutoff, corrected)
Full CPT (α = 0)−2.23+8.41+6.16
MLP-only splicing−0.99+7.52+5.70
Interpolation (α = 0.3)−1.19+6.85+5.24
Interpolation (α = 0.5)−0.66+5.07+3.80
Interpolation (α = 0.7)−0.35+3.11+2.37

Splicing outperforms interpolation at α = 0.3 on all three metrics on average. This also holds individually for Llama and OLMo. On Qwen, splicing improves both acquisition scores at an additional MMLU cost of 0.27 pp. At α = 0.5, interpolation retains 0.33 pp more MMLU accuracy on average, but gains 2.45 pp less on Wikipedia and 1.90 pp less on Daily Oracle than splicing.

References:

  • [1] Biderman et al., LoRA Learns Less and Forgets Less, TMLR 2024.
  • [2] Öncel et al., Time-Incremental Continued Pretraining of LLMs: Knowledge Updates Without Catastrophic Forgetting, arXiv:2609.23916.
Reviewer_y2V7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper makes 3 main contributions: First, it curates a CC-News corpus, containing news articles from 2024-2026. The raw CC data is processed to remove duplicates + other standard filtering is performed. Second, they perform CPT on this CC-News dataset + a control dataset (FineWebEdu) and report various findings on fact acquisition and forgetting. Main conclusions are that the CC-news dataset reports better fact acquisition than FineWeb, that the strength of facts acquisition is correlated with frequency in the training data, amongst others. Finally, the paper includes experiments on knowledge propagation (can the model reason about the consequences of fact updates that it has memorized), and forgetting.

The evaluation is conducted using MCQ style QA on 3 datasets: they construct a CC-news oriented dataset by extracting facts in their corpus (+ some questions from the pre-cutoff corpus), Wikipedia and Daily Oracle questions. It is important to highlight that all these evaluation datasets are biased towards news (e.g. Wikipedia updates are derived from new news). Therefore, their central claim "Across eight models and three benchmarks, CC-News CPT consistently outperforms matched-budget FineWeb-Edu CPT for post-cutoff factual acquisition." is very misleading as this observation can almost entirely be attributed to the match between CC-News and evaluation datasets.

Critical strengths

The news dataset, deduplicated and cleaned from CC, present a useful corpus for studying continued pre-training. The evaluation datasets / pipeline for creating MCQs and identifying support documents can also be useful for future work.

Critical weaknesses
  1. The main claim of the paper: "Across eight models and three benchmarks, CC-News CPT consistently outperforms matched-budget FineWeb-Edu CPT for post-cutoff factual acquisition." is very misleading. The main reason why this claim holds in the reported experiments is because evaluation datasets are very news-biased. One is explicitly derived from CC-News. I expect the % of news-adjacent documents in FineWebEdu is quite low (<10%). The paper does acknowledge "Our central finding is that the update corpus shapes both what models learn.." in a few places (introduction; line 67) but it makes the outperform claims without qualifying it appropriately more times and more prominently (e.g in the abstract).

To be convinced of this central finding, I would require an apples-to-apples comparison. Extract facts that have equivalent (to the extent possible) support in BOTH datasets. This can likely be achieved by either (1) extracting half the facts from the news subset of FineWebEdu, (2) the paper already implements the pipeline to identify #documents corresponding to each fact -- ensure that the distribution of #support documents for the facts in the evaluation set is same for CC-News and FineWeb.

  1. The other main findings in the paper are not new insights and have been extensively previously discussed for continued pre-training e.g. studying fact acquisition with simple CPT (https://aclanthology.org/2025.acl-long.1551.pdf), that facts learned are a function of frequency (https://proceedings.mlr.press/v202/kandpal23a.html). The findings from Section 3.3., that facts do not propagate is exactly established in https://aclanthology.org/2025.findings-acl.1326/ and others). Synthetic continued pre-training (https://arxiv.org/pdf/2409.07431) establishes that raw CPT (used in this work) saturates.

The other questions studied (e.g. forgetting and if replay mitigates forgetting) have also been extensively studied. In fact, adding replay data is such a standard technique for mitigation that it is now used in all papers on continual learning by default (https://arxiv.org/pdf/2409.07431). The fact that FineWebEdu replay data helps (another finding in this paper) is not surprising either as that corpus is likely to be closer to the pre-training distribution of these models as well as downstream benchmarks. These have been established in prior works.

The paper itself cites these papers. In my opinion, the differences highlighted are minimal (e.g. moving away from a perplexity based analysis) and the results do not reveal new insights.

  1. I have a number of questions/concerns about the experimental setup.

    • In Figure 4, the text says "Also, FineWeb-Edu checkpoints show no relationship between accuracy gains and the same counts on the same questions". I do not understand this statement. For the FineWebEdu line in the graph, are the questions in a bin decided based on the count of the corresponding statements in the CC-News corpus or the FineWebEdu corpus? My reading suggests it is the CC-News corpus. If so, then FineWebEdu trends cannot be used to draw any conclusions about frequency-accuracy correlation. If no, then the highlighted claim that "More frequent factual exposure predicts larger accuracy gains. " is false.
  2. It was extremely difficult to follow and understand experimental setups in a number of places. For e.g., Figure 4 reports results on two CC news eval subsets b1 and b2. Nowhere in the main body of the paper is it even mentioned what these two subsets are. Figure 4 is on page 5, I had to go to page 23 of the Appendix to understand why there were two subsets. They state that the goal is to test if the effect holds on two independent sets. But, the question generation pipeline differs on the two subsets, so it is a flawed experiment design to study that.

Questions for authors
  • What are substantial new insights from this work that have not been discussed in prior works?
  • Please clarify the experimental details in weaknesses 3 and 4 above.
5 comments

We thank the reviewer for their feedback. We address each point below.

Response to Weakness 1

We agree that our claim should identify the evaluation domain more clearly. We study factual world knowledge acquisition through continued pretraining, evaluated using post-cutoff facts about recent world events. We will qualify the abstract and conclusions accordingly.

Our evaluations are constructed around the knowledge we aim to measure. Matching questions or factual exposure across training corpora would answer a different question: learning efficiency given comparable exposure. Under our fixed-budget comparison, factual coverage and repetition are properties of the corpora being evaluated.

To test how the evaluation source affects our findings in a cross-domain setting (trained on one dataset and tested on another), we added question banks derived from FineWeb-Edu, FineWeb, and Wikidata. FineWeb-Edu checkpoints outperform CC-News checkpoints by 1.7 pp on average on FineWeb-Edu-derived questions. Results on questions from FineWeb's news and non-news pages are within noise. Some advantage on questions derived from a model's own training corpus is expected. However, this advantage is small compared with the substantially larger gains for CC-News on independently sourced benchmarks of recent world-event knowledge: on questions constructed from Wikidata's structured records, without news articles, CC-News outperforms FineWeb-Edu across all eight models by an average of 7.8 points, and FineWeb by 7.2 points. This is consistent with its advantage on our existing Wikipedia and Daily Oracle benchmarks.

These findings support a qualified claim: CC-News is particularly effective for acquiring recent world-event knowledge, while its advantage depends on the knowledge being evaluated and does not extend uniformly across domains.

Response to Weakness 2

Summary. Our main contribution is to study factual updating at scale, using realistic data and heavily pretrained models: we update eight released checkpoints from three model families on billions of tokens of naturally occurring news, with training-budget experiments extending to 29B tokens. This setting captures a practical question: what can practitioners expect when updating an existing LLM on newly available information? We agree that frequency-dependent acquisition, limited propagation, and the benefits of replay have precedents. But knowing the direction of an effect does not establish its magnitude or practical significance under these conditions. Our contribution is to quantify their extent, interactions, and limitations in this setting, using factual QA evaluations and matched-budget controls.

  • TiC-LM. TiC-LM is closely related, and we acknowledge it as such. It studies continual training along a controlled trajectory, starting with 110B tokens of initial pretraining of only 1B–3B models; we instead update released, heavily pretrained checkpoints. Equally importantly, TiC-LM's dynamic evaluations are perplexity-based, a limitation its authors explicitly acknowledge. We measure changes in factual question-answering accuracy, which is much more aligned with real-world tasks. Our work therefore complements TiC-LM by directly measuring how much usable factual knowledge these updates provide.
  • Exposure. Our exposure analysis goes beyond establishing that frequency matters. Kandpal et al. provide quantitative evidence linking pretraining frequency to factual accuracy. We measure a different quantity: the change in accuracy following a specific update, as a function of how often the update corpus states the fact. We give concrete accuracy gains for each tenfold increase in statement counts. These results quantify the dependence of acquisition on coverage in a realistic update corpus, beyond the general observation that frequent facts are easier to learn.
  • Propagation. Our propagation results extend prior findings to large-scale updating on real events. Li & Goyal use 1,000 fictitious updates, and their indirect evaluation includes 60 manually curated questions. Our evaluation covers 1,358 real post-cutoff anchors and 2,561 follow-up questions across seven checkpoints trained on billions of tokens of natural news. Thus, we quantify how limited propagation remains when facts are acquired through substantial exposure to naturally occurring coverage.
  • Saturation. The saturation results likewise concern a substantially different data regime. Synthetic continued pretraining (Yang et al., 2025) studies augmentation of a small, fixed source corpus, including a raw CPT baseline on 1.3M unique tokens. Our experiments measure the returns to increasing training on a large natural news stream up to 29B tokens. Most gains occur within approximately the first 10B tokens for OLMo and 20B for Llama in the tested runs. We do not claim that diminishing returns are a new phenomenon; we provide concrete measurements of where they arise in this updating setting.
  • Replay. We agree that replay is standard; its effectiveness is the empirical question. At a fixed 6B-token budget, replacing 0.5B news tokens with FineWeb-Edu recovers approximately 30% and 14% of the downstream accuracy loss for the tested 3B and 8B Llama models. The majority of the loss therefore remains at this replay fraction. This is a scoped result about the effectiveness of a practical mitigation, rather than a claim to introduce replay or establish its general limits.

We will revise the framing to distinguish established qualitative effects from our contributions more explicitly. The contribution is a controlled, quantitative account of factual updating at realistic scale: how acquisition depends on natural coverage, how much acquired knowledge propagates, where additional training yields diminishing returns, and how much downstream degradation persists despite replay.

Response to Weakness 3

Yes, both curves use CC-News statement counts: the FineWeb-Edu plot is a matched corpus control, not an analysis of frequency–accuracy correlations within FineWeb-Edu. Its flat/negative trend argues against trivial alternative explanations, e.g., that CC-News counts only capture question properties, such as popularity or ease, that would also predict gains regardless of the training corpus. The evidence for the exposure–accuracy relationship comes from the CC-News plot; we will revise the wording and legend to make this distinction explicit.

Response to Weakness 4

As noted in the general response, we added a two-sentence summary of the findings in each section. Regarding b1 and b2: we agree that the two subsets should have been defined in the main text, and we apologise for the unclear presentation. Both subsets in Figure 4 use the same question-generation pipeline and generator (Qwen3.6-27B), applied to disjoint article samples drawn using the same sampling procedure; the Mistral-generated questions mentioned in Appendix C belong to a separate generator comparison and are not plotted in Figure 4. Thus, Figure 4 tests whether the relationship holds across article samples with the generation procedure fixed, and we will make this explicit in the main text and figure caption.

Response to Question 1

Our contribution is to establish the practical extent and limits of factual updating in released, heavily pretrained models, rather than to claim that frequency effects, limited propagation, or replay are newly discovered phenomena. Specifically, we:

  • quantify accuracy gains per tenfold increase in factual exposure;
  • find that limited propagation persists after billion-token training on real events;
  • locate diminishing returns at approximately 10B tokens for OLMo and 20B for Llama in the tested runs;
  • show that a standard replay intervention recovers only 14–30% of downstream accuracy loss at the tested budget.

Our additional evaluations also clarify that corpus advantages are knowledge-dependent: FineWeb-Edu performs better on its own derived questions, whereas CC-News retains a 7.8-point average advantage on independently constructed Wikidata world-event questions across all eight models.

Reviewer_dvjGOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how continued pretraining (CPT) on newly produced text updates an LLM's factual knowledge: which new facts are acquired, how far the knowledge propagates, and what the update costs. The authors build a globally deduplicated, time-ordered corpus from CC-NEWS (205M articles, January 2024 to May 2026) and continue pretraining eight models from three families (1.7B to 13B) on 6B English news tokens, with a matched-budget control trained on post-cutoff FineWeb-Edu. On three post-cutoff question sets, news CPT beats both the base models and the control for every model (+4.5 / +8.2 / +4.7 points over base, and +4.5 / +6.3 / +3.6 over the control, Table 9). At the level of individual facts, accuracy gains grow roughly log-linearly with how many documents state the fact (about +2.4 to +3.6 points per tenfold increase, Fig. 4). A ripple-effect evaluation shows that learned facts propagate only weakly to their consequences (+3.0 and +6.8 points on second- and third-order follow-ups of acquired facts, Sec. 3.3). The paper also reports a 0.6 to 2.5 point downstream cost that replay only partly recovers, and shows that keeping only the MLP updates preserves most of the factual gain (Table 16).

Critical strengths
  1. A practical and important problem, studied with well-designed experiments at a realistic scale. Keeping deployed models up to date without retraining from scratch is a real need, and the paper shows clearly that the choice of update corpus matters: news CPT improves post-cutoff knowledge for all eight models and all three benchmarks, while a matched-budget FineWeb-Edu control barely moves (Table 9). The design is careful: deduplication across time removes hidden repetition between crawl slices (Sec. 2.1), rotation-averaged multiple-choice scoring removes letter bias (Fig. 10), the smallest gap over the control (1.76 points) is about four times the seed standard deviation of a difference (App. F), and the exposure analysis comes with proper null checks (no slope before the cutoff or for the control, Fig. 13). The released corpus and checkpoints would be a valuable resource for research on knowledge updating.

  2. The ripple-effect study of consequence facts is interesting and, to my knowledge, novel in the CPT setting. Asking whether a fact learned from news also updates what follows from it (Sec. 3.3, App. G) moves the evaluation beyond "can the model repeat the fact". The design is thoughtful: follow-ups must depend on the anchor, the pre-event answer is offered as a distractor, and the analysis is restricted to anchors the model actually acquired. The result is clear and useful: even when anchor accuracy rises from 28% to 92%, follow-up accuracy improves by only 5 to 8 points (Table 15), and models still pick the outdated answer 43.6% of the time. This tells practitioners that CPT alone will not make a model consistent about recent events.

Critical weaknesses
  1. The writing makes the paper harder to follow than it needs to be. Several sections pose a question and then take a long detour through setup and caveats before giving the answer, so the reader can lose track of what was asked. I would suggest answering each question directly right after it is posed (one or two sentences with the key number), then giving the details. Key numbers also sit in the appendix (Table 9 for the main results, Tables 13 to 16), and a few values disagree between sections: seed SD 0.4 (Sec. 2.2) vs 0.28 (App. F), 92.6% (Sec. 3.6) vs 91.2% from Table 16, Fig. 25's "at least 82%" vs 77.8% for OLMo-2-1B, and the Wikipedia gains in Table 16 vs Table 9 (e.g. Llama-3.1-8B +11.77 vs +12.19). This matters because the paper's contributions are mostly empirical findings, and their value depends on the reader being able to find and trust them quickly. It is fixable in a revision.

  2. Positioning and controls could be stronger. The relationship between repetition and fact learning, and the difficulty of long-tail facts, have been studied before, including at smaller scale with synthetic or knowledge-graph style data. The paper should position its exposure result more clearly against that line of work and against Oncel et al. (2026), whose CPT setup (models, schedule, Daily Oracle splits, forgetting analysis) it closely follows. The FineWeb-Edu control is educationally filtered and contains few recent events (it moves after-cutoff accuracy by only +1.9 / +1.1 / +0.1 points), so an unfiltered post-cutoff web control and a retrieval baseline over the same news (Daily Oracle already provides a BM25 setting) would make the "news matters" claim sharper. The exposure result is correlational; a small experiment that deliberately repeats or paraphrases a random set of facts would make it causal. Finally, all questions are LLM-generated, judge agreement is low (kappa 0.23 to 0.27, App. D), and the ripple bank was written and checked by LLMs (App. G.1); a small human audit would increase confidence. None of this undermines the main findings, but it limits how far they can be generalized.

Other issues:

  • In the ripple study, second-order accuracy is close to chance (28.1% to 31.0% on four-way questions) and per-model cells are small (58 to 228 questions). Some "acquired" anchors may be lucky guesses, and the paper cannot yet separate failed integration from consequences that never appear in the corpus; counting exposure for the follow-up facts would help.
  • The corpus has ten languages, but only English is used in the experiments.
  • The OLMo-2 paper does not state a knowledge cutoff, and Qwen3 is used with an assumed cutoff; excluding Qwen3 strengthens the results, so this is a reporting point.
  • No code or data link is given, although the release is a main contribution.
Questions for authors
  1. [Suggestion] Could you restructure the sections that pose a question so that the answer, with its key number, comes immediately after the question?

  2. [Suggestion] How does news CPT compare with an unfiltered post-cutoff web control and with BM25 retrieval over the same news tokens? Does CPT add anything on top of retrieval?

  3. [Clarification] Can you count how often the follow-up (consequence) facts are stated in the corpus, to separate "never seen" from "seen but not integrated"?

  4. [Clarification] What is the human-audited error rate on a sample of the Wikipedia, CC-News and ripple questions?

  5. [Clarification] What exactly will be released (full text or URLs plus scripts, checkpoints, questions), and under what license?

2 comments

We thank the reviewer for a careful and highly constructive review. The suggested controls and analyses have substantially strengthened the paper, and we address each point below, with the corresponding manuscript changes marked in red.

Response to Weakness 1

  • Presentation. Presenting both the setup, which differs across experiments, and the results makes the exposition challenging, and we recognize that this can obscure the main answer. As noted in the general response, we state the answer and key quantitative result immediately after each question before explaining the experimental details.
  • Seed SD. 0.28 accuracy points is the pooled across-seed standard deviation for a single checkpoint, whereas 0.4 points is the corresponding standard deviation for the difference between two independently seeded runs (2×0.28≈0.4\sqrt{2}\times 0.28 \approx 0.4). We will revise Sec. 2.2 and cross-reference App. F. This correction does not change the noise level used in our significance statements.
  • Inconsistencies. Tables 9 and 16, as well as Figure 25, are updated after resolving the consistency issues as outlined in the general response. The difference in Llama-3.1-8B's numbers (+11.77 vs +12.19) stems from the random selection of few-shot examples, and matches the SD (0.4 vs 0.42). Please note that the updated tables do not change our conclusions.

Response to Weakness 2

  • Positioning against exposure literature. Our contribution is not to establish that repetition aids factual learning, but to quantify its practical effects when updating heavily pretrained models on naturally emerging data. As the reviewer acknowledged, much of the related mechanistic work uses synthetic or knowledge-graph-derived data with controlled repetition, reporting thresholds such as ~1k exposures for capacity saturation (Allen-Zhu & Li) and 1k–2k subject–object co-occurrences for linear relational representations at 6–7B (Merullo et al., 2025). These contextualise how much sparser exposure is in our more realistic and less controlled setting: the median statement count is 24, and 24.9% of questions have no matching statement within the chosen window. In this regime, each tenfold increase in statement count is associated with gains of 3.18, 2.41, and 3.60 percentage points on our three question sets, which strengthen their findings.
  • Positioning against Oncel et al. We explicitly build on Oncel et al. (2026), retaining their models and evaluation protocol by design to isolate the update-corpus comparison. Their post-cutoff web crawl need not consist of newly published material, whereas our timestamped, globally deduplicated news corpus provides a clearer temporal basis for studying factual updates. It outperforms matched-budget FineWeb-Edu on post-cutoff questions across all models and benchmarks (4.8 percentage points on average). Beyond this corpus comparison, we measure factual exposure under naturally occurring repetition and extend the analysis to knowledge propagation, topic-level forgetting, and weight-space localisation.
  • Training on unfiltered web control. Please see our general response.
  • Causal exposure experiment. We agree that Section 5 (Fig. 3) is observational. To address the reviewer's concern, we took a subset of 2,031 articles and varied the number of renderings of each article's facts from 1 to 4, 12, and 28 using four rewriting styles and up to seven generator LLMs. With the same student (Llama-3.2-3B), training recipe, and evaluation questions, accuracy increased monotonically from 38% to 53.8%, 62.1%, and 66.8%. Repeated training on the original summaries of the news plateaued, whereas additional paraphrases continued to improve performance. Controls suggest that rewrite volume matters more than generator diversity. These results support a causal effect of adding synthetic restatements in this setting, although fixed epochs mean that larger corpora also receive more training steps.
  • Judge agreement. The κ we report is agreement on a single yes/no accept decision, and the two judges' total scores typically differ by only about a point. However, since the cutoff sits right where most questions pile up, that one point is enough to put them on opposite sides of it and register as complete disagreement. On the individual quality criteria themselves, the judges agree about 88% of the time, and nearly all of the disagreement comes from one subjective criterion: whether an answer is too obvious. Most importantly, restricting the benchmark to the questions both judges accept moves the mean CPT gain only from +0.7 to +0.6 points, so this borderline disagreement has little bearing on what we measure.

Response to other issues

  • Ripple study. We agree that the per-model second-order cells are too small to be read individually, and Sec. 3.3 no longer does so. Averaged over seven checkpoints, follow-up accuracy on acquired anchors increases by 3.0 and 6.8 percentage points on second- and third-order questions. Bootstrapping over anchors gives 95% intervals of −0.1 to +6.3 and +4.8 to +8.8 points, so the gains remain small even at the upper ends. Accuracy stays low partly because outdated answers remain common, with the checkpoints selecting the pre-event answer 43.6% of the time when it is offered (Table 14). We also agree that some acquired anchors are lucky guesses. Requiring the correct answer under all four rotations of the choices raises the gains to 5.6 and 7.4 points, still far below the gain on the anchors themselves (App. G.3).

    As suggested, we counted statements for the follow-up facts in the new App. G.5 (Table R1). Among multi-word answers, no document matches for 35.7% of second-order and 12.5% of third-order follow-ups, compared with 1.6% of anchors. However, at 100 to 999 matching documents, anchors gain 18.8 points while follow-ups gain only 3.3 and 5.9. Thus, both cases occur, and missing exposure alone does not explain the weak propagation.

  • Language. We run English-only experiments since pretraining language would complicate the interpretation of the findings and could be a confounder in the deduced conclusions.

  • Knowledge cutoffs. The official OLMo-2 repository (https://huggingface.co/allenai/OLMo-2-0425-1B) reports the knowledge cutoff date. We are in favor of keeping Qwen as supporting evidence that our findings hold consistently and generalize to models with similar publication dates.

  • Release. Upon acceptance, we will publish the dataset as well as the question banks.

Responses to questions

  • Q1. Thanks for this great suggestion; we updated the experiment section accordingly.
  • Q2. Please see the general response for our results on FineWeb-trained checkpoints. Detailed results are in Table 21.
  • Q5. We plan to release the question sets, answer keys, and evaluation metadata on Hugging Face under CC BY 4.0; code and reproduction scripts under MIT; and corpus manifests under CC0. We will provide source URLs, CC references, and article-retrieval scripts rather than full copyrighted article text. Checkpoints will follow their base-model licenses. Final releases will be subject to verification of applicable third-party terms.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reading and constructive feedback. We appreciate their recognition of our news corpus as a valuable resource for studying temporal knowledge updates (dvjG, y2V7, 8sDS), the usefulness of our evaluation datasets and question-generation pipeline (y2V7), and the careful experimental design and empirical breadth across models and benchmarks (dvjG, 8sDS). We also appreciate dvjG's emphasis on the practical importance of keeping models up to date and the value of studying how acquired facts propagate to their consequences.

This general response summarizes the corrections, additional evaluations, and revisions made in response to the reviews. These updates resolve inconsistencies in reported results, broaden the evidence, and clarify the scope of our conclusions. Across the expanded evaluations, news-focused continued pretraining consistently outperforms a new unfiltered web control (FineWeb), and MLP-only training preserves virtually all aggregate factual gains while approximately halving downstream degradation. All manuscript changes are marked in red.

Training on unfiltered web control (Table 9)

Per reviewer dvjG's suggestion, we trained on standard FineWeb (no educational filter) with the same 6B-token budget and learning rate. Averaged across our three after-cutoff benchmarks, FineWeb improves over the base models by +2.8 points; CC-News gains average +6.1 points. Removing the filter does yield a stronger control: FineWeb exceeds FineWeb-Edu (+1.5 points on average) in all eight models. CC-News nonetheless outperforms it in all 24 model-by-set comparisons, by 1.3 to 9.3 points. These results support the benefit of news-focused continued pretraining beyond the comparison with FineWeb-Edu.

MLP-only training

We added a direct training-time test of the splicing finding on five Llama and Qwen checkpoints. Updating only MLP parameters during CPT preserves 101.2% of the aggregate full-CPT gains on Wikipedia after-cutoff questions and 99.9% on corrected Daily Oracle, while approximately halving the mean downstream drop from 1.02 to 0.52 points. On the same five checkpoints, post-hoc MLP splicing retains 91.6% and 92.6% of the respective gains, with a downstream drop of 0.70 points. Thus restricting training to MLPs also improves the aggregate acquisition–retention trade-off, extending the evidence beyond post-hoc splicing. The benefit is not uniform across models and metrics; the appendix reports the full comparison.

Daily Oracle evaluation correction

We identified formatting cues in 3,355/6,935 after-cutoff Daily Oracle items from Dai et al. (2025): whitespace or stray delimiters distinguish the correct option. Checkpoints respond differently to these cues, shifting affected FineWeb-Edu scores by 8–16 points in either direction. We found no such cues in the before-cutoff split or in our CC-News and Wikipedia sets. We removed stray delimiters, trimmed whitespace, collapsed internal newlines, and re-evaluated all 6,935 items. After correction, FineWeb-Edu gains range from −1.2 to +2.3 points across the eight models, while the mean news CPT gain on Daily Oracle increases from +4.5 to +5.5 points (Table 9).

Corrected and expanded splicing evaluations

  • Wikipedia (Table 16). We fixed a bug in our modified OLMES suite affecting OLMo-2-1B. After correction, MLP-only splicing retains 88.7% of full-CPT gains on Wikipedia after-cutoff questions (previously 92.6%).
  • Daily Oracle (Table 16, DO column). On this new evaluation, MLP-only splicing retains 93.9% of full-CPT gains, confirming that most acquired knowledge is preserved on both evaluations.
  • Downstream (Table 18). We evaluated the spliced checkpoints on all downstream tasks. Splicing reduces the average drop from 1.38 to 0.85 points. Separately, updated few-shot examples revise the estimated recovery of the MMLU drop from 42.6% to 45.8%.

Additional evaluations (Section I, Tables 20–21)

Per reviewer y2V7's suggestion, we added four question banks derived from Wikidata, FineWeb news and non-news pages, and FineWeb-Edu news pages to test how evaluation content affects our findings. FineWeb-Edu checkpoints outperform CC-News by 1.7 points on FineWeb-Edu-derived questions, while differences on FineWeb-derived questions are within noise. On independently constructed Wikidata questions, CC-News outperforms FineWeb-Edu in all eight models by 7.8 points on average, and FineWeb by 7.2 points. With the Wikipedia and corrected Daily Oracle results, this shows CC-News is particularly effective for acquiring post-cutoff knowledge of recent world events.

Presentation improvements

Per reviewer dvjG's suggestion, each experimental section is now titled by its research question and opens with a two-sentence summary of the findings, making the main answers easier to locate before the experimental details.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewers and AC,

We sincerely thank you for your time and constructive feedback on our submission.

As the discussion period closes in today, we would like to kindly follow up on our responses. We have replied to each review individually, and we would be grateful if you could let us know whether our responses address your concerns. If any questions remain, we would be glad to provide further clarification or additional results in the remaining time.

Thank you again for your effort and consideration.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.