- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies temporal knowledge updating through continued pretraining on news. It introduces a timestamped, globally deduplicated corpus of 205 million news articles and dated factual evaluations. Eight models from three families are trained on 6 billion English news tokens and compared with base models and matched-token-budget FineWeb-Edu controls. The authors report improved post-cutoff factual accuracy, an association between factual exposure and acquisition, limited propagation to related questions, and degradation on broader capability benchmarks. Post-hoc weight-splicing experiments suggest that combining CPT-updated MLPs with the remaining base-model weights preserves much of the Wikipedia factual accuracy gain while partially reducing MMLU degradation.
Critical strengths
The paper provides a substantial empirical resource for studying temporal knowledge updates using real news. It constructs a timestamped, globally deduplicated corpus of 205 million articles and evaluates eight models from three families against base models and matched-token-budget FineWeb-Edu controls. The coverage of three factual benchmarks, including the independently collected Daily Oracle, strengthens the empirical breadth of the study. The consistent post-cutoff improvements across the tested models and benchmarks provide useful evidence about the benefits and costs of this update corpus (Section 2; Table 9).
Critical weaknesses
-
The quantitative evidence supporting the central MLP-splicing result is internally inconsistent. Table 16 explicitly states that it uses the main-results protocol and the full Wikipedia-after set, yet its full-CPT gains differ from those implied by Table 9: 11.77 versus 12.19 percentage points for Llama-3.1-8B, and 2.79 versus 3.45 for OLMo-2-1B. Figure 25 introduces a further discrepancy, showing approximately 1.8 points for the latter model. Consequently, the aggregate gain retained by MLP splicing is approximately 91.2% using Table 16, compared with the reported 92.6%. These inconsistencies require reconciliation of checkpoint identities, evaluation settings, and reported results before the magnitude of this key finding can be assessed confidently.
-
The MLP analysis lacks matched controls needed to establish its mechanistic interpretation and practical significance. Appendix H constructs the hybrid model by retaining base-model values for all non-MLP parameters, so the experiment does not isolate which removed updates account for the recovery. Furthermore, the interpolation baseline is reported on Daily Oracle (Figure 17), while the main splicing result uses Wikipedia-after and MMLU. A comparison on identical evaluations, at matched factual acquisition or capability retention, is needed to establish the benefit of module selection beyond simply reducing the overall update. This matters particularly because Appendix H.1 acknowledges larger acquisition losses on Daily Oracle and new downstream regressions for some models, without providing complete per-model results for these expanded evaluations.
Questions for authors
-
Can you reconcile the numerical inconsistencies in the MLP analysis? For OLMo-2-1B, the full-CPT Wikipedia-after gain is 3.45 percentage points according to Table 9, 2.79 in Table 16, and approximately 1.8 in Figure 25. Please clarify the checkpoint identities, question sets, and scoring configurations used in these results, and provide consistent base/full-CPT/MLP-only accuracies together with the calculations underlying the reported 92.6% gain retention and 42.6% MMLU recovery. Resolving these discrepancies would materially increase confidence in the central quantitative finding.
-
Does MLP-only splicing provide a better acquisition–retention tradeoff than uniform weight interpolation under matched evaluation conditions? A focused comparison using existing checkpoints on Wikipedia-after, Daily Oracle-after, and MMLU would be informative, particularly at comparable factual gains or comparable MMLU losses. Please also provide the complete per-model results underlying Appendix H.1, including the reported regressions. These results would help establish whether the benefit of module selection is robust across evaluations and exceeds that obtainable by simply scaling down the full update.
-
Which reverted parameter groups account for the recovery observed with MLP-only splicing? Appendix H defines this intervention as keeping all non-MLP parameters at their base-model values. Could you specify the exact parameter groups involved and, for a small representative subset of models, compare it with an intervention that starts from the full CPT checkpoint and restores only the attention projections, leaving all other parameters at their CPT values? This would help distinguish the contribution of attention rollback from other parameter reversions and strengthen the interpretation of the module-wise results.
4 comments
Response to Weakness 2
Matched controls (Weakness 2.1 and Question 3). Thank you for identifying this ambiguity. The MLP-only splice retains the CPT values of gate_proj, up_proj, and down_proj in every layer and restores all remaining parameters to their base-model values. The reverted groups are: (i) token embeddings and the output head (embed_tokens and lm_head, respecting weight tying where present); (ii) normalization parameters, including attention Q/K normalization where present; (iii) attention query/key projections (q_proj, k_proj); and (iv) attention value/output projections (v_proj, o_proj).
To distinguish attention rollback from other modules' reversions, we compare interventions that start from the full CPT checkpoint and restore only the selected group to base, leaving every other parameter at its CPT value. The table below reports means over the eight CC-News checkpoints; both metrics are changes relative to full CPT within this experiment.
| Parameters restored to base | MMLU change (pp) | Wikipedia-after change (pp) |
|---|---|---|
| Embeddings and output head | +0.07 | −0.21 |
| Normalization | +0.01 | −0.30 |
| Q, K projections | +0.17 | −0.23 |
| V, O projections | +0.58 | −0.67 |
| All attention projections (Q, K, V, O) | +0.74 | −0.79 |
Restoring only the attention projections therefore roughly reproduces the MMLU recovery of the full non-MLP reset (0.74 vs 0.85 pp), while sacrificing slightly less post-cutoff Wikipedia accuracy (0.79 vs 0.92 pp). Among the individual groups, restoring V, O produces the largest average MMLU recovery. Restoring embeddings/output heads or normalization alone has a much smaller average effect.
These results support the interpretation that reverting attention-projection updates is sufficient to obtain comparable average MMLU recovery while retaining CPT values for embeddings, normalization, and MLP parameters. The effects vary across models and need not combine additively, so we do not interpret them as a unique decomposition of forgetting or as evidence that attention-frozen training would yield the same outcome.
Practical significance (Weakness 2.1). We distinguish the practical implications of our findings from the stronger mechanistic interpretation. Selective module splicing can recover capabilities affected by CPT while preserving much of the acquired knowledge, providing a way to adjust the acquisition–retention trade-off after training without repeating CPT. The new MLP-only training experiments summarized in our general response extend this evidence to a training-time intervention, with both factual acquisition and downstream capability retention evaluated. Together, these findings support post-hoc module selection and MLP-only training as promising interventions, without establishing a unique mechanistic explanation or replacing matched comparisons with whole-model interpolation on identical evaluations.
Splicing on common evaluations (Weakness 2.2). We have now evaluated all four splice configurations on after-cutoff Daily Oracle in Table 16. MLP-only splicing retains 93.9% of the aggregate full-CPT acquisition gain on Daily Oracle and 88.7% on Wikipedia-after, compared with 15.7% and 21.4% for attention-only splicing. This supports a consistent concentration of factual acquisition in MLP updates across both benchmarks.
Appendix H.1's earlier claim of larger acquisition losses on Daily Oracle arose from an answer-formatting artefact, described in our general response; the aggregate pattern reverses after correction. We have revised that discussion and provide complete per-model results in Table 16 for Daily Oracle and Table 18 for the expanded downstream suite. Splicing reduces the average downstream change from base from −1.38 to −0.85 points, although some model-specific regressions remain.
MLP-only splicing vs. weight interpolation (Weakness 2.2 and Question 2). Tables 16 and 17 now compare module splicing and uniform weight interpolation on Wikipedia-after, Daily Oracle-after, and MMLU under the same evaluation conditions. At comparable mean MMLU losses, MLP-only splicing and interpolation at α=0.3 produce changes of −1.01 and −0.89 percentage points, and splicing achieves larger after-cutoff gains on both Wikipedia (+7.28 versus +6.86 pp) and Daily Oracle (+5.14 versus +4.41 pp). The ordering also holds at matched factual gains, where reaching splicing's gains requires α<0.3 and a larger MMLU loss. Interpolation instead offers a tunable operating point that splicing does not: α=0.5 reduces the MMLU loss to −0.40 pp at smaller gains (+5.35 and +3.29 pp). Module selection therefore preserves more acquisition at a given capability loss, while interpolation provides continuous control over where on the trade-off a practitioner lands.
Thank you for the detailed rebuttal. The numerical corrections and additional experiments have addressed several of my original concerns.
I noticed that you added MLP-only training experiments in Appendix J, which seem to achieve comparable or even better results than post-hoc MLP splicing. This makes me wonder:
-
What is the practical advantage of post-hoc MLP splicing? If training only the MLPs already achieves a better acquisition–retention trade-off, why would we prefer full CPT followed by weight splicing?
-
Could standard PEFT methods achieve similar results? For example, would LoRA or adapters also allow the model to acquire new knowledge while preserving existing capabilities? Without such baselines, it is unclear whether the benefits are specific to MLP updates or simply arise from restricting the parameter updates.
One minor remaining concern is that Tables 16 and 17 use different few-shot examples. Could you clarify whether the advantage over weight interpolation holds under identical evaluation settings?
Q1. What is the practical advantage of post-hoc MLP splicing?
We agree with the reviewer that, given our new results, MLP-only training is the better-supported choice when planning a new training run. This finding emerged from additional experiments conducted during rebuttal, motivated by our post-hoc splicing results. Splicing showed that retaining only the MLP updates preserves most factual gains while reducing downstream degradation, which prompted us to test whether restricting training to MLPs could achieve an even better trade-off. We therefore view splicing's primary contribution as an experimental intervention that motivated and guided this training strategy, rather than as a preferred alternative to MLP-only training. Splicing also retains a practical advantage when a full-CPT checkpoint is already available: it can improve the acquisition–retention trade-off without further training, whereas adopting MLP-only training at that stage would require a new training run.
Q2. Could LoRA or adapters achieve a similar acquisition–retention trade-off?
Biderman et al. [1] found that low-rank LoRA learned less and forgot less than full CPT, with similar forgetting at comparable levels of acquisition. In a time-incremental setting closer to ours, Öncel et al. [2] found that LoRA on attention and MLP projections at ranks 128 and 256 recovered most of full CPT's after-cutoff Daily Oracle gain on Llama-3.2-3B, while before-cutoff accuracy stayed close to baseline. Given our limited rebuttal time, we are unable to empirically evaluate this. We will include such a comparison in our camera-ready version.
Minor concern. Does the advantage over interpolation hold with identical few-shot examples?
Yes, it does. We re-evaluated interpolation using the same pipeline, questions, and few-shot examples as splicing. The table reports changes from each base model in percentage points (pp), averaged across Llama-3.2-3B, OLMo-2-7B, and Qwen3-4B:
| Method | MMLU | Wikipedia (after cutoff) | Daily Oracle (after cutoff, corrected) |
|---|---|---|---|
| Full CPT (α = 0) | −2.23 | +8.41 | +6.16 |
| MLP-only splicing | −0.99 | +7.52 | +5.70 |
| Interpolation (α = 0.3) | −1.19 | +6.85 | +5.24 |
| Interpolation (α = 0.5) | −0.66 | +5.07 | +3.80 |
| Interpolation (α = 0.7) | −0.35 | +3.11 | +2.37 |
Splicing outperforms interpolation at α = 0.3 on all three metrics on average. This also holds individually for Llama and OLMo. On Qwen, splicing improves both acquisition scores at an additional MMLU cost of 0.27 pp. At α = 0.5, interpolation retains 0.33 pp more MMLU accuracy on average, but gains 2.45 pp less on Wikipedia and 1.90 pp less on Daily Oracle than splicing.
References:
- [1] Biderman et al., LoRA Learns Less and Forgets Less, TMLR 2024.
- [2] Öncel et al., Time-Incremental Continued Pretraining of LLMs: Knowledge Updates Without Catastrophic Forgetting, arXiv:2609.23916.
We thank the reviewer for the careful review and the detailed, actionable questions, which helped us correct and strengthen the splicing analysis. We address each point below.
Response to Weakness 1 and Question 1
We are grateful to the reviewer for identifying these inconsistencies. For OLMo-2-1B, Table 16 contained a copy-paste error (+2.79 instead of +1.79 percentage points); the remaining discrepancy with Table 9 (+1.79 versus +3.45) arose from the OLMES scoring-protocol bug described in our general response. The Llama-3.1-8B discrepancy comes from re-scoring with different few-shot examples. We have aligned the scoring protocol with Table 9 and regenerated the affected results. Checkpoint identities and base/full-CPT/MLP-only accuracies are provided in Table 16 for Wikipedia-after and Daily Oracle; Figure 25 presents both evaluations. Corrected MLP-only gain retention is 88.7% on Wikipedia-after and 93.9% on Daily Oracle, while the revised MMLU recovery is 45.8%. Please see our general response for details of the evaluation corrections underlying these revised estimates.
The retention figures of 88.7% and 93.9% are calculated by dividing the summed accuracy gains over base of the MLP-only spliced models by the corresponding summed gains of the full-CPT models, on Wikipedia-after and Daily Oracle, respectively. The 45.8% MMLU recovery is calculated by dividing the summed accuracy improvement from full CPT to MLP-only splicing by the summed accuracy drop from base to full CPT. Each calculation uses the same checkpoint set in its numerator and denominator.