acceptodds
Under review as a conference paper at ICLR 2027

Replay on Demand: An Emergent Curriculum for Balancing Adaptation and Forgetting in Continued Pretraining

Abstract

Continued pretraining enables language models to adapt to new domains and knowledge, but often at the cost of forgetting previously acquired capabilities. Replay can mitigate this trade-off, but fixed replay mixtures allocate training independently of the model's actual retention needs. This is particularly limiting because forgetting varies across capabilities and data sources and evolves throughout training. We introduce *Replay on Demand* (RoD), which instead derives the replay allocation from the model's learning dynamics. RoD jointly prioritizes adaptation samples by their remaining learning potential and replay samples by their observed forgetting. Their competition for a shared training budget yields an online curriculum that determines what to train on at each step. Across models, scales, and adaptation domains, RoD reaches or improves upon the adaptation–forgetting frontier of tuned fixed-replay baselines and model merging without prescribing a replay allocation in advance. Replay concentrates on sources that are more vulnerable to forgetting and dynamically increases and redistributes as forgetting emerges during training. Together, our results show that replay can be allocated online from the model's evolving state, targeting *what is needed, when it is needed*.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continued pretraining, continual learning, catastrophic forgetting, experience replay, data selection, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pXELOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Replay on Demand (RoD) for continued pretraining. Adaptation candidates are scored by their remaining reducible loss relative to an adaptation specialist, replay candidates by their loss increase relative to the base model, and a joint top-k over a balanced pool selects each batch, so the replay share and composition emerge during training. Experiments on Legal and German adaptation with Nemotron-12B (native replay) and Qwen3.5-9B/4B (proxy replay) compare RoD with fixed-replay sweeps and model merging, and transfer curricula from smaller models to 30B and 35B targets.

Critical strengths
  • Simple rule. One top-k over two reducible-loss scores decides both how much and what to replay, and the reference losses are cached, so the method is easy to add to a standard pipeline.
  • Clear native-replay result. On German adaptation of Nemotron-12B, RoD adapts better than every fixed-replay arm at about the forgetting level of the 32% arm, and replay concentrates on the most vulnerable sources.
Critical weaknesses

1. Small single-seed margins. Against the fixed arm closest to RoD's emergent share (Table 3), the gaps are a few thousandths of a nat:

Setting (RoD vs. fixed share)Adaptation ↓Val. forgetting ↓Task forgetting ↓
Nemotron-12B, Legal (21% vs. 20%)0.967 vs. 0.9700.071 vs. 0.0820.049 vs. 0.055
Qwen3.5-9B, German (16% vs. 17%)1.790 vs. 1.7960.028 vs. 0.0350.061 vs. 0.052
Qwen3.5-4B, German (18% vs. 17%)1.859 vs. 1.8630.028 vs. 0.0360.107 vs. 0.090

All runs use a single seed, and the main forgetting metric (Eq. 34) is closely aligned with RoD's own replay score (Eq. 3); on the independent task metric, RoD trails the nearest fixed arm in both German Qwen settings.

2. Unclear source of the gain. In Fig. 5a, selecting replay by its score at a fixed share gives about the same trade-off as random selection, and forgetting drops only once the share is freed, which also changes when replay happens (Fig. 4). A control that follows RoD's share trajectory with randomly drawn replay is missing, as is an online data-mixing baseline such as [1] or [2].

Typos. L182 "Figure 1b" → "Figure 1 (right)"; L198 "(nem, 2025)" → "(NeMo-RL, 2025)"; L643 "Progress compress" → "Progress & Compress"; L672 "Binbinliu" → "Binbin Liu".

Questions for authors
  1. Please report the total compute of RoD, including the no-replay reference and candidate scoring, since the Table 3 caption says compute-matched while Sec. 4 describes token-matched runs, and clarify the model, domain and fixed share used in Fig. 5a.
  2. The share-trajectory control with random replay and two or three seeds for RoD and the nearest fixed arm should be reported, together with a comparison to upweighting the adaptation samples that the base model already fits well [3] in the proxy-replay setting.

References

[1] Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws. ICLR 2025.
[2] Aioli: A Unified Optimization Framework for Language Model Data Mixing. ICLR 2025.
[3] Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting. ICML 2025.

4 comments

Thank you for the detailed feedback and for highlighting RoD’s simple selection rule and the strong native-replay results on German Nemotron-12B. We address your concerns and report the additional ablations and baseline comparisons below.


Single-seed comparisons and capability retention

We appreciate the close reading of these comparisons and agree that small margins warrant caution when each configuration is run with a single seed. We chose this design to evaluate RoD across model families, adaptation domains, and replay sources within our compute budget. The standalone RoD and fixed-replay sweeps across the five main settings in Figure 2 consumed approximately 20,000 GPU-hours in total. Extending this full comparison to three seeds would require approximately 40,000 additional GPU-hours, equivalent to USD 134,000–170,000 at an assumed B200 rental rate of USD 3.35–4.25 per GPU-hour. Given this high cost of repeating full training runs, single-seed comparisons are a recurring practice in large-scale CPT studies [1, 2].

Nevertheless, we agree that small differences from single-seed runs warrant caution. We therefore assess the performance of RoD relative to the fixed-replay frontier across the five different settings. Our main finding, that RoD reaches or improves upon this frontier (Sec. 4.1), holds across all settings. Matching the frontier is meaningful because RoD finds a competitive operating point without sweeping replay ratios. In the three nearest-share comparisons you highlight, RoD also achieves lower adaptation loss and lower validation-loss forgetting. Prior CPT replay comparisons report loss differences of comparable magnitude [2, Table 2].

We agree that our replay score and validation-loss forgetting metric are closely aligned. The held-out evaluation tests whether gains extend beyond the examples selected for replay to unseen sequences from the replay distribution. We assess capability retention separately through task evaluations. In both German Qwen settings, RoD achieves lower adaptation loss and lower validation-loss forgetting than 17% fixed replay, while aggregate task-based forgetting is slightly higher. The capability breakdown for Qwen3.5-9B shows that the gap is concentrated in reasoning-intensive tasks, with RoD retaining parametric knowledge about as well as fixed replay (Appendix C.2).

We have clarified in Section 4.1 that the frontier comparisons use one seed per configuration and assess consistency across five settings, with a pointer to Appendix B.3. Section 4 now distinguishes retention on held-out replay data from task-based capability retention, and Section 4.1 explicitly reports the German Qwen trade-off, with a capability breakdown in Appendix C.2.

Contributions of replay scheduling and sample selection

Thank you for suggesting a control that tests what score-based replay selection adds when the amount and timing of replay are held fixed. We agree that this ablation would be useful to further disentangle the source of RoD’s gain. Figure 5a uses Nemotron-12B on Legal, with the fixed-share arms set to 7.2% replay, the average replay share of the full RoD run (m=4). We ran the new control in the same setting, matching RoD’s replay share at each training step and drawing replay examples at random.

MethodAdaptation loss ↓Mean validation forgetting ↓Task-based forgetting ↓
RoD (m = 4)0.97340.07990.049
Matched replay schedule, random selection in both streams0.98250.07420.060

Random replay on RoD’s schedule comes close to the overall trade-off and achieves lower mean validation forgetting. It reduces forgetting on code and reasoning/math data, roughly matches RoD on world knowledge, and forgets more on multilingual data. RoD achieves better adaptation and lower task-based forgetting. The random-replay control underperforms on 16 of 21 knowledge tasks and 12 of 13 multilingual tasks, with a tie on the reasoning group. Its mean paired disadvantage is 1.1 percentage points (SE 0.4 points across tasks).

Why does the mix matter? Random draws reproduce the candidate pool’s source proportions in expectation, corresponding to a sampling ratio of 1.0. RoD oversamples sources vulnerable to forgetting, as illustrated in Figure 3b and in the Legal setting in Appendix C.3.3. Multilingual data are a clear example: RoD oversamples them in the Legal analysis, and the matched-trajectory control shows greater multilingual forgetting even though it follows the same overall replay schedule. The schedule captures much of the aggregate adaptation–retention trade-off. Score-based selection helps distribute that retention across capabilities, with multilingual retention showing the clearest benefit in this control.

We have added the matched-schedule control to Figure 5a and clarified the model, domain, and fixed replay share in Section 4.3. This section now also explains that following RoD’s replay schedule with random selection in both streams recovers much of the overall trade-off, while full RoD achieves better adaptation and task retention at the same schedule. Appendix C.4 (Table 6) breaks down forgetting by source and capability.

Online data mixing. We evaluated Aioli [3] on Nemotron-12B Legal as an online mixing baseline. We used Legal and general replay as the two groups, with uniform sampling within each group. Aioli started at a 50/50 mixture and ran for 2,960 steps, close to the 2,958-step budget of the frontier runs. All probe steps counted toward that budget, and the probe set was disjoint from the reported validation sets.

MethodReplay allocationAdaptation loss ↓Validation forgetting ↓Task-based forgetting ↓
Fixed replay (50%)50%0.9920.0240.036
Aioli50% → 84% (67% mean)1.0330.0350.036
RoD21%0.9670.0710.049

Aioli shifts toward a replay-heavy mixture. Despite allocating more replay on average, it has higher adaptation loss and higher forgetting than fixed 50% replay. RoD reaches a different operating point, with better adaptation and higher forgetting than either comparator at a substantially lower replay share. The matched-trajectory ablation above complements this comparison by showing the contribution of sample selection when the replay schedule is held fixed.

We have included the Aioli comparison in Appendix C.5 (Table 7) of the revised manuscript, together with the experimental setup and a discussion of the resulting adaptation–retention trade-offs.

Typographical corrections

Thank you for catching these errors. We have corrected all four in the revised manuscript.


Computational overhead and comparison budgets

Thank you for catching the inconsistency. We have corrected Table 3’s caption to specify that the runs are trained-token-matched, consistent with Section 4. Appendix D.1 (Table 8) now reports normalized B200 GPU-hours, including training the no-replay adaptation reference, precomputing reference losses, and scoring candidates online.

The appropriate replay ratio is generally unknown before training. To obtain the operating points shown on the fixed-replay frontier in Figure 2, we ran a sweep and selected favorable ratios retrospectively. Including the cost of that search, RoD uses 0.72–0.83× the compute of the evaluated fixed-replay campaign, a 17–28% reduction, even after accounting for its reference and scoring overhead. RoD thus finds a competitive adaptation–retention trade-off at lower cost than the evaluated replay-ratio search.

When a suitable replay ratio is already available, a single fixed-replay run remains cheaper: RoD costs 2.85–3.27× as much across the directly scored settings. The search-campaign savings therefore concern the cost of identifying a competitive allocation and do not establish performance superiority under equal total compute.

Section 4.4 shows how curriculum transfer reduces the overhead at larger scales. For German Qwen3.5-35B, constructing the curriculum at 4B and training the 35B model on it costs 0.65× the evaluated sweep, a 35% reduction, including curriculum construction. This corresponds to 2.60× a single fixed-replay run. Curriculum transfer moves candidate scoring off the target model while preserving a competitive adaptation–retention trade-off.


Comparison with easy-sample weighting

Thank you for suggesting the easy-sample weighting baseline. We evaluated FLOW [4] on German Qwen3.5-4B. We kept the data, learning-rate schedule, and trained-token budget fixed and weighted each adaptation block’s loss by wi=exp⁡(−ℓi/τ)w_i = \exp(-\ell_i/\tau), where ℓi\ell_i is the block’s mean token loss under the base model and τ\tau is the median of these block losses over the adaptation dataset. FLOW uses no replay data.

MethodAdaptation loss ↓Validation forgetting ↓
No replay1.8490.521
FLOW1.8640.521
Fixed replay (17%)1.8630.036
RoD1.8590.028

FLOW and unweighted no-replay training follow nearly identical general-loss trajectories and both end at 0.521 validation forgetting. Sanyal et al. [4] found that FLOW improved general task retention when they used supervised fine-tuning to adapt base models to math reasoning on MetaMathQA. Our setting involves continued pretraining on roughly 12B tokens of German text. Here, a low base-model loss identifies blocks that are easier to predict within the German corpus, and training on those blocks can still shift the model away from the general distribution. This may explain why reweighting alone had little effect on retention in our setting. General-domain replay directly exposes the model to that distribution during training, reducing validation forgetting to 0.036 with 17% fixed replay and 0.028 with RoD.


We hope the additional ablations, baseline comparisons, and clarifications address your concerns. If any point remains unresolved or needs further explanation, please let us know.

References

[1] Ibrahim, A., et al. (2024). Simple and Scalable Strategies to Continually Pre-train Large Language Models. TMLR.

[2] Tong, X., et al. (2026). SELECting over Tokens: Curating Pre-training Data at Scale via Token Classification. ACL.

[3] Chen et al. (2025). Aioli: A Unified Optimization Framework for Language Model Data Mixing. ICLR.

[4] Sanyal et al. (2025). Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting. ICML.

Thank you again for your time and detailed feedback. Following your suggestion, we have now completed additional seed experiments and would like to share the results.

Consistency across seeds and capability retention

During the rebuttal period, we ran a second seed for both RoD and the nearest fixed-replay baseline (17%) on German Qwen3.5-4B:

MethodSeedAdaptation loss ↓Validation-loss forgetting ↓Task-based forgetting ↓
RoD421.85920.02760.107
RoD441.85830.02680.105
Fixed replay (17%)421.86260.03600.090
Fixed replay (17%)441.86890.03850.101

The results reproduce the same adaptation-retention trade-off in both seeds: RoD achieves lower adaptation loss and lower validation-loss forgetting, while fixed replay achieves lower aggregate task-based forgetting. The magnitude of the task-retention gap varies, however, narrowing from 1.7 to 0.4 percentage points in the additional seed.

RoD’s results are closely matched across the two seeds: adaptation loss is 1.8592 and 1.8583, validation-loss forgetting is 0.0276 and 0.0268, and task-based forgetting is 10.7 and 10.5 percentage points. The small differences between these runs provide additional evidence of stability across seeds in this setting. Together with the consistent adaptation–retention trade-offs observed across model families, scales, and domains, these results further support the robustness of our findings.


We hope these additional results and our earlier responses address your concerns. If any questions or concerns remain unresolved, please let us know. We would welcome the opportunity to discuss them during the remaining discussion period.

Reviewer_ujfFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Replay on Demand (RoD), a method for continual pretraining that dynamically balances adaptation to new domains against forgetting previously acquired knowledge. Experiments cover legal and German continued pretraining using Nemotron and Qwen model families, with model scales from 4B to 35B parameters. RoD is compared against no-replay training, fixed-replay ratios, model merging, and a higher-budget max-replay baseline. Results show that RoD generally matches or improves upon the adaptation-forgetting frontier without requiring replay-ratio tuning.

Critical strengths
  1. The proposed approach in this paper is well-motivated. It's a simple and principled mechanism for adaptive replay allocation. The paper formulates adaptation and retention as a unified data-selection problem, using source-specific reducible-loss signals to let adaptation and replay examples compete for a shared training budget. This jointly determines both the amount and composition of replay without prescribing a fixed replay ratio. This is a substantive contribution because it replaces a manually tuned allocation strategy with an online mechanism driven by the model’s evolving learning and forgetting states.
  2. This paper presents strong empirical results. The paper evaluates RoD across two model families, legal and German adaptation, native and proxy replay, and model scales from 4B to 35B parameters. The results show competitive or improved adaptation-forgetting trade-offs relative to fixed-replay baselines, while analyses of replay allocation demonstrate that the method preferentially revisits knowledge categories more vulnerable to forgetting.
Critical weaknesses
  1. This paper doesn't not establish a compute-matched advantage. Although RoD and fixed-replay baselines use matched trained-token budgets, RoD incurs additional online forward passes for candidate scoring and offline costs for constructing the adaptation reference model. Consequently, the reported frontier improvements do not establish that RoD achieves a better trade-off under equal total computational resources.
Questions for authors
  1. This is not a weakness of this paper, but I'm curious about do you think the proposed approach will apply to RL post-training?
3 comments

Thank you for the constructive feedback and for highlighting RoD’s unified approach to adaptation and retention, as well as the breadth of the empirical evaluation. We address your compute concern and discuss how the approach might extend to RL post-training below.

Computational overhead and comparison budgets

Thank you for highlighting this distinction. Figure 2 compares RoD and fixed-replay baselines at matched trained-token budgets. To assess their computational costs, it is useful to distinguish the cost of identifying a suitable replay allocation from the cost of training with an allocation already chosen. We have added both comparisons, together with normalized B200 GPU-hour estimates and the accounting methodology, in Appendix D.1, Table 8.

Comparison with replay-ratio search. The appropriate replay ratio is generally unknown before training. Figure 2 therefore compares RoD with fixed-replay operating points identified retrospectively through a sweep. Including the cost of that search, RoD uses 0.72–0.83× the compute of the evaluated fixed-replay campaign, a 17–28% reduction, even after accounting for adaptation-reference training, reference-loss precomputation, and online candidate scoring. RoD thus finds a competitive adaptation–retention trade-off at lower cost than the evaluated replay-ratio search.

When a suitable replay ratio is already available, a single fixed-replay run remains cheaper: RoD costs 2.85–3.27× as much across the directly scored settings. The search-campaign savings therefore reflect the cost of identifying a competitive allocation; they do not establish performance superiority under equal total compute.

Reducing costs through smaller models. Section 4.4 demonstrates two complementary approaches. A 4B adaptation reference recovers a similar trade-off to a 9B reference for Qwen3.5-9B, reducing offline reference costs. Curriculum transfer moves selection to a smaller model, eliminating target-scale candidate scoring. For German Qwen3.5-35B, including curriculum construction and target training, this costs 2.60× a single fixed-replay run and 0.65× the evaluated sweep, while retaining a competitive adaptation–retention trade-off.

These measured savings are obtained at moderate model scales, while the relative overhead of curriculum construction decreases as target-model training becomes more expensive. Using standard (C\approx6ND) FLOP accounting (Kaplan et al., 2020; Hoffmann et al., 2022), RoD on a dense 400B target with the curriculum of a 35B model would cost approximately 1.26× a single baseline run and 0.32× a four-run sweep.

We have corrected the experimental description, Figure 2 caption, and remaining budget references to distinguish trained-token matching from total-compute matching. The expanded Appendix D.1 now explicitly reports both standalone overhead and search-campaign costs.

Potential extension to RL post-training

Thank you for this interesting question. We think the principle behind RoD could extend to RL post-training. Related work already connects reducible-loss selection to post-training: DavIR [1] adapts RHO-based scoring to supervised fine-tuning data selection and develops a related objective for direct preference optimization. This provides a useful precedent, although extending RoD’s joint allocation of adaptation and retention to online RL remains an interesting open question. The key is finding signals that distinguish what the model still needs to learn from what it has stopped doing well.


We hope the added analyses clarify RoD’s computational costs and the scope of our comparisons. If any concerns remain unresolved, or if you have further questions about the potential extension to RL, please let us know.

References

[1] Zhou et al. (2025). DavIR: Data Selection via Implicit Reward for Large Language Models. ACL.

Thank you again for your time and constructive feedback. Does the added compute analysis address your concern about the comparison budgets, or are there aspects of the accounting or interpretation that remain unclear? We would be happy to discuss these or any further questions during the remaining discussion period.

Thank you for providing details on the compute analysis. It addressed my concern, and therefore I will keep my positive score of 3.

Reviewer_PPhWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper explores data selection/curriculum to mitigate forgetting. It provides a fairly intuitive metric that combines existing ideas of reducible holdout loss. The main contribution is to include terms that account for both adaptation and forgetting in the data selection. Doing this adaptively across epochs also allows an adaptive split between focusing on learning vs adaptation. Across a variety of experiments, they find that their method (RoD) achieves better pareto frontier than fixed replay in many cases.

Critical strengths
  • I found the paper fairly easy to understand and intuitively written
  • The paper has a lot of experiments and ablations and considers all important baselines etc
  • The results show that their simple method can offer improvements in various settings.
Critical weaknesses

The strongest weakness is the fact that the benefits are not present in all settings/ all tasks. There are many important cases where the baseline of fixed replay outperforms their method. While acknowledged and results present in the appendix, there is no clear exposition of when exactly this works and why. The paper only makes a cursory explanation that the proxy loss is less aligned with downstream capabilities. But that's exactly the non trivial question. What makes something a good proxy loss? Of course, if we had a good proxy loss, choosing points based on that would help. Without a clear discussion of this, the paper essentially takes existing ideas, presents one fixed combination of the losses and reports that it helps in some cases. This feels incomplete as a contribution

  • What if you didn't have access to the base model?
  • Why fix 1:1 weight on forgetting vs. learning?
  • (Minor) Loss based evals make it a bit harder to contextualize how significant the gains are
Questions for authors

See weaknesses above

3 comments

Thank you for the constructive feedback and for highlighting the paper’s clarity, the breadth of the experiments, and the gains RoD achieves with a simple selection rule. We address your concerns and questions below.


Performance trade-offs and the reliability of proxy replay

You raise two closely related concerns: RoD does not improve every metric in every setting, and the paper needs to explain more clearly when replay loss provides a useful signal of capability retention. We agree that both require clearer discussion.

Comparing adaptation and forgetting jointly. Across the evaluated settings, no tested fixed-replay configuration outperforms RoD jointly on adaptation and forgetting (Figure 2). The cases highlighted by the reviewer involve lower forgetting at the cost of weaker adaptation. In German Qwen3.5-9B and Qwen3.5-4B, for example, the nearest fixed-replay arm achieves slightly lower task-based forgetting, while RoD achieves slightly lower adaptation loss. These results place RoD on the non-dominated frontier: fixed replay offers alternative trade-offs, but does not dominate RoD.

Why matching the frontier matters. The appropriate fixed replay ratio for a given setting is unknown before training. Comparing RoD against the best fixed-replay run selected retrospectively from the sweep is therefore an oracle-like comparison. RoD determines its allocation in a single run without prescribing a replay ratio, yet matches or improves upon the trade-offs obtained through this sweep.

What drives RoD’s gains? Beyond the aggregate results in Section 4.1, Sections 4.2 and 4.3 investigate how replay allocation and sample selection contribute to RoD’s performance. Section 4.2 shows that RoD preferentially replays categories most vulnerable to forgetting and adjusts both the overall replay share and its source composition over time. During the rebuttal period, we added a matched-schedule ablation, now reported in Section 4.3, to disentangle the contributions of replay scheduling and sample selection. This control shows that following RoD’s replay schedule with random selection in both streams recovers much of the overall trade-off, while RoD’s sample selection further improves adaptation and task-based retention at the same schedule.

What makes a good proxy loss? The guiding principle is to select replay data representative of the knowledge and capabilities one aims to preserve, so that degradation of those capabilities is reflected in increased replay loss. When original pretraining data are unavailable, this suggests assembling a proxy corpus with relevant domain and capability coverage, then assessing loss changes against independent capability evaluations. Such assessments can identify coverage or signal-alignment gaps and guide refinement of the corpus.

We agree that the original presentation did not make these distinctions sufficiently explicit. We have therefore revised Section 4.1 to define the joint comparison criterion and directly describe the German Qwen task-retention trade-offs. We have also expanded Appendix D.3 with further practical guidance for constructing and assessing proxy replay corpora, including the use of independent development evaluations separate from final test evaluations.

Questions

What if you didn't have access to the base model?

Thank you for this question. In RoD, the “base model” denotes the checkpoint from which continued pretraining starts and whose capabilities we aim to preserve. This does not necessarily need to be the original, pre-instruction-tuning checkpoint.

If only an instruction-tuned model is available, or adapting that model is preferred (e.g., as in [1]), it can serve as both the initialization and the replay reference. RoD’s replay score would then measure increases in loss relative to that starting checkpoint. The formulation remains unchanged, although we have not evaluated this setting separately.

We have clarified the definition of the base model in Section 3.1 of the revised manuscript accordingly.

Why fix 1:1 weight on forgetting vs. learning?

Thank you for this question. We distinguish the relative weighting of the two RHO scores from the proportions of adaptation and replay candidates.

Score weighting. Both scores measure reference-relative differences in the same token-normalized language-modeling loss. Equal weighting therefore provides a natural default: adaptation and retention candidates compete directly according to their remaining learning potential and observed forgetting, without introducing an additional preference parameter. A relative weight could instead express an application-specific preference for adaptation or retention, as discussed in Appendix D.2.

Candidate-pool proportions. A 1:1 split allows either source to fill the entire training batch using the smallest possible candidate pool: twice the batch size. Neither this split nor equal score weighting fixes the selected replay share, which emerges from joint selection and varies throughout training (Figure 4). Increasing either source’s candidate count offers greater selectivity at additional scoring cost. Our pool-size ablations increase selectivity for both sources and show little further improvement (Figure 5b), motivating this efficient default, although asymmetric candidate ratios were not separately tested.

We have clarified the rationale for equal score weighting and distinguished it from the computational motivation for balanced candidate pools in Section 3.3, with a cross-reference to Appendix D.2.

(Minor) Loss based evals make it a bit harder to contextualize how significant the gains are

Thank you for this observation. Using held-out language-modeling loss follows common practice in evaluating pretraining and continued pretraining. Our objective is to adapt a model to a specified corpus while minimizing forgetting, so we measure adaptation through loss on held-out data from that corpus. As explained in Appendix B.5, this directly assesses learning of the target distribution without assuming which downstream Legal or German tasks should benefit. Developing a new measure of adaptation is outside the scope of this work.

To contextualize retention beyond loss, we also report task-based forgetting across four capability groups: parametric knowledge, reasoning and problem solving, commonsense and robustness, and multilingual capabilities (Figure 2, bottom row). For example, on Nemotron-12B Legal, RoD matches the adaptation loss of 10% fixed replay while reducing task-based forgetting from 6.7 to 4.9 percentage points. The loss-based and task-based evaluations therefore provide complementary views of the adaptation–retention trade-off.

RoD’s reference-relative scoring principle is not inherently restricted to language-modeling loss. It could also be instantiated with other training objectives, provided that differences relative to the reference models yield meaningful and comparable signals of remaining learning potential and forgetting. We view such extensions as future work. Our experiments establish its effectiveness for language-model continued pretraining.

We have revised Section 4 and Appendix B.5 to clarify what each evaluation measures and how the loss-based and task-based results together characterize the adaptation–retention trade-off.


We hope these revisions clarify both the benefits and limitations of RoD. If any concern remains unresolved, or if any part of our response needs further clarification, please let us know.

References

[1] Chen, S., Parker, J., Bang, Y., et al. (2026). Thomson: Continual Learning of Frontier Models for SovereignAI. Thomson Reuters.

Thank you again for your time and thoughtful feedback. After reviewing our response and the revised manuscript, do you have any remaining questions or concerns? We would welcome the opportunity to clarify any unresolved points during the remaining discussion period.

Comments on the paper

1 comment

We thank the reviewers and Area Chair for their time and constructive feedback. All three reviewers highlighted RoD’s simplicity and recognized the approach's empirical benefits. PPhW and ujfF also emphasized the breadth of the experiments, while ujfF and pXEL highlighted the unified selection rule that determines both how much and what to replay. In addition, ujfF identified the replacement of manually tuned replay allocation with an online mechanism as a substantive contribution, and pXEL highlighted the strong native-replay result on the German Nemotron-12B. We summarize our main additions and clarifications below.

  • Performance trade-offs and proxy replay (PPhW, pXEL). We clarified that RoD remains non-dominated by the tested fixed-replay configurations when adaptation and forgetting are assessed jointly. We explicitly discuss the German Qwen task-retention gaps and provide guidance on proxy-corpus coverage and signal alignment (Section 4.1; Appendices C.2 and D.3–D.4).

  • Mechanism and baselines (pXEL). The new matched-schedule control shows that replay scheduling captures much of the trade-off, while RoD’s sample selection further improves adaptation and task retention (Section 4.3; Appendix C.4). We also added comparisons with Aioli (Appendix C.5) and FLOW (response to pXEL).

  • Consistency across seeds (pXEL). A second seed for both RoD and 17% fixed replay on German Qwen3.5-4B reproduces the same trade-off. RoD’s outcomes are closely matched across seeds, complementing the consistency across model-domain settings (17 November follow-up to pXEL).

  • Computational costs (ujfF, pXEL). We corrected the budget terminology and quantified reference and scoring costs. RoD uses 17-28% less compute than the evaluated replay-ratio search campaigns, while costing more than a single fixed-replay run. We explicitly distinguish these savings from equal-total-compute performance superiority (Appendix D.1, Table 8). Reviewer ujfF confirmed that this addressed their concern and maintained their positive assessment.

We also clarified the reference-checkpoint and weighting choices, discussed RL extensions, and corrected typos. These additions strengthen the evidence that RoD finds competitive adaptation-retention trade-offs through online replay allocation without a replay-ratio sweep.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.