- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Replay on Demand (RoD) for continued pretraining. Adaptation candidates are scored by their remaining reducible loss relative to an adaptation specialist, replay candidates by their loss increase relative to the base model, and a joint top-k over a balanced pool selects each batch, so the replay share and composition emerge during training. Experiments on Legal and German adaptation with Nemotron-12B (native replay) and Qwen3.5-9B/4B (proxy replay) compare RoD with fixed-replay sweeps and model merging, and transfer curricula from smaller models to 30B and 35B targets.
Critical strengths
- Simple rule. One top-k over two reducible-loss scores decides both how much and what to replay, and the reference losses are cached, so the method is easy to add to a standard pipeline.
- Clear native-replay result. On German adaptation of Nemotron-12B, RoD adapts better than every fixed-replay arm at about the forgetting level of the 32% arm, and replay concentrates on the most vulnerable sources.
Critical weaknesses
1. Small single-seed margins. Against the fixed arm closest to RoD's emergent share (Table 3), the gaps are a few thousandths of a nat:
| Setting (RoD vs. fixed share) | Adaptation ↓ | Val. forgetting ↓ | Task forgetting ↓ |
|---|---|---|---|
| Nemotron-12B, Legal (21% vs. 20%) | 0.967 vs. 0.970 | 0.071 vs. 0.082 | 0.049 vs. 0.055 |
| Qwen3.5-9B, German (16% vs. 17%) | 1.790 vs. 1.796 | 0.028 vs. 0.035 | 0.061 vs. 0.052 |
| Qwen3.5-4B, German (18% vs. 17%) | 1.859 vs. 1.863 | 0.028 vs. 0.036 | 0.107 vs. 0.090 |
All runs use a single seed, and the main forgetting metric (Eq. 34) is closely aligned with RoD's own replay score (Eq. 3); on the independent task metric, RoD trails the nearest fixed arm in both German Qwen settings.
2. Unclear source of the gain. In Fig. 5a, selecting replay by its score at a fixed share gives about the same trade-off as random selection, and forgetting drops only once the share is freed, which also changes when replay happens (Fig. 4). A control that follows RoD's share trajectory with randomly drawn replay is missing, as is an online data-mixing baseline such as [1] or [2].
Typos. L182 "Figure 1b" → "Figure 1 (right)"; L198 "(nem, 2025)" → "(NeMo-RL, 2025)"; L643 "Progress compress" → "Progress & Compress"; L672 "Binbinliu" → "Binbin Liu".
Questions for authors
- Please report the total compute of RoD, including the no-replay reference and candidate scoring, since the Table 3 caption says compute-matched while Sec. 4 describes token-matched runs, and clarify the model, domain and fixed share used in Fig. 5a.
- The share-trajectory control with random replay and two or three seeds for RoD and the nearest fixed arm should be reported, together with a comparison to upweighting the adaptation samples that the base model already fits well [3] in the proxy-replay setting.
References
[1] Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws. ICLR 2025.
[2] Aioli: A Unified Optimization Framework for Language Model Data Mixing. ICLR 2025.
[3] Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting. ICML 2025.
4 comments
Contributions of replay scheduling and sample selection
Thank you for suggesting a control that tests what score-based replay selection adds when the amount and timing of replay are held fixed. We agree that this ablation would be useful to further disentangle the source of RoD’s gain. Figure 5a uses Nemotron-12B on Legal, with the fixed-share arms set to 7.2% replay, the average replay share of the full RoD run (m=4). We ran the new control in the same setting, matching RoD’s replay share at each training step and drawing replay examples at random.
| Method | Adaptation loss ↓ | Mean validation forgetting ↓ | Task-based forgetting ↓ |
|---|---|---|---|
| RoD (m = 4) | 0.9734 | 0.0799 | 0.049 |
| Matched replay schedule, random selection in both streams | 0.9825 | 0.0742 | 0.060 |
Random replay on RoD’s schedule comes close to the overall trade-off and achieves lower mean validation forgetting. It reduces forgetting on code and reasoning/math data, roughly matches RoD on world knowledge, and forgets more on multilingual data. RoD achieves better adaptation and lower task-based forgetting. The random-replay control underperforms on 16 of 21 knowledge tasks and 12 of 13 multilingual tasks, with a tie on the reasoning group. Its mean paired disadvantage is 1.1 percentage points (SE 0.4 points across tasks).
Why does the mix matter? Random draws reproduce the candidate pool’s source proportions in expectation, corresponding to a sampling ratio of 1.0. RoD oversamples sources vulnerable to forgetting, as illustrated in Figure 3b and in the Legal setting in Appendix C.3.3. Multilingual data are a clear example: RoD oversamples them in the Legal analysis, and the matched-trajectory control shows greater multilingual forgetting even though it follows the same overall replay schedule. The schedule captures much of the aggregate adaptation–retention trade-off. Score-based selection helps distribute that retention across capabilities, with multilingual retention showing the clearest benefit in this control.
We have added the matched-schedule control to Figure 5a and clarified the model, domain, and fixed replay share in Section 4.3. This section now also explains that following RoD’s replay schedule with random selection in both streams recovers much of the overall trade-off, while full RoD achieves better adaptation and task retention at the same schedule. Appendix C.4 (Table 6) breaks down forgetting by source and capability.
Online data mixing. We evaluated Aioli [3] on Nemotron-12B Legal as an online mixing baseline. We used Legal and general replay as the two groups, with uniform sampling within each group. Aioli started at a 50/50 mixture and ran for 2,960 steps, close to the 2,958-step budget of the frontier runs. All probe steps counted toward that budget, and the probe set was disjoint from the reported validation sets.
| Method | Replay allocation | Adaptation loss ↓ | Validation forgetting ↓ | Task-based forgetting ↓ |
|---|---|---|---|---|
| Fixed replay (50%) | 50% | 0.992 | 0.024 | 0.036 |
| Aioli | 50% → 84% (67% mean) | 1.033 | 0.035 | 0.036 |
| RoD | 21% | 0.967 | 0.071 | 0.049 |
Aioli shifts toward a replay-heavy mixture. Despite allocating more replay on average, it has higher adaptation loss and higher forgetting than fixed 50% replay. RoD reaches a different operating point, with better adaptation and higher forgetting than either comparator at a substantially lower replay share. The matched-trajectory ablation above complements this comparison by showing the contribution of sample selection when the replay schedule is held fixed.
We have included the Aioli comparison in Appendix C.5 (Table 7) of the revised manuscript, together with the experimental setup and a discussion of the resulting adaptation–retention trade-offs.
Typographical corrections
Thank you for catching these errors. We have corrected all four in the revised manuscript.
Computational overhead and comparison budgets
Thank you for catching the inconsistency. We have corrected Table 3’s caption to specify that the runs are trained-token-matched, consistent with Section 4. Appendix D.1 (Table 8) now reports normalized B200 GPU-hours, including training the no-replay adaptation reference, precomputing reference losses, and scoring candidates online.
The appropriate replay ratio is generally unknown before training. To obtain the operating points shown on the fixed-replay frontier in Figure 2, we ran a sweep and selected favorable ratios retrospectively. Including the cost of that search, RoD uses 0.72–0.83× the compute of the evaluated fixed-replay campaign, a 17–28% reduction, even after accounting for its reference and scoring overhead. RoD thus finds a competitive adaptation–retention trade-off at lower cost than the evaluated replay-ratio search.
When a suitable replay ratio is already available, a single fixed-replay run remains cheaper: RoD costs 2.85–3.27× as much across the directly scored settings. The search-campaign savings therefore concern the cost of identifying a competitive allocation and do not establish performance superiority under equal total compute.
Section 4.4 shows how curriculum transfer reduces the overhead at larger scales. For German Qwen3.5-35B, constructing the curriculum at 4B and training the 35B model on it costs 0.65× the evaluated sweep, a 35% reduction, including curriculum construction. This corresponds to 2.60× a single fixed-replay run. Curriculum transfer moves candidate scoring off the target model while preserving a competitive adaptation–retention trade-off.
Comparison with easy-sample weighting
Thank you for suggesting the easy-sample weighting baseline. We evaluated FLOW [4] on German Qwen3.5-4B. We kept the data, learning-rate schedule, and trained-token budget fixed and weighted each adaptation block’s loss by , where is the block’s mean token loss under the base model and is the median of these block losses over the adaptation dataset. FLOW uses no replay data.
| Method | Adaptation loss ↓ | Validation forgetting ↓ |
|---|---|---|
| No replay | 1.849 | 0.521 |
| FLOW | 1.864 | 0.521 |
| Fixed replay (17%) | 1.863 | 0.036 |
| RoD | 1.859 | 0.028 |
FLOW and unweighted no-replay training follow nearly identical general-loss trajectories and both end at 0.521 validation forgetting. Sanyal et al. [4] found that FLOW improved general task retention when they used supervised fine-tuning to adapt base models to math reasoning on MetaMathQA. Our setting involves continued pretraining on roughly 12B tokens of German text. Here, a low base-model loss identifies blocks that are easier to predict within the German corpus, and training on those blocks can still shift the model away from the general distribution. This may explain why reweighting alone had little effect on retention in our setting. General-domain replay directly exposes the model to that distribution during training, reducing validation forgetting to 0.036 with 17% fixed replay and 0.028 with RoD.
We hope the additional ablations, baseline comparisons, and clarifications address your concerns. If any point remains unresolved or needs further explanation, please let us know.
References
[1] Ibrahim, A., et al. (2024). Simple and Scalable Strategies to Continually Pre-train Large Language Models. TMLR.
[2] Tong, X., et al. (2026). SELECting over Tokens: Curating Pre-training Data at Scale via Token Classification. ACL.
[3] Chen et al. (2025). Aioli: A Unified Optimization Framework for Language Model Data Mixing. ICLR.
[4] Sanyal et al. (2025). Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting. ICML.
Thank you again for your time and detailed feedback. Following your suggestion, we have now completed additional seed experiments and would like to share the results.
Consistency across seeds and capability retention
During the rebuttal period, we ran a second seed for both RoD and the nearest fixed-replay baseline (17%) on German Qwen3.5-4B:
| Method | Seed | Adaptation loss ↓ | Validation-loss forgetting ↓ | Task-based forgetting ↓ |
|---|---|---|---|---|
| RoD | 42 | 1.8592 | 0.0276 | 0.107 |
| RoD | 44 | 1.8583 | 0.0268 | 0.105 |
| Fixed replay (17%) | 42 | 1.8626 | 0.0360 | 0.090 |
| Fixed replay (17%) | 44 | 1.8689 | 0.0385 | 0.101 |
The results reproduce the same adaptation-retention trade-off in both seeds: RoD achieves lower adaptation loss and lower validation-loss forgetting, while fixed replay achieves lower aggregate task-based forgetting. The magnitude of the task-retention gap varies, however, narrowing from 1.7 to 0.4 percentage points in the additional seed.
RoD’s results are closely matched across the two seeds: adaptation loss is 1.8592 and 1.8583, validation-loss forgetting is 0.0276 and 0.0268, and task-based forgetting is 10.7 and 10.5 percentage points. The small differences between these runs provide additional evidence of stability across seeds in this setting. Together with the consistent adaptation–retention trade-offs observed across model families, scales, and domains, these results further support the robustness of our findings.
We hope these additional results and our earlier responses address your concerns. If any questions or concerns remain unresolved, please let us know. We would welcome the opportunity to discuss them during the remaining discussion period.
Thank you for the detailed feedback and for highlighting RoD’s simple selection rule and the strong native-replay results on German Nemotron-12B. We address your concerns and report the additional ablations and baseline comparisons below.
Single-seed comparisons and capability retention
We appreciate the close reading of these comparisons and agree that small margins warrant caution when each configuration is run with a single seed. We chose this design to evaluate RoD across model families, adaptation domains, and replay sources within our compute budget. The standalone RoD and fixed-replay sweeps across the five main settings in Figure 2 consumed approximately 20,000 GPU-hours in total. Extending this full comparison to three seeds would require approximately 40,000 additional GPU-hours, equivalent to USD 134,000–170,000 at an assumed B200 rental rate of USD 3.35–4.25 per GPU-hour. Given this high cost of repeating full training runs, single-seed comparisons are a recurring practice in large-scale CPT studies [1, 2].
Nevertheless, we agree that small differences from single-seed runs warrant caution. We therefore assess the performance of RoD relative to the fixed-replay frontier across the five different settings. Our main finding, that RoD reaches or improves upon this frontier (Sec. 4.1), holds across all settings. Matching the frontier is meaningful because RoD finds a competitive operating point without sweeping replay ratios. In the three nearest-share comparisons you highlight, RoD also achieves lower adaptation loss and lower validation-loss forgetting. Prior CPT replay comparisons report loss differences of comparable magnitude [2, Table 2].
We agree that our replay score and validation-loss forgetting metric are closely aligned. The held-out evaluation tests whether gains extend beyond the examples selected for replay to unseen sequences from the replay distribution. We assess capability retention separately through task evaluations. In both German Qwen settings, RoD achieves lower adaptation loss and lower validation-loss forgetting than 17% fixed replay, while aggregate task-based forgetting is slightly higher. The capability breakdown for Qwen3.5-9B shows that the gap is concentrated in reasoning-intensive tasks, with RoD retaining parametric knowledge about as well as fixed replay (Appendix C.2).
We have clarified in Section 4.1 that the frontier comparisons use one seed per configuration and assess consistency across five settings, with a pointer to Appendix B.3. Section 4 now distinguishes retention on held-out replay data from task-based capability retention, and Section 4.1 explicitly reports the German Qwen trade-off, with a capability breakdown in Appendix C.2.