- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose a shared likelihood objective for pretraining, midtraining, supervised fine tuning, and selected reinforcement learning objectives. The paper derives likelihood targets for reward weighted generations and implements a trainer that mixes generated responses with static data. The 124 million parameter model tests training across stages on controlled reasoning tasks. A 756.8 million parameter model tests data mixing during mathematical reinforcement learning. The evaluations cover reasoning, coding, knowledge retention, and subsequent adaptation.
Critical strengths
The derivation states the boundary of the common objective clearly. In particular, fitting generations from an earlier policy does not generally produce the gradient of reinforcement learning for the current policy. This distinction makes the proposed training formulation useful for analyzing asynchronous training.
The controlled study accounts for pretraining, midtraining, and generation compute. It also expands the comparison to several staged training budgets. The separate loss and policy regularization comparisons help identify when data timing and mixture composition matter.
Critical weaknesses
The evidence for one algorithm across the full training lifecycle is narrow. The only experiment that mixes data before reinforcement learning uses a 124 million parameter model and one synthetic puzzle family. The larger model completes conventional pretraining, midtraining, and supervised fine tuning before the proposed mixing begins. Appendix C.9 also shows that a tuned, separately normalized supervised loss performs better than the shared loss on Difficulty Levels 5 and 6 in some settings. These results do not establish that the shared loss drives the reported gains. The novelty discussion should also address two omitted, published studies [1] interleave reinforcement learning with targeted fine tuning, while [2] apply reinforcement learning to pretraining text. Neither study is identical to this paper, but both bear directly on its lifecycle claim.
The practical claims about compute and capability retention are not supported by a matched comparison. Appendix B.2 states that the larger model experiments do not match total compute. Appendix D.6 reports different generation procedures and reward baselines for unified and staged training. Available generations lag the trainer by 12 to 44 updates, while the age of consumed examples is not recorded. At a 2% mixture, Table 12 reports HumanEval+ accuracy of 10.98%, versus 15.51% for staged training, and MBPP+ accuracy of 19.00%, versus 25.74%. Mathematical pass@1 is 24.16% for both. The paper therefore demonstrates a retention tradeoff, but does not establish broad capability preservation or a practical compute advantage. [3] provide a relevant omitted analysis of how data from the current policy affects forgetting.
[1] Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions.
[2] Reinforcement Learning on Pre-Training Data.
[3] Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting.
Questions for authors
Can the authors compare unified and staged training for the larger model using the same generation procedure, reward baseline, number of fresh generations, and total model compute? This would determine whether the measured difference comes from data mixing.
What criterion supports the claim of preserved broader capabilities when both coding accuracy measures decline at every reported constant mixing proportion? Please distinguish accuracy retention from the reported gains in coverage and later adaptation.
3 comments
Q1 / W2: Matching generation, reward baseline, fresh generations, and compute. **All of these are already matched at 124M. We interpret the reviewer’s “the larger model” as the 0.76B model. **
As a summary, the new comparisons don't change the order between unified and staged training: at matched compute, unified keeps a SciQ lead of 11.5–16.0 points and staged keeps a coding pass@1 lead of 3.1–4.1 points at every mixing proportion. Mixing raises SciQ from 50.2% to 67% with more mid-training data, though not strictly monotonic increasing.
Matching compute for the 0.76B model leaves the comparison unchanged: unified keeps its SciQ lead and staged keeps its coding pass@1 lead at every budget. Under the paper's FLOP accounting, the mid-training data that unified training adds costs only 0.3–3.2% more than staged RL updates. The continued-trained results across budget-matched steps are presented in the table below.
| Mixing | Model | RL updates | Math | Coding | ARC-C | SciQ |
|---|---|---|---|---|---|---|
| – | Staged (submission) | 12,000 | 24.16 (68.56) | 20.63 (44.40) | 31.48 | 53.00 |
| 0.5% | Unified (submission) | 12,000 | 24.27 (68.32) | 16.51 (46.38) | 32.34 | 62.00 |
| Staged, compute-matched | 12,039 | 24.54 (68.61) | 19.62 (44.58) | 32.08 | 50.50 | |
| 1% | Unified (submission) | 12,000 | 23.86 (68.25) | 15.49 (44.89) | 33.53 | 66.10 |
| Staged, compute-matched | 12,075 | 24.42 (68.28) | 19.59 (44.40) | 32.59 | 50.10 | |
| 2% | Unified (submission) | 12,000 | 23.99 (69.61) | 15.39 (45.51) | 34.22 | 67.00 |
| Staged, compute-matched | 12,157 | 24.65 (68.52) | 19.50 (45.23) | 32.00 | 51.20 | |
| 5% | Unified (submission) | 12,000 | 23.85 (69.68) | 15.93 (45.17) | 30.12 | 65.10 |
| Staged, compute-matched | 12,400 | 25.24 (69.36) | 19.67 (44.31) | 31.40 | 53.20 | |
| Scores are pass@1 with pass@32 in parentheses. |
0.76B training without mixing: the 0% mixing run shares the asynchronous trainer, generation, and reward weighting with the mixed runs. Relative to it, mixing lowers held-out corpus loss and raises SciQ performance from 50.2% to 62%–67%, which isolates the effect of the data within the same implementation.
124M, everything matched: in the existing experiment, staged and unified share the trainer, generator, reward weighting, and number of fresh generations, and both use only data generated at most one update earlier; only the data schedule differs.
W2: Policy lag. In the 0.76B experiments, unified training uses responses from earlier policies, whereas staged RL collects responses from the current policy before optimization. In the 124M study, policy lag is the same for staged and unified training: training examples must be generated by a policy at most one update old. Our theory suggests that the unified training should allow this reasonable level of staleness (12-44 steps as discussed in Appendix D.6 for 0.76B experiments). Despite this lag, unified training with 0% mixing matches staged RL on Math (pass@1 24.75 vs. 24.16; pass@32 68.88 vs. 68.56). In the 124M runs, the trainer uses only data generated at most one update earlier, for both methods, so staleness does not differ between the experiments compared. A controlled lag sweep is future work.
Q2 / W2: The criterion for "preserved broader capabilities". You are right that coding accuracy declines. We will replace "preserves broader capabilities" with the specific outcomes, which form a tradeoff:
-
Accuracy: at 0.5–5% mixing, held-out corpus loss decreases as mixing increases. MMLU (37.1 to 40.8) and SciQ (50.2 to 67) improve. Coding pass@1 ends at or below the pre-RL model. Staged training shows the reverse pattern. We will report this as a measured tradeoff of mixing mid-training text into math RL.
-
Coverage: coding pass@32 is higher than the 0% run at every nonzero proportion, and at the 0.5% budget it is higher than compute-matched staged training (46.4 vs. 44.6).
We believe coverage is a better indicator of capability preservation. One reason the model’s pass@1 capability decreases may be the decreased instruction following capability introduced by the diverse midtraining corpus. Since we see that after the same coding RL, unified checkpoints reach higher coding accuracy and coverage than staged ones, except for MBPP+ pass@1 at 0.5% (Appendix D.7), we see the claim of broader capability preservation as valid. We will revise our wording to distinguish accuracy preservation and coverage preservation in the writing.
[1] Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions
[2] Reinforcement Learning on Pre-Training Data
[3] Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting
[4] The Art of Scaling Reinforcement Learning Compute for LLMs
Thank you for the careful reading and for noting the explicit treatment of fixed generators and the compute accounting in the controlled study. We answer each weakness and question in turn below.
W1: one synthetic task: We agree that multi-task learning is an important future direction, and we acknowledge it in our existing discussion section already. However, existing RL-only scaling laws such as [4] only focus on single-task scaling laws (note that their multi-task results still only focus on single-task metrics). We believe that understanding multi-task training within RL training is worth its own research program, so it is beyond the scope of this work and we leave it as a future direction.
W1: 0.76B starts with instruction-tuned model: The 0.76B study starts from an instruction-tuned model by design: it measures what mixing real midtraining data into RL does at a scale where we cannot afford a full sweep of pretraining configurations. Our research questions are about the whole pipeline, from pretraining to RL, and answering them takes many complete runs under controlled compute. That is why the full-pipeline study is at 124M and the real-data study at 0.76B.
W1: Shared vs. separately normalized loss: The goal of the majority of our experiments is to provide scientific understanding instead of claiming that unified training is always better. It is not surprising that the separately normalized loss can sometimes outperform the unified (shared) loss under certain configurations. However, even if the end goal is to pick the best objective in practice, note that the best-performing separately normalized loss requires different hyperparameters in different settings. We still believe the results demonstrate that unified loss should be preferred in practice.
W1 / W2: Omitted related work. Prior work adds supervised data to an RL loss, or turns pretraining text into an RL task. We do the reverse: we turn RL into likelihood fitting, so every stage shares one loss. The revision adds all three studies:
ReLIFT [1] alternates RL with fine-tuning on solutions to the hardest questions; in our framework, this is a schedule that switches between two sources.
RLPT [2] turns pretraining text into an RL task, the reverse direction from ours.
Retaining by Doing [3] attributes RL's lower forgetting to on-policy data; our retention results vary on a different lever, the share of static data mixed into RL.