acceptodds
Under review as a conference paper at ICLR 2027

RL-Guided Mid-Training for Iterative LLM Evolution

Abstract

Standard training pipelines for large language models (LLMs) are typically unidirectional, progressing from pre-training to post-training. However, the potential for a bidirectional process—where insights from post-training retroactively improve the pre-trained foundation—remains unexplored. We aim to establish a self-reinforcing flywheel: a cycle in which reinforcement learning (RL)-tuned model strengthens the base model, which in turn enhances subsequent post-training performance, requiring no specially trained teacher model. To realize this, we analyze training dynamics and identify the mid-training phase as a critical turning point for model capabilities. This phase typically occurs at the end of pre-training, utilizing high-quality corpora under a rapidly decaying learning rate. % While utilizing RL models to guide this phase via Knowledge Distillation (KD) seems intuitive, we demonstrate that KD suffers from critical limitations: it induces overfitting to the teacher model, thereby impairing generalization and yielding temporary gains that diminish during subsequent post-training. Building upon this insight, we introduce ReMiT (Reinforcement Learning-Guided Mid-Training) as a concrete feedback mechanism, which leverages the reasoning priors of RL models to dynamically reweight tokens during the mid-training phase. Empirically, ReMiT achieves an average improvement of 3% on 10 pre-training benchmarks, spanning math, code, and general reasoning, and sustains these gains by over 2% throughout the post-training pipeline. These results validate an iterative feedback loop, enabling self-reinforcing evolution of LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models (LLM), Token Reweighting, Iterative Evolution

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_nXxHOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new way to dynamically re-weight tokens during mid-training phase of models using a Reference RL frozen model leading to improvements in base model capabilities and next phase of post-training with iterative feedback loop.

Critical strengths
  • The method is easy to follow and authors showed using 1B/2B/3B models that the bounded token dynamic re-weighting helps improve performance over uniform NTP or other baseline methods.
Critical weaknesses
  • From the Table 1 we can see that gains very across 3 scales of 1B/2B/3B, in 1B model we can see a big gap between the proposed ReMIT approach compared to uniform NTP, but gap shrinks with 2B and 3B, also KD also achieves similar mid-train average performance on OLMo-1B and Youtu-LLM-2B compared to ReMIT. So overall it's not evident to me what specifically the method contributes given the modest performance gains?

  • Authors wrote that they used for SmolLM3 model at sequence length 8192 for mid-training but generally mid-training is done at larger context length, even SmolLM3 did it at 128k , authors also missed a prior work about Mid-training at scale [1] where the paper shows that mid-training performance gains need to be evaluated across different domains and one need to make sure that general short context and long context performance is preserved.

[1]Runwal, Bharat et al. “PRISM: Demystifying Retention and Interaction in Mid-Training.” ArXiv abs/2603.17074 (2026): n. pag.

Questions for authors
  • From the Section 4.4, authors showed that RL tuned reference model is better than SFT one, so it raises this question that does this token re-weight mechanism contributes heavily compared to using a good RL reference model? A OPD style baseline would answer this question if student-teacher distillation works as good as using the re-weight token method with RL reference model.
  • Does the approach preserves general capabilities of the base model? For example the long context performance I flagged in past section, as most mid-training is done at larger context length, current experiments doesn't answer this.
  • I also didn't find any information about the exact data mixture the authors trained their models, can authors point me to them?
4 comments

We thank the reviewer for the detailed comments.


W1. Modest gains and comparison with KD

  • Scale. Gain size depends on several factors besides scale, since the three families differ in pre-training data, base checkpoint quality and mid-training corpus. In particular, OLMo-1B has much more headroom: its pre-mid-training average is 16.44 vs. 31.18 (SmolLM3-3B) and 30.34 (Youtu-LLM-2B). Gains do not shrink from 2B to 3B (+1.78 / +1.84). At these scales, the other reference-guided baselines fall below NTP (MiniPLM −0.48 / −0.11; RHO-1 −1.33 / −4.56), while ReMiT stays above it.
  • KD. Our claim concerns the model after post-training, which is the purpose of mid-training. KD matches or exceeds ReMiT at mid-training but falls behind after RL; with 3 post-training seeds, ReMiT leads KD by +2.20 ± 0.25 (OLMo-1B) and +1.54 ± 0.35 (Youtu-LLM-2B), winning on every seed. On Youtu-LLM-2B, ReMiT's gain over NTP grows from +1.78 (mid-training) to +2.68 (after RL).
  • Cost. KD needs the teacher's full (or top-k) distribution at every position; ReMiT needs only one scalar per token (the teacher log-prob of the observed token).

Q1. Reference vs. mechanism

We isolate the two factors by varying one at a time (avg. %, OLMo-1B / Youtu-LLM-2B):

MethodReferenceMechanismMid-trainAfter RL (3 seeds)
ReMiTSFTreweighting23.72 / 35.37–
RHO-1RLtoken selection23.56 / 32.66–
KDRLdistribution matching29.78 / 38.6730.96 / 50.64
ReMiTRLreweighting27.56 / 38.5833.16 / 52.18

With the mechanism fixed, the RL reference beats the SFT reference; with the reference fixed, reweighting beats selection and distribution matching after post-training. Both matter: the reference supplies the signal, and the mechanism decides whether it survives post-training.

We did not include OPD because it is not a like-for-like mid-training baseline. (i) OPD trains on student-generated sequences, so it replaces the mid-training corpus rather than changing the objective on it; a base checkpoint before mid-training also lacks the prompts/task distribution OPD needs. (ii) Generating and scoring 50B tokens autoregressively costs orders of magnitude more than one teacher forward pass on existing text. (iii) The corpus-level counterpart of distillation is KD with the same teacher, which we compare above. OPD is a post-training method and is complementary: it can be applied after ReMiT.

W2 / Q2. Sequence length and general capabilities

Sequence length. We modify the decay/annealing phase on high-quality data and follow each family's setting for it: OLMo-2-1B mid-trains on 50B Dolmino tokens at 4k; SmolLM3's decay phase (10T→11.1T tokens) runs at 4k; Youtu-LLM pre-trains at 8k. SmolLM3 extends context in a separate later stage (4k→32k→64k); 128k is reached by YaRN extrapolation at inference, not by training [2]. ReMiT only rescales per-token losses and does not change data, length or positional encoding, so it is orthogonal to long-context extension and can be applied unchanged in that stage; we excluded it to isolate the objective.

General capabilities. Our suite already covers knowledge and general ability beyond math/code: MMLU-Pro, TruthfulQA, ARC-C, BBH and IFEval. ReMiT is above NTP on all 15 of these results (5 tasks × 3 families, Table 1), and above NTP on 9/10 post-training tasks in each family. This is expected by design: unlike data selection, ReMiT keeps every corpus token with weight in [0.8, 1.2], so no domain is dropped. We will cite PRISM and discuss retention in the revision.

Q3. Data mixture

Mid-training data for all three families follows official documentation; within each family, all methods use the identical 50B-token corpus.

  • OLMo-2-1B: the public Dolmino-mix-1124 50B mix [1], i.e., the official OLMo 2 stage-2 (mid-training) data: DCLM web 47.2%, FLAN 16.6%, math 20.8% (TinyGSM-MIND, MathCoder2, TuluMath, MetaMath, GSM8K, etc.), Wikipedia 7.1%, peS2o STEM papers 5.9%, StackExchange 2.5%.
  • SmolLM3-3B / Youtu-LLM-2B: a mixture following the official SmolLM3 decay-stage recipe [2]: web (FineWeb-Edu, DCLM, FineWeb2, FineWeb2-HQ), code (The Stack v2 / Stack-Edu, StackExchange), and math (FineMath4+, InfiWebMath4+, MegaMath, OpenMathReasoning).

We will add exact sources and proportions to Appendix D.1.


We hope these clarifications address your concerns, and we would be truly grateful if you would kindly reconsider your assessment of our manuscript. Thank you again for your time and constructive feedback.

References.

[1] AllenAI, Dolmino-mix-1124. huggingface.co/datasets/allenai/dolmino-mix-1124

[2] Hugging Face, SmolLM3: smol, multilingual, long-context reasoner. huggingface.co/blog/smollm3

Thank you for the rebuttal. For the first question the provided table helps but overall, the results suggest that the choice of reference matters, but the incremental benefit of ReMiT over standard KD remains less convincing to me. For the second question I was hinting more towards capabilities like Long context, as I said mid-train is generally done at longer context phase, even in smol-lm 3 they include 4K -> 32K-> 64K context extension as part of mid-training, so claiming ReMiT is orthogonal to context extension does not establish that it preserves long-context capabilities. I will retain my current score based on the current results.

We thank the reviewer for the follow-up and for engaging with our response.

ReMiT vs. KD. With the reference fixed to the same RL model, the mechanism alone changes the final outcome. Measured against uniform NTP after RL (3 seeds), KD gains +1.43 (OLMo-1B) and +1.28 (Youtu-LLM-2B), while ReMiT gains +3.63 and +2.82, i.e., more than twice KD's improvement from the same teacher, on every seed.

This comparison is at matched compute. Under this matched budget, ReMiT delivers more than twice KD's post-training gain. Beyond this, the two signals differ sharply in what they require:

  • ReMiT consumes a single scalar per token: the teacher's log-probability of the observed token. Since the teacher is frozen and the corpus is fixed, this signal can be computed exactly in a separate offline inference pass and stored as one number per token, so that mid-training itself runs without the teacher in the loop, at the cost of standard NTP.
  • KD consumes the teacher's full distribution. Online, it must retain the teacher's full-vocabulary logits to compute the KL term; offline, it has no exact form: even a truncated top-5 needs 10 numbers per token and is only an approximation, while the full distribution is infeasible to store at 50B tokens.

Moreover, even with the teacher computed online, ReMiT reaches the final accuracy of NTP 3.5× faster in GPU hours (Fig. 16).

Long context. Our comparison is controlled: NTP and ReMiT are trained on exactly the same data, with the same sequence length, positional encoding, token budget, and optimization schedule. ReMiT only reweights the per-token loss within each sequence; it changes neither the inputs the model sees nor how positions are encoded. Any context-extension stage applied afterward would be applied identically to both models, so the comparison remains fair on this axis, and our conclusions are drawn under this matched setting. Time permitting, we plan to extend this matched comparison to longer-context training data, training NTP and ReMiT under identical long-sequence settings.

As the discussion period is closing, we would be grateful if you could let us know whether our latest reply addresses your concerns. Thank you for your time.

Reviewer_rf8oOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose ReMiT, a mid-training technique which uses an RL-tuned reference model to reweight token losses while retaining the original next-token target. They evaluate across three model families, showing improvements across benchmarks compared to uniform training and knowledge distillation. They also test using a ReMiT-tuned model as the RL-tuned reference, demonstrating even higher gains in that setting.

Critical strengths

The authors thoroughly evaluate the downstream consequences of mid-training choices rather than just looking at the base-model performance. ReMiT improves final average scores by 2-3 pp. in the evaluated settings.

The comparison with KD is informative: KD achieves higher mid-training averages, but ReMiT performs better after post-training. On OLMo, this reversal also occurs within the same post-training evaluation suite.

The method is simple and clearly specified. Experiments include several relevant baselines, SFT versus RL and a weaker intermediate RL reference.

Critical weaknesses

The RHO-1 baseline changes both the reference model and the weighting mechanism. Using the same RL reference for token selection would help establish what ReMiT’s bounded weighting adds beyond a better reference.

The explanation involving “pivotal reasoning tokens” is supported mainly by qualitative examples. The local analysis is mathematically reasonable, but does not establish that the proposed weights align with downstream utility. A control normalizing the mean weight would also help distinguish token prioritization from changes in update scale.

The iterative-improvement claim rests on one additional cycle on OLMo, with only five benchmarks shown at each stage. This is encouraging evidence for reference reuse, but insufficient to establish sustained improvement across cycles.

No repeated-run results or uncertainty estimates are reported.

Questions for authors

How is KD implemented, including the divergence direction, temperature, and any mixing with next-token loss?

Can you provide the stage-specific post-training hyperparameters and budgets?

3 comments

We thank the reviewer for the careful reading and for highlighting the cross-stage ranking reversal. All new runs follow the main-paper protocol.


W1 & W2. Reference vs. mechanism, and what drives the gain

We reran RHO-1 (retention 0.8) with the same RL reference as ReMiT.

Table R1. Mid-training average (%).

MechanismReferenceOLMo-1BYoutu-LLM-2B
Uniform NTP–22.3536.80
RHO-1 (w∈{0,1}w\in\lbrace 0,1\rbrace)own23.1032.24
RHO-1 (w∈{0,1}w\in\lbrace 0,1\rbrace)RL23.5632.66
ReMiT (w∈[0.8,1.2]w\in[0.8,1.2])SFT23.7235.37
ReMiT (w∈[0.8,1.2]w\in[0.8,1.2])intermediate RL25.11–
ReMiT (w∈[0.8,1.2]w\in[0.8,1.2])RL27.5638.58

Same reference, different mechanism (W1). With the RL reference, ReMiT beats RHO-1 by 4.00 / 5.92. RHO-1 drops 20% of targets, whereas ReMiT keeps every target and only rescales its loss, which is more robust to an imperfect reference.

Same mechanism, different reference (W2). Here only the token assignment changes. For RHO-1 the weight distribution is identical (exactly 20% zeros), yet per-task scores shift by up to 3 points. For ReMiT, scores rise monotonically as the reference advances through post-training (SFT → intermediate RL → RL: 23.72 → 25.11 → 27.56). The gain thus depends on which tokens are up-weighted (the covariance term in Eq. 14), and stronger post-trained references yield more useful assignments.

Mean weight (W2). Centering makes δ^\hat\delta zero-mean, but the sigmoid and clipping are nonlinear and δ^\hat\delta is right-skewed, so E[w]≠1\mathbb{E}[w]\neq 1. Measured E[w]\mathbb{E}[w] stays within 0.96–0.98 on OLMo-1B: ReMiT takes slightly smaller average steps than NTP, so update scale cannot explain the +5.21 gain. We will add these results and a mean-normalized control to complement the qualitative pivotal-token examples.

W3. Iterative improvement

Table R2. ReMiT² on OLMo-1B, all 10 benchmarks (%).

Mid-trainGSM8KMATHGPQABBHIFEHEMBPPTQAARC-CMMLU-PAvg.
ReMiT61.6414.5024.5532.0728.5412.809.2025.5849.2317.4427.56
ReMiT²62.7015.7024.3332.7130.7013.4110.8026.0749.6618.2028.43
After RLGSM8KMATHMBPPMBPP+HEHE+ARC-CGPQA-DMMLU-PIFEAvg.
ReMiT65.7319.6611.6022.0928.3524.3944.8025.2516.7370.6232.92
ReMiT²66.1121.1612.8025.5629.2726.2245.3926.3417.4570.5034.08

The second cycle improves both stages (+0.87 mid-training, +1.16 after RL, on 9/10 tasks each), confirming that the RL model from one cycle is a better reference for the next. A smaller increment is expected: the corpus, post-training data and RL environment are fixed, and only the reference changes. RLVR mainly sharpens capabilities already in the starting model [1], and the guidance signal weakens as the student approaches the reference. Larger gains in further cycles come from new data or RL environments, which ReMiT feeds back into the base model.

W4. Repeated runs

Table R3. Final 10-task average after RL (%), 3 post-training seeds, mean ± SD; Δ = paired ReMiT − KD.

ModelNTPKDReMiTΔ
OLMo-1B29.53±0.1730.96±0.2033.16±0.33+2.20±0.25
Youtu-LLM-2B49.36±0.3150.64±0.2052.18±0.26+1.54±0.35

ReMiT beats KD on every seed; its worst seed exceeds KD's best in both families.

Q1. KD implementation

KD uses the same RL reference, corpus and budget as ReMiT, with full-vocabulary teacher logits and forward KL at τ = 1, mixed equally with NTP: L_KD = 0.5 · L_NTP + 0.5 · KL(p_T ‖ p_θ)

Q2. Post-training hyperparameters

We follow the official Open-Instruct recipe [2]. We will add these to Appendix D.3.

  • SFT (tulu-3-sft-olmo-2-mixture-0225): lr 3e-5, linear decay, 3% warmup, 2 epochs, batch 128, length 4096.
  • DPO (olmo-2-0425-1b-preference-mix): length-normalized, β = 5, lr 2.5e-6, linear decay, 10% warmup, 1 epoch, batch 128, length 2048.
  • RLVR/GRPO (RLVR-GSM-MATH-IF-Mixed-Constraints): lr 5e-7 constant, KL coef. 0.01, 48 prompts × 16 rollouts per step, response length 2048, temperature 1.0.

We hope these clarifications address your concerns. Thank you again for your time and effort; we would be happy to address any further questions.

References.

[1] Yue et al., Does RL Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?, NeurIPS 2025.

[2] AllenAI, Open-Instruct OLMo 2 commands. github.com/allenai/open-instruct/blob/main/docs/olmo2.md

Thank you for the additional experiments, which address my main empirical concerns. On the mean weight, a smaller mean weight doesn't on its own rule out an update-scale effect, so the mentioned mean-normalized control would be helpful. Overall though, the rebuttal strengthened my confidence in the results, and I maintain my positive evaluation.

We thank the reviewer for the positive evaluation and are glad the additional experiments addressed the main empirical concerns.

Reviewer_3nSCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReMiT (Reinforcement Learning-Guided Mid-Training), a framework that leverages feedback from RL-tuned models to improve the mid-training stage of LLMs. Specifically, ReMiT computes token-level prediction discrepancies between a frozen RL reference and the student model, using sequence-level centering and bounded nonlinear weighting to adjust the standard next-token prediction objective.

The paper also provides a local gradient analysis to characterize the potential benefits of reference-guided token reweighting. Experiments on three model families demonstrate improvements over standard mid-training and existing data-selection baselines. Importantly, these improvements persist after subsequent SFT, DPO, and RL stages. An additional feedback round on OLMo-1B further explores the potential for iterative model improvement.

Critical strengths
  1. Practical and well-motivated training framework.

The paper explores an interesting direction of incorporating post-training feedback into pre-training. ReMiT provides a simple and practical mechanism for transferring RL-derived guidance through bounded token reweighting, without requiring a separately trained teacher model. This makes the approach potentially useful for improving existing LLM training pipelines.

  1. Comprehensive evaluation across training stages.

The experiments cover three model families and evaluate performance beyond mid-training, including subsequent SFT, DPO, and RL. ReMiT improves final post-training accuracy over both standard NTP and knowledge distillation on the evaluated models. The comparison across training stages is particularly valuable, showing that better intermediate performance does not necessarily translate into better final performance.

Critical weaknesses
  1. Insufficient controlled comparisons and statistical validation.

ReMiT and RHO-1 use different reference models, making it difficult to isolate the benefit of the proposed weighting mechanism. Furthermore, the experiments do not report variance across independent training runs, leaving the reliability of the relatively modest improvements over KD uncertain.

  1. Limited evidence for the iterative training mechanism.

The claimed self-reinforcing training loop is demonstrated through only one additional feedback round on OLMo-1B, using the same initial checkpoint. Moreover, the paper does not directly compare interventions at different training stages, leaving the claimed importance of mid-training insufficiently validated.

Questions for authors

Q1. Could the authors compare ReMiT against RHO-1 or other token-selection approaches using the same RL reference model? This would help isolate the contribution of the proposed weighting mechanism.

Q2. Could the authors provide results across multiple training seeds or additional uncertainty estimates, particularly for the final post-training comparisons with KD? This would help establish the reliability of the reported improvements.

Q3. Could the authors provide the complete evaluation results for the second feedback round and clarify whether the improvements persist over further iterations? Since ReMiT² restarts from the same pre-mid-training checkpoint, it would be useful to understand how this setting supports the proposed self-reinforcing evolution.

2 comments

We thank the reviewer for the constructive feedback. We (1) rerun RHO-1 with the same RL reference, (2) add multi-seed post-training results, and (3) report the full second-round results and discuss further iterations. All new runs follow the main-paper protocol (same θ₀, corpus, 50B-token budget, post-training recipe and evaluation).


Q1. RHO-1 with the same RL reference

We reran RHO-1 (retention 0.8) with the same RL reference as ReMiT, so only the use of the reference differs.

Table R1. Mid-training scores (%).

Method (ref.)GSM8KMATHGPQABBHIFEHEMBPPTQAARC-CMMLU-PAvg.
OLMo-1B
RHO-1 (own)50.4210.3225.4529.3319.066.716.2023.3846.4213.6823.10
RHO-1 (RL)52.2410.8424.3331.5216.556.717.8024.4848.2912.8123.56
ReMiT (RL)61.6414.5024.5532.0728.5412.809.2025.5849.2317.4427.56
Youtu-LLM-2B
RHO-1 (own)42.4616.4426.7940.4532.0127.4439.2026.6849.2321.6932.24
RHO-1 (RL)40.7118.9627.9041.5028.9029.8837.4028.4050.6822.2732.66
ReMiT (RL)52.6924.5029.6947.2136.9339.9447.0027.4254.6125.8538.58

The RL reference shifts RHO-1 scores in both directions and raises its average only slightly (+0.46 / +0.42); ReMiT still leads by 4.00 / 5.92. The gap comes from the mechanism: RHO-1 hard-drops 20% of tokens, while ReMiT keeps all targets and only rescales losses within [1−ε, 1+ε], which is more robust to an imperfect reference.

Q2. Multiple post-training seeds

Each mid-trained checkpoint goes through the full post-training pipeline with 3 seeds.

Table R2. 10-task average (%), mean ± SD; Δ = paired ReMiT − KD.

Model / StageNTPKDReMiTΔ
OLMo / SFT27.74±0.3430.52±0.1829.94±0.37−0.58±0.20
OLMo / DPO29.26±0.2529.40±0.1730.56±0.27+1.16±0.34
OLMo / RL29.53±0.1730.96±0.2033.16±0.33+2.20±0.25
Youtu / SFT45.59±0.2147.16±0.2047.97±0.18+0.81±0.30
Youtu / RL49.36±0.3150.64±0.2052.18±0.26+1.54±0.35

After RL, ReMiT beats KD on every seed; its worst seed exceeds KD's best in both families.

Why KD's mid-training lead does not carry over. Prior work shows a stronger intermediate checkpoint need not give a stronger final model: over-trained models are harder to fine-tune [1]; distillation raises pre-RL accuracy without necessarily raising post-RL accuracy and lowers entropy [2,3], which limits RL headroom [4]. ReMiT keeps corpus targets and does not pull the student toward the reference.

Q3. Full second-round results

Table R3. ReMiT² on OLMo-1B (%).

Mid-trainGSM8KMATHGPQABBHIFEHEMBPPTQAARC-CMMLU-PAvg.
ReMiT61.6414.5024.5532.0728.5412.809.2025.5849.2317.4427.56
ReMiT²62.7015.7024.3332.7130.7013.4110.8026.0749.6618.2028.43
After RLGSM8KMATHMBPPMBPP+HEHE+ARC-CGPQA-DMMLU-PIFEAvg.
ReMiT65.7319.6611.6022.0928.3524.3944.8025.2516.7370.6232.92
ReMiT²66.1121.1612.8025.5629.2726.2245.3926.3417.4570.5034.08

Why further cycles should not compound. The corpus, post-training data and RL environment are fixed; only the reference changes. ReMiT yields a better post-training starting point, but with unchanged data and environment, the benefit of a better start is bounded: RLVR mainly sharpens capabilities already in the starting model [5]. The guidance signal also weakens as the student nears the reference. Further gains need new data or RL environments; ReMiT is the channel that feeds them back into the base model.


We hope these clarifications address your concerns, and we would be truly grateful if you would kindly reconsider your assessment of our manuscript. Thank you again for your time and constructive feedback.

References.

[1] Springer et al., Overtrained Language Models Are Harder to Fine-Tune, ICML 2025.

[2] O'Neill et al., When Does Distillation Help Reinforcement Learning?, Baseten Labs, 2026.

[3] Dong et al., RL Starts before RL, arXiv:2609.28145.

[4] Cui et al., The Entropy Mechanism of RL for Reasoning LMs, 2025.

[5] Yue et al., Does RL Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?, NeurIPS 2025.

Thank you again for your time and thoughtful review. As the discussion period is coming to a close, we would like to kindly check whether our response has addressed your concerns, in particular the RHO-1 comparison under the same RL reference, the 3-seed results with confidence intervals, and the second-round ReMiT² experiment. If any questions remain, we would be glad to clarify them before the deadline. We hope our response might lead you to reconsider your assessment of our manuscript.

Reviewer_Hb7pOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ReMiT, a token-weighted next-token objective for mid-training. For each corpus token it computes the log-probability gap δt=log⁡pT(xt∣x<t)−log⁡pθ(xt∣x<t)\delta_t = \log p_T(x_t \mid x_{<t}) - \log p_\theta(x_t \mid x_{<t}) between a frozen RL-tuned reference TT (the model family's publicly released RL checkpoint) and the current student, centers the gap within the sequence, and uses wt=clip(2σ(δ^t),0.8,1.2)w_t = \mathrm{clip}(2\sigma(\hat\delta_t), 0.8, 1.2) as a detached loss weight. Applied to 50B-token mid-training of OLMo-1B, SmolLM3-3B and Youtu-LLM-2B, ReMiT raises the 10-benchmark base-model average over uniform NTP by 5.2, 1.8 and 1.8 pp, and it outperforms MiniPLM-style and RHO-1-style selection. After the same downstream post-training (OLMo-1B: SFT, DPO and RLVR with Open-Instruct; Youtu-LLM-2B: SFT and RL), the final average exceeds uniform NTP by 3.35 and 2.68 pp and exceeds knowledge distillation (KD) from the same RL reference by 1.92 and 1.15 pp, although KD has the higher mid-training average. Further results: an SFT reference helps less than the RL reference; a second round that uses the round-1 RL model as reference (ReMiT²) improves further on OLMo-1B; and a first-order expansion splits the effect of weighting into a weight–utility covariance term and a mean-weight term. Each comparison is a single run.

Critical strengths

1. A simple, inexpensive change with consistent gains through a full post-training pipeline. The weighting adds one detached factor to the NTP loss and needs only a forward pass of an existing checkpoint (43% lower throughput, Fig. 16). It improves the mid-training average in every model family, by 5.2 pp on OLMo-1B (GSM8K 48.1 → 61.6). The paper evaluates the intervention after SFT, DPO and RLVR with fixed downstream recipes, which is the evaluation a mid-training method needs and which most data-selection papers omit.

2. The KD comparison exposes a ranking reversal between stages. KD from the same reference has the higher mid-training average on both families (29.78 vs 27.56 on OLMo-1B) but the lower final average after RL (31.00 vs 32.92). If it holds across seeds, this is a practically useful observation: base-model benchmarks measured right after mid-training are a poor proxy for the value of a mid-training intervention, and matching the reference's distribution transfers less than reweighting the corpus toward it.

Critical weaknesses

The reference model was already post-trained on the downstream data, so the gains cannot yet be attributed to RL feedback improving the foundation. For OLMo-1B, Appendix D.3 post-trains with tulu-3-sft-olmo-2-mixture-0225, olmo-2-0425-1b-preference-mix and RLVR-GSM-MATH-IF-Mixed-Constraints. According to the public Hugging Face model cards, the released OLMo-2-0425-1B-RLVR1 checkpoint was produced from the OLMo-2-0425-1B base by SFT, DPO and RLVR on exactly these datasets, and the Instruct checkpoint adds a further RLVR-MATH stage on top of it. ReMiT therefore up-weights the corpus tokens that a model trained on the downstream data predicts better, and the student is then trained on that same data. The largest mid-training gains are on GSM8K (+13.5 pp) and IFEval (+12.4 pp), the targets of that post-training data, and the KD-trained base model scores 45.6 on IFEval. The paper lacks the comparisons that would separate RL-derived guidance from earlier exposure to the post-training distribution: (a) uniform NTP with the post-training data, or the reference's generations, mixed into the mid-training corpus at matched tokens; (b) uniform NTP followed by a longer post-training budget; (c) ReMiT with a reference that was not post-trained but has comparable loss on the corpus. Mixing post-training data into mid-training is the standard way to expose a base model to it, so (a) is the baseline ReMiT has to beat. The same fact makes the motivating analysis in Sec. 2.2 circular. The RL reference was fine-tuned from the post-mid-training checkpoint, so that checkpoint being closer to the reference than the pre-mid-training one is expected by construction; it does not show that mid-training is a turning point toward RL capabilities. The flywheel claim (Sec. 4.5) rests on one extra round on one model, reported on 5 of the 10 benchmarks, with a different subset at each stage (Fig. 8).

Questions for authors
  1. Which checkpoint served as the reference for each family (for OLMo-1B, RLVR1 or Instruct), and which post-training datasets produced it? For Youtu-LLM-2B and SmolLM3-3B, do those datasets overlap with the downstream data DpD_p?
  2. Can you report at least 3 seeds for the OLMo-1B pipeline (uniform NTP, KD and ReMiT through RL), with confidence intervals on the final average and on the ReMiT − KD difference?
  3. How does ReMiT compare with uniform NTP when the downstream SFT and RLVR data, or the reference's own generations on corpus prompts, are mixed into the mid-training corpus at a matched token budget?
  4. What is E[w]\mathbb{E}[w] over training, and how does uniform NTP compare when its learning rate is scaled by that factor?
  5. Please report ReMiT² on all 10 benchmarks at both stages.
3 comments

We thank the reviewer for the thorough review.


W1 / Q1 / Q3. Is the gain just earlier exposure to post-training data?

References. All references are the final released Instruct models (OLMo-2-0425-1B-Instruct, SmolLM3-3B, Youtu-LLM-2B), and downstream post-training follows the Open-Instruct recipe [1].

A post-trained model naturally encodes its post-training data and environment; feeding this back is the goal. The question is whether ReMiT does more than early exposure. Several results say yes:

  • Mid-training already bridges to post-training. As the end of pre-training, mid-training is designed to connect pre-training and post-training, so its corpus naturally includes high-quality, SFT-style data. OLMo's official mix (Dolmino-mix-1124 [2]) already contains such data, including instruction data (FLAN, 16.6%) and math QA (20.8%), some from the same sources as Tulu (e.g., TuluMath). Uniform NTP is therefore already exposed to post-training-style data through the standard data-mixing channel of baseline (a), and ReMiT improves on it.
  • RL adds more than data. Beyond the Tulu SFT data seen by both, the RL reference adds preference and RLVR training, which learn from comparisons, rollouts and rewards rather than new imitation targets. With the same weighting, gains rise from SFT → intermediate RL → final RL reference (23.72 → 25.11 → 27.56).
  • KD is the stronger exposure channel, yet loses. KD matches the reference's full distribution at every position; ReMiT passes one bounded scalar in [0.8, 1.2] per corpus token, never trains on reference outputs, and uses exactly the same corpus as NTP. KD leads at mid-training but trails ReMiT by 2.20 after RL on OLMo-1B. Pure exposure would favor KD.
  • Exposure gains fade, ReMiT's do not. Once the student sees the same data in post-training, an exposure advantage should shrink, as KD's does. ReMiT's persists and on Youtu-LLM-2B grows (+1.78 → +2.68).

On (b), all budgets follow the official recipe. ReMiT's SFT loss stays below the baseline across both epochs (Fig. 13), and in RL both runs plateau over the final steps with ReMiT still ahead (Fig. 14), so a longer budget is unlikely to close the gap.

Sec. 2.2. Fig. 2 is meant as motivation, not proof. The case for mid-training rests on the end-to-end comparisons in Sec. 4 and on prior evidence that mid-training choices shape later RL (OctoThinker).

Q2. Seeds and confidence intervals

Table R1. Final 10-task average after RL (%), 3 post-training seeds, mean ± 95% CI; Δ = paired ReMiT − KD.

ModelNTPKDReMiTΔ
OLMo-1B29.53±0.4230.96±0.5033.16±0.82+2.20±0.62
Youtu-LLM-2B49.36±0.7750.64±0.5052.18±0.65+1.54±0.87

Both Δ intervals exclude zero, and ReMiT beats KD on every seed.

Q4. Mean weight

E[w] stays within 0.96–0.98 during training (mean 0.97), so ReMiT takes slightly smaller average steps than NTP; scaling NTP's learning rate by this factor would not be expected to reproduce the gain. The reference comparison above, which fixes the weighting rule, shows that the gain depends on which tokens are up-weighted.

W1 / Q5. ReMiT² on all 10 benchmarks

Table R2. ReMiT² on OLMo-1B (%).

Mid-trainGSM8KMATHGPQABBHIFEHEMBPPTQAARC-CMMLU-PAvg.
ReMiT61.6414.5024.5532.0728.5412.809.2025.5849.2317.4427.56
ReMiT²62.7015.7024.3332.7130.7013.4110.8026.0749.6618.2028.43
After RLGSM8KMATHMBPPMBPP+HEHE+ARC-CGPQA-DMMLU-PIFEAvg.
ReMiT65.7319.6611.6022.0928.3524.3944.8025.2516.7370.6232.92
ReMiT²66.1121.1612.8025.5629.2726.2245.3926.3417.4570.5034.08

The second cycle improves both stages (+0.87, +1.16; 9/10 tasks each). A smaller increment is expected with a fixed corpus, post-training data and RL environment, since RLVR mainly sharpens capabilities already in the starting model [3]; larger gains in further cycles come from new data or environments, which ReMiT feeds back into the base model.


We hope these clarifications address your concerns. Thank you again for your time and effort; we would be happy to address any further questions.

References.

[1] AllenAI, Open-Instruct: OLMo 2 training commands. github.com/allenai/open-instruct/blob/main/docs/olmo2.md

[2] AllenAI, Dolmino-mix-1124. huggingface.co/datasets/allenai/dolmino-mix-1124

[3] Yue et al., Does RL Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?, NeurIPS 2025.

Thank you for the new results and I would like to maintain my score if more references are added such as D-CPT Law, EvoLM, https://arxiv.org/abs/2510.14865

We thank the reviewer for the positive feedback and for pointing us to these relevant works. We will add and discuss them in the related work section of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.