- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a new way to dynamically re-weight tokens during mid-training phase of models using a Reference RL frozen model leading to improvements in base model capabilities and next phase of post-training with iterative feedback loop.
Critical strengths
- The method is easy to follow and authors showed using 1B/2B/3B models that the bounded token dynamic re-weighting helps improve performance over uniform NTP or other baseline methods.
Critical weaknesses
-
From the Table 1 we can see that gains very across 3 scales of 1B/2B/3B, in 1B model we can see a big gap between the proposed ReMIT approach compared to uniform NTP, but gap shrinks with 2B and 3B, also KD also achieves similar mid-train average performance on OLMo-1B and Youtu-LLM-2B compared to ReMIT. So overall it's not evident to me what specifically the method contributes given the modest performance gains?
-
Authors wrote that they used for SmolLM3 model at sequence length 8192 for mid-training but generally mid-training is done at larger context length, even SmolLM3 did it at 128k , authors also missed a prior work about Mid-training at scale [1] where the paper shows that mid-training performance gains need to be evaluated across different domains and one need to make sure that general short context and long context performance is preserved.
[1]Runwal, Bharat et al. “PRISM: Demystifying Retention and Interaction in Mid-Training.” ArXiv abs/2603.17074 (2026): n. pag.
Questions for authors
- From the Section 4.4, authors showed that RL tuned reference model is better than SFT one, so it raises this question that does this token re-weight mechanism contributes heavily compared to using a good RL reference model? A OPD style baseline would answer this question if student-teacher distillation works as good as using the re-weight token method with RL reference model.
- Does the approach preserves general capabilities of the base model? For example the long context performance I flagged in past section, as most mid-training is done at larger context length, current experiments doesn't answer this.
- I also didn't find any information about the exact data mixture the authors trained their models, can authors point me to them?
4 comments
Thank you for the rebuttal. For the first question the provided table helps but overall, the results suggest that the choice of reference matters, but the incremental benefit of ReMiT over standard KD remains less convincing to me. For the second question I was hinting more towards capabilities like Long context, as I said mid-train is generally done at longer context phase, even in smol-lm 3 they include 4K -> 32K-> 64K context extension as part of mid-training, so claiming ReMiT is orthogonal to context extension does not establish that it preserves long-context capabilities. I will retain my current score based on the current results.
We thank the reviewer for the follow-up and for engaging with our response.
ReMiT vs. KD. With the reference fixed to the same RL model, the mechanism alone changes the final outcome. Measured against uniform NTP after RL (3 seeds), KD gains +1.43 (OLMo-1B) and +1.28 (Youtu-LLM-2B), while ReMiT gains +3.63 and +2.82, i.e., more than twice KD's improvement from the same teacher, on every seed.
This comparison is at matched compute. Under this matched budget, ReMiT delivers more than twice KD's post-training gain. Beyond this, the two signals differ sharply in what they require:
- ReMiT consumes a single scalar per token: the teacher's log-probability of the observed token. Since the teacher is frozen and the corpus is fixed, this signal can be computed exactly in a separate offline inference pass and stored as one number per token, so that mid-training itself runs without the teacher in the loop, at the cost of standard NTP.
- KD consumes the teacher's full distribution. Online, it must retain the teacher's full-vocabulary logits to compute the KL term; offline, it has no exact form: even a truncated top-5 needs 10 numbers per token and is only an approximation, while the full distribution is infeasible to store at 50B tokens.
Moreover, even with the teacher computed online, ReMiT reaches the final accuracy of NTP 3.5× faster in GPU hours (Fig. 16).
Long context. Our comparison is controlled: NTP and ReMiT are trained on exactly the same data, with the same sequence length, positional encoding, token budget, and optimization schedule. ReMiT only reweights the per-token loss within each sequence; it changes neither the inputs the model sees nor how positions are encoded. Any context-extension stage applied afterward would be applied identically to both models, so the comparison remains fair on this axis, and our conclusions are drawn under this matched setting. Time permitting, we plan to extend this matched comparison to longer-context training data, training NTP and ReMiT under identical long-sequence settings.
As the discussion period is closing, we would be grateful if you could let us know whether our latest reply addresses your concerns. Thank you for your time.
We thank the reviewer for the detailed comments.
W1. Modest gains and comparison with KD
Q1. Reference vs. mechanism
We isolate the two factors by varying one at a time (avg. %, OLMo-1B / Youtu-LLM-2B):
With the mechanism fixed, the RL reference beats the SFT reference; with the reference fixed, reweighting beats selection and distribution matching after post-training. Both matter: the reference supplies the signal, and the mechanism decides whether it survives post-training.
We did not include OPD because it is not a like-for-like mid-training baseline. (i) OPD trains on student-generated sequences, so it replaces the mid-training corpus rather than changing the objective on it; a base checkpoint before mid-training also lacks the prompts/task distribution OPD needs. (ii) Generating and scoring 50B tokens autoregressively costs orders of magnitude more than one teacher forward pass on existing text. (iii) The corpus-level counterpart of distillation is KD with the same teacher, which we compare above. OPD is a post-training method and is complementary: it can be applied after ReMiT.
W2 / Q2. Sequence length and general capabilities
Sequence length. We modify the decay/annealing phase on high-quality data and follow each family's setting for it: OLMo-2-1B mid-trains on 50B Dolmino tokens at 4k; SmolLM3's decay phase (10T→11.1T tokens) runs at 4k; Youtu-LLM pre-trains at 8k. SmolLM3 extends context in a separate later stage (4k→32k→64k); 128k is reached by YaRN extrapolation at inference, not by training [2]. ReMiT only rescales per-token losses and does not change data, length or positional encoding, so it is orthogonal to long-context extension and can be applied unchanged in that stage; we excluded it to isolate the objective.
General capabilities. Our suite already covers knowledge and general ability beyond math/code: MMLU-Pro, TruthfulQA, ARC-C, BBH and IFEval. ReMiT is above NTP on all 15 of these results (5 tasks × 3 families, Table 1), and above NTP on 9/10 post-training tasks in each family. This is expected by design: unlike data selection, ReMiT keeps every corpus token with weight in [0.8, 1.2], so no domain is dropped. We will cite PRISM and discuss retention in the revision.
Q3. Data mixture
Mid-training data for all three families follows official documentation; within each family, all methods use the identical 50B-token corpus.
We will add exact sources and proportions to Appendix D.1.
We hope these clarifications address your concerns, and we would be truly grateful if you would kindly reconsider your assessment of our manuscript. Thank you again for your time and constructive feedback.
References.
[1] AllenAI, Dolmino-mix-1124. huggingface.co/datasets/allenai/dolmino-mix-1124
[2] Hugging Face, SmolLM3: smol, multilingual, long-context reasoner. huggingface.co/blog/smollm3