RL-Guided Mid-Training for Iterative LLM Evolution

Standard training pipelines for large language models (LLMs) are typically unidirectional, progressing from pre-training to post-training. However, the potential for a bidirectional process—where insights from post-training retroactively improve the pre-trained foundation—remains unexplored. We aim to establish a self-reinforcing flywheel: a cycle in which reinforcement learning (RL)-tuned model strengthens the base model, which in turn enhances subsequent post-training performance, requiring no specially trained teacher model. To realize this, we analyze training dynamics and identify the mid-training phase as a critical turning point for model capabilities. This phase typically occurs at the end of pre-training, utilizing high-quality corpora under a rapidly decaying learning rate. % While utilizing RL models to guide this phase via Knowledge Distillation (KD) seems intuitive, we demonstrate that KD suffers from critical limitations: it induces overfitting to the teacher model, thereby impairing generalization and yielding temporary gains that diminish during subsequent post-training. Building upon this insight, we introduce ReMiT (Reinforcement Learning-Guided Mid-Training) as a concrete feedback mechanism, which leverages the reasoning priors of RL models to dynamically reweight tokens during the mid-training phase. Empirically, ReMiT achieves an average improvement of 3% on 10 pre-training benchmarks, spanning math, code, and general reasoning, and sustains these gains by over 2% throughout the post-training pipeline. These results validate an iterative feedback loop, enabling self-reinforcing evolution of LLMs.