acceptodds
Under review as a conference paper at ICLR 2027

Beyond Passive Imitation: On-Policy Mid-Training Distillation for Language Models

Abstract

Continual pre-training (CPT) adapts large language models (LLMs) to specialized domains, yet its standard next-token prediction (NTP) objective treats every corpus token as a fixed target, regardless of its utility to the current model. In a controlled study on the same 1B-token mathematical corpus, we observe sharply different outcomes across base models: NTP improves Llama-3.2-3B's performance on mathematical benchmarks but substantially degrades Qwen3-1.7B-Base's performance, indicating that, under a fixed training recipe, the same training sequences can have model-dependent effects. To address this problem, we introduce On-Policy Mid-Training Distillation (OPMD), which reformulates mid-training from offline text imitation into active, query-driven knowledge acquisition. At selected anchor positions, OPMD samples trajectories from the evolving student policy and applies dense token-level reverse-KL rewards from a frozen teacher, training the model on states it actually visits rather than only on ground-truth prefixes. This on-policy objective reduces the training–generation mismatch of teacher-forced NTP. On MATH-500 and Minerva, OPMD improves the Qwen student over its base performance and outperforms the evaluated NTP baseline, which degrades on both benchmarks. OPMD integrates into CPT pipelines without added inference-time overhead and offers a model-aware alternative to passive domain adaptation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Mid-Training, On-Policy Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_prBBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the model-dependent effects of next-token prediction (NTP) during domain-specific continual pretraining. Motivated by opposite outcomes when applying the same mathematical corpus and NTP recipe to two base models, the authors propose On-Policy Mid-Training Distillation (OPMD). The method selects anchor positions using a frozen proxy model’s predictive entropy, turns preceding contexts into prompts, and trains the student on its own continuations using token-level teacher supervision. Experiments show improvements over the initial Qwen3-1.7B-Base student on MATH-500 and Minerva, avoiding the degradation observed under the evaluated NTP recipe. The paper also compares teacher sizes and anchor-selection strategies.

Critical strengths
  1. The paper examines the dependence of continual-pretraining outcomes on the starting model. The contrasting results obtained with the same mathematical corpus and NTP recipe provide a concrete motivation for investigating training signals better suited to the student’s current state.
  2. The method provides a practical pipeline for applying on-policy distillation to raw domain text without requiring annotated answers or external verifiers. The experiments demonstrate mathematical reasoning gains over both the evaluated NTP baseline and the initial student.
Critical weaknesses
  1. The contribution beyond existing OPD and its connection to domain knowledge acquisition remain unclear. Student-generated trajectories, token-level teacher supervision, and reverse-KL distillation have been studied previously; the main addition is a pipeline for constructing prompts from raw corpora. The experiments do not sufficiently distinguish learning new knowledge from the corpus from transferring capabilities already present in the teacher, or establish the advantage over more direct domain-distillation approaches. Furthermore, the degradation observed under one NTP recipe does not establish that its underlying cause is specifically addressed by on-policy training.

  2. The connection between entropy-based anchor selection and effective knowledge acquisition is insufficiently established. A proxy model’s local predictive entropy measures next-token uncertainty, but does not necessarily indicate knowledge importance, the current student’s learning needs, or the availability of useful teacher supervision. Domain knowledge often depends on broader context, whereas the method uses uncertainty at individual positions to select starting points for extended continuations. The selection-strategy comparisons provide some empirical support, but do not sufficiently explain why the selected anchors represent more valuable learning opportunities or how much they contribute to the overall gains.

Questions for authors
  1. The paper is motivated by opposite outcomes when applying the same corpus and NTP recipe to Llama and Qwen. How is OPMD intended to mitigate this model dependence? The current experiments primarily establish that it improves the Qwen regression case, leaving unclear whether the gains arise from on-policy training or capability transfer from the particular teacher. Can the authors provide results for both original student models using the same source corpus and comparable teacher-supervision conditions, to clarify how the method addresses the motivating observation beyond a particular teacher–student pairing?
  2. What evidence shows that high-entropy positions identified by the proxy correspond to knowledge-learning opportunities from which the current student can benefit through teacher supervision? Can the existing results distinguish the contribution of anchor selection from the benefits of teacher-guided continuation more generally?
Reviewer_hgj4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces On-Policy Mid-Training Distillation (OPMD) as an alternative to standard next-token prediction (NTP) for continual pre-training. The method uses a frozen proxy model to identify anchor positions in mathematical documents based on high entropy, low entropy, or random selection. The text preceding each selected anchor is then used as a prompt for the student model to generate an on-policy continuation. A frozen Qwen3 teacher provides a learning signal based on the Top-16 teacher-student log probability ratios, which are treated as token-level advantages for PPO training.

The experiments mainly focus on Qwen3-1.7B as the student, with Qwen3-4B or Qwen3-14B serving as the teacher. Performance is evaluated on MATH-500 and Minerva using pass@16 and avg@16. The results show that standard NTP substantially hurts the performance of the initial Qwen3 checkpoint under the authors' training setup, while OPMD leads to improvements. The paper also studies different anchor-selection strategies and reports performance across training checkpoints.

Critical strengths
  1. The proposed approach is practical and intuitive. It uses prefixes from the original training corpus as prompts and allows the student to generate its own continuations during training. This provides a reasonable way to incorporate on-policy distillation into continual pre-training.

  2. The experimental results are clearly reported. The paper provides detailed checkpoint results for different teacher models and anchor-selection strategies. Both teacher configurations outperform the initial Qwen3-1.7B student across all four reported metrics. In addition, the anchor-selection ablation uses the same number of retained prompts for each strategy, which helps make the comparison fairer.

Critical weaknesses
  1. The main comparison does not clearly isolate the benefits of OPMD: compared with NTP, OPMD introduces several changes at the same time, including a stronger frozen teacher, student-generated continuations, PPO training, reference-KL regularization, Top-16 truncation, and a different learning rate. In particular, NTP uses a learning rate of 1e-5, while OPMD uses 1e-6. The paper does not include a learning-rate sweep, matched regularization, early-stopping comparisons, or an off-policy distillation baseline using the same teacher. Given these differences, I do not think the significant performance drop observed with NTP is sufficient to show that passive imitation is the main issue or that OPMD is necessary to address it. More controlled experiments would help clarify which components are responsible for the improvements.

  2. The training objective needs more clarification, and the experimental evaluation is somewhat limited: the Top-16 formulation in Equation (10) does not appear to be a standard KL divergence, since the student probabilities are normalized within the selected Top-16 tokens while the log probabilities are computed over the full vocabulary. Equation (11) also does not include the reference-KL penalty mentioned in the experimental setup. I would like to see a more complete description of the PPO objective, including the reference policy, clipping, stop-gradient handling, and action mapping. On the evaluation side, the experiments are limited to one student model, one domain, and two benchmarks, without multiple training seeds or uncertainty estimates. Some ablation conclusions also rely on selecting the best checkpoint based on repeated evaluations on the same benchmarks, which may introduce selection bias. I think additional controlled experiments and clearer implementation details are needed to support the paper's main claims.

Questions for authors
  1. Does the NTP baseline still show the same performance degradation when using a learning-rate sweep, early stopping, replay, and regularization comparable to OPMD? I think these controls are important for understanding whether the degradation is caused by NTP itself or by the specific training configuration.

  2. How does OPMD compare with off-policy knowledge distillation or supervised fine-tuning on teacher-generated continuations? It would be helpful to use the same teacher, anchor prompts, number of updates, and token budget to make the comparison fair.

  3. Could the authors provide a more detailed explanation of the exact PPO objective used in training? In particular, I would like clarification on the reference policy, reference-KL penalty, clipping, mapping from candidates to actions, stop-gradient operations, and normalization. Also, how does Equation (10) approximate the full-vocabulary reverse KL divergence?

  4. Could the authors clarify the evaluation prompts, decoding settings, parsing and truncation rates, and response-length distributions? It would also be useful to know whether OPMD preserves the student's general capabilities beyond mathematical reasoning and whether the training corpus was checked for overlap with MATH-500 and Minerva.

Reviewer_HrGaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies mid-training with the normal next-token prediction (NTP) loss. The premise of authors was that on a 1B-token math corpus, NTP helps Llama-3.2-3B but badly hurts Qwen3-1.7B-Base. So they argue that the NTP does not affect each model similarly.

The authors propose OPMD as training paradigm. A proxy model picks anchor positions in the corpus using entropy, and the text before each anchor becomes a prompt. The student writes a continuation, and a frozen teacher gives token-level feedback using a Top-16 reverse-KL style reward, trained with PPO. Interestingly, the OPMD improves student Qwen model on MATH500 and Minerva but doing NTP was worse, showing the merit of OPMD due to high entropy anchors

Critical strengths
  • The paper is easy to follow and comprehend, and also well written. Ideas were clearly presented. No concerns regarding the experimental setup as well.

  • Using raw corpus text as prompts for on-policy distillation is a simple and new idea. It needs no instruction data or verifier. The effect is worth investigating and the authors did a fair job in the empirical evaluation

  • NTP drop, as authors argued is an interesting warning, and I did not expect this much fall on the benchmark, and hence also raise an alarm for mid training stage.

Critical weaknesses
  • It makes me a bit hesitant to conclude the utility of OPMD (at the same time the issue of fall on benchmark due to NTP), with empirical evaluation on just two model (from just two families). I do not know how difficult and time/resource consuming the training would be and if at all feasible during the rebuttal. Evaluation on more models, may be slightly bigger and another one from different family would have increased my confidence. Since it is an empirical ablation, the certain observation could stem from variety of reasons. I am not saying that the argument is false. It could be a real and practical observation regarding NPT that the authors raised in the paper and OPMD is a useful contribution as well, however, its generalization is a bit of my concern right now.

  • Even though NPT does not hamper Llama, but it would have been nice to see OPMD impact on the Llama as well and what if it does worse than the NPT

  • OPMD uses a learning rate of 10−610^{-6} and NTP uses 10−510^{-5}. OPMD also has a reference KL term that keeps it near the base model, and NTP has nothing like it. A ten times larger step with no such constraint could explain much of the collapse by itself. I do not see an NTP run with the lower LR, an LR sweep, or an OPMD run with β=0\beta = 0.

  • I also feel the gains may not be from the teacher model and I am curious how student can beat teacher by a large margin. How would you justify that? At the same time I feel the argument of 4B and 14B teacher having similar impact on student is believable since the task knowledge or the value added by different sized model could be very similar. Infact I would wonder for some domains, it could also happen that a smaller model with better pre training data for the domain can guide better than bigger teacher.

  • There is no test of non-math ability, even though preserving broader capability is part of the CPT motivation.

Questions for authors

I raised my concerns as part of the weakness section

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.