acceptodds
Under review as a conference paper at ICLR 2027

Move Selection Upstream: Choosing What the Teacher Trains On in On-Policy Distillation

Abstract

On-policy distillation (OPD) trains a student on its own generated responses under teacher supervision, and existing data-selection methods choose the student's prompts for a given teacher. Yet even a well-trained teacher performs unevenly across prompts, and selecting student prompts leaves it unchanged. When the teacher can also be trained, can selecting its training data produce a better final student under the same training budget? We select teacher-training data that targets the initial student's weaknesses, prioritizing data that is challenging for the initial student but demonstrably solvable. Our theoretical analysis gives sufficient conditions for teacher-data selection to outperform any selection of the student's prompts at the same training budget: the teacher must improve sufficiently and the student must train long enough. Empirically, teacher-data selection improves final-student performance in instruction following and mathematics, with absolute gains of 6.9% and 4.1% over uniformly sampled teacher-training data. It also outperforms the evaluated student-prompt selection baselines under matched training budgets. In replicated multi-teacher experiments, selecting the instruction-following teacher's data yields absolute gains of 2.2% to 5.2% over uniform sampling and student-prompt selection baselines on two public instruction-following benchmarks. Improving the teacher through data selection can thus be more effective than selecting the student's prompts.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy distillation, data selection, knowledge distillation, teacher training, large language models, multi-teacher distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ShuuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies data selection for on-policy distillation when the teacher is itself trained. It asks whether choosing the teacher's training data can beat choosing the student's prompts at matched training budgets. Prompts are scored by the pass-rate gap between an existing reference model and the initial student, and the top prompts are used to train the teacher with GRPO before the student is distilled. A prompt-local Bernoulli model gives a sufficient condition for teacher-data selection to beat any student-prompt allocation. On SmolLM3-3B, the method improves the final student over uniform teacher data by 6.9% in instruction following and 4.1% in math.

Critical strengths
  • The question is well posed and new for OPD. When OPD teachers are trained for the purpose with multi-teacher domain specialists, the teacher's data is a real selection lever. Comparing it directly with student-prompt selection at matched budgets is a useful framing

  • The core in-domain result is well supported. The gain over uniform teacher data reproduces across five independent fresh-probe selections. The teacher's gain also transfers almost fully to the student.

Critical weaknesses
  • The gain holds at matched training presentations, not at matched total cost, and the paper's own cheaper alternatives do as well or better. Probing costs about 16x a uniform teacher's rollout tokens and needs a reference that already solves the task, while distilling directly from that reference ties on the primary score and is ahead under the five-word floor. Training a uniform teacher 3x longer, still cheaper than probing, gives 51.1% versus 43.5% for selected data, and selection adds nothing at that budget. The experiments never show a setting where this method is the right practical choice.

  • Teacher and student start from the same SmolLM3-3B checkpoint, so the student pass rate in the score is also the teacher's own initial pass rate. In that setup, the method reduces to reference-guided RL data selection for the teacher, which is close to RHO-LOSS-style reducible-loss selection. Targeting the student's weaknesses is never actually tested. The student-prompt selectors also give essentially no gain over uniform in every comparison, so beating them mostly restates beating uniform.

Questions for authors
  • Can you run a setting where teacher and student differ at initialization, for example a larger teacher, and compare scoring with the student's pass rate against scoring with the teacher's own initial pass rate? This directly tests the "target the student's weaknesses" claim.

  • Clarification: The in-domain gains come with shorter responses and shrink or reverse under relative-length thresholds, while the single-teacher IFEval/IFBench gains are unresolved. Beyond the five-word floor, what evidence shows that the students follow instructions better rather than answer more tersely?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Shuu

W1. Total cost and cheaper alternatives

We thank the reviewer for distinguishing matched training budgets from total cost. Our comparison does not establish that selection is cheaper end to end. We match training presentations at each stage, not the cost of constructing the teacher's dataset. The probe takes about 16 times the rollout tokens of a 40-step uniform teacher, or 1.3–2.1 such runs in recorded GPU-hours. Direct reference distillation was unresolved on IF Primary and ahead under Raw-5. A 120-step uniform teacher's student scored 51.1% on IF Primary, against 43.5% for 40-step selected data, but its median answer was 21 words rather than 61. The gain from longer uniform training reversed at both relative-length cutoffs against the 40-step uniform control. That is a qualification on how to read the longer-training result, not a matched-cost win for selection. See Section 4.4 and Appendices D.1.13 and E.5.

W2 / Q1. Shared initialization and student targeting

We thank the reviewer for pointing out that the shared checkpoint makes the two gap scores identical in the main experiments. We compared both scoring rules using different SmolLM3-3B starting checkpoints. The teacher started from a checkpoint trained on uniform data, and the student started from the original supervised checkpoint. With student prompts and training budgets fixed, six paired runs gave 49.76% using the student's initial pass rate and 50.58% using the teacher's. We did not find an advantage from using the student's pass rate. As with RHO-LOSS, reference performance guides selection. Here we measure its effect on the distilled student. See Section 2 and Appendix E.4.

The student-prompt rules were close to uniform: 39.5% for the gap rule and 40.6% for the learnability filter, versus 39.9% for uniform prompts. The contrast is informative because we also applied the gap rule to teacher data and observed a gain. This does not rule out other ways of selecting student prompts. See Appendix E.2.

Q2. Beyond a five-word minimum

We thank the reviewer for asking whether the gains reflect better instruction following or just shorter answers. The placement advantage survives both a stricter length check and a separate requirement that answers complete the task. We applied the same short-answer penalty to every teacher and compared six teacher–student pairs per arm: gap-selected teacher data with uniform student prompts versus uniform teacher data with gap-selected student prompts. We fixed the comparison before scoring 750 new prompts outside the Open-MOPD training corpus. The IF Primary gain was an absolute 6.17%. Requiring each answer to reach 25% or 50% of the initial student's median length for that same prompt left gains of 5.94% and 3.98%, respectively. Unlike a five-word minimum, this check uses a different threshold for each prompt. See Section 4.5 and Table 5A.

Length by itself does not establish completion. In a separate audit, Claude Opus 5.5 judged full answers without seeing which model wrote them. On the 183 prompts in the main analysis, counting only verifier passes rated as complete left a gain of 6.2%. An exploratory check of these same models also found gains over uniform of 3.1% on IFEval and 3.9% on IFBench. These results address the brevity concern for the follow-up with the short-answer penalty. The original runs remain sensitive to length, and their public benchmark differences in Table 1 remain unresolved. See Appendix E.2.

Reviewer_bsgUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes selecting teacher-training prompts to improve the final student in on-policy distillation. It ranks candidates by the verified success-rate gap between an existing reference model and the initial student, with response-length stratification, and trains a teacher on the selected prompts before distillation. Conditional theory and experiments support benefits over uniform teacher data and evaluated student-prompt selectors at matched training-use budgets, while comparisons with direct reference distillation and longer uniform training limit the practical conclusions.

Critical strengths
  1. Useful selection-stage comparison. The paper examines an actionable choice in pipelines that train their own teachers and compares the same gap score at both stages. The distinction between improving supervision and reallocating student prompts is informative.

  2. Extensive controls and transparent reporting. Independent probe-to-student replications support the IF gains, alongside mathematics and multi-teacher experiments. The paper also reports direct-reference baselines, longer training, length sensitivity, and negative results.

Critical weaknesses
  1. The selection score's added value is insufficiently established. The experiments do not establish that higher reference–student gaps produce better final students, and cutoff-only sampling shows no resolved disadvantage to full ranking. The score also does not distinguish persistent capability gaps from errors that the student could correct itself.

  2. The extra teacher-training stage lacks a clear practical benefit. Gains over direct reference distillation are small and inconsistent across paired runs, while Raw-5 favors direct distillation. The additional probing and teacher-training costs therefore lack a demonstrated payoff in the tested setting.

  3. The single-teacher distillation stage is not clearly justified. The GRPO-trained teacher and final student have the same model size and nearly identical IF performance. The paper does not establish what this stage offers over deploying the trained teacher directly, although multi-teacher integration provides a separate motivation.

  4. Model coverage is limited. The main teacher and student experiments use SmolLM3-3B, without a systematic comparison across sizes or families. Changing the reference model does not establish that the pipeline generalizes to different teacher–student configurations.

Questions for authors
  1. Does final-student performance increase across controlled low-, medium-, and high-gap selections, holding dataset size and length composition fixed? Does distinguishing correctable errors from persistent failures improve selection?
  2. When does training an additional teacher offer a reliable advantage over direct reference distillation after accounting for selection and training costs?
  3. In the single-teacher setting, what does distilling into a same-size student provide over directly deploying the GRPO-trained teacher? Are there benefits in other capabilities or deployment requirements?
  4. Do the results hold for different teacher/student sizes and families, beyond changing only the reference model?
2 comments
AuthorsOpenReview ↗

Response to Reviewer bsgU

W1 / Q1. Gap size, cutoff-only sampling, and correctable errors

We thank the reviewer for asking whether higher gap scores pick better teacher data and whether ranking adds anything beyond a cutoff. High-gap data helped. Low- and middle-gap data did not. We grouped candidate prompts by response length, then drew equal numbers from the bottom, middle, or top of each group's gap ranking. Each teacher dataset had 640 prompts. We used six teacher–student pairs per arm and kept both training budgets and the student prompts fixed. On a separate 750-prompt test set, IF Primary scores were 28.8% (low), 29.2% (middle), and 39.1% (high), against 33.0% for uniform data. High beat low by an absolute 10.3% and by 7.7% under the stricter length check. Thus the useful region is the top band in this experiment. The data do not show a steady rise across bands. See Section 4.5 and Table 5B.

The cutoff accounts for most of the gain. We kept the same dataset size and length quotas but sampled at random from prompts that the reference solved more often than the initial student. This scored 36.9%, compared with 33.0% for uniform data and 39.1% for the top-ranked data. Ranking added 2.2% on IF Primary, but the stricter length and task-completion checks did not resolve that smaller advantage. See Appendix E.3.

A failed attempt does not establish that the student cannot solve the prompt. Of the 4,415 candidate prompts the reference solved at least half the time, the initial student failed all 24 attempts on only 98. Repeated sampling tells us what the starting model can already solve on a retry. It does not tell us what that model could learn after training. We did not test a separate selection rule based on correctability. See Appendix E.3.

W2 / Q2. When is training a teacher worth it?

We thank the reviewer for asking when training a teacher is worth the extra cost. We do not show a total-cost advantage over direct reference distillation. At the same student training budget, the main instruction-following score did not distinguish training a separate teacher from using the reference directly. Raw-5, which scores answers under five words as incorrect, favored direct distillation. We therefore cannot recommend adding a teacher stage on cost grounds when a suitable reference is already available. Our recommendation is narrower: if the pipeline already trains a teacher, the experiments compare where to use selection at the same per-stage training budgets. See Section 4.4, Appendix D.1.4, and the revised conclusion.

W3 / Q3. Why not deploy the same-size teacher?

We thank the reviewer for asking what the student offers if the trained teacher is already available. The single-teacher runs show transfer, not a deployment advantage. If only instruction following is needed, the trained teacher is a reasonable choice. Nor did we see a clear code advantage from distilling: across 12 matched pairs, student and teacher scored 9.32% and 9.34% on LiveCodeBench. The multi-teacher setting is different: one student learns from instruction-following, mathematics, and code specialists. Selecting data for the instruction-following teacher improved the shared student's instruction-following scores without changing the other specialists. See Section 4.3 and Appendix E.5.

W4 / Q4. Other families and model sizes

We thank the reviewer for asking whether the result holds for other model families and sizes. We tested a second family with both equal-size and smaller students. We trained Qwen2.5-1.5B teachers on selected or uniform data and distilled them into Qwen2.5 students of 1.5B or 0.5B, with three runs per condition. This changes the teacher and student, not just the reference. On a development set, selection raised the final student's IF Primary score from 3.91% to 12.00% at 1.5B and from 1.72% to 6.10% at 0.5B. Much of the equal-size gain came from avoiding nearly empty answers in two uniform-data runs, and no 0.5B student beat its starting checkpoint. This gives another model-family comparison against uniform teacher data, but it does not show that teacher-data selection beats student-prompt selection in Qwen2.5. See Appendix E.6.

Thank you for the additional experiments and clarifications. Simple cutoff-based sampling captures most of the benefit, and the additional gain from ranking is not consistently supported by the stricter length and task-completion checks.

The revised scope also limits the contribution to pipelines in which teacher training is already required, without establishing an advantage over direct reference distillation or deploying the trained teacher in the single-teacher setting. While the selection-placement study is informative, I remain unconvinced that the incremental methodological contribution is sufficient for acceptance. I will keep my original score.

Reviewer_QGjCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes selecting the teacher's RL training data rather than the student's prompts in on-policy distillation: prompts are ranked by the reference-model minus initial-student pass-rate gap (weak for the student, solvable by the reference), length-stratified, then used to GRPO-train a teacher that is distilled into the student.

Critical strengths
  1. Budgets are matched across stages, and the same scoring rule is applied at both the teacher and student stages, cleanly isolating the effect of where selection acts.
  2. The paper are well organized.
Critical weaknesses
  1. Stripped of the distillation framing, the contribution amounts to solvability-gated hard-example selection before RL, with limited novelty. The prompt-selection method for the RL model is itself straightforward: using pre-computed pass rates, it selects prompts on which the initial model is weak but whose learnability is confirmed by a reference model.

  2. The distillation stage adds no demonstrated novelty: it is a standard on-policy objective fitting the student to stronger frozen teachers.

Questions for authors

The authors could examine whether training the teacher too strong on selected prompts hurts distillation: selecting data that keeps the teacher within reach of the student could beat maximizing teacher accuracy.

1 comment
AuthorsOpenReview ↗

Respond to Reviewer QGjC

W1 / W2. What the paper contributes

We thank the reviewer for asking what is new beyond the gap score and the distillation objective. At the same training budgets, the gap rule helped more when applied to teacher data than to student prompts. We trained teachers on selected data and used uniform student prompts in one arm, then reversed those choices in the other. We used standard on-policy distillation and matched the number of training presentations at each stage, with six teacher–student pairs per arm and the same short-answer penalty for every teacher. On 750 new prompts outside the Open-MOPD training corpus, the final student scored 45.7% with teacher-data selection and 39.5% with student-prompt selection. The absolute gain was 6.17% and held under both the relative-length checks and a separate task-completion check. The result shows why the choice of stage matters even when the scoring rule itself is unchanged. See Section 4.5 and Table 5A.

Q1. Could a stronger teacher hurt the student?

We thank the reviewer for raising the possibility that a better teacher might not lead to a better student. The student benefited from the teacher's gains in our comparison. With student training fixed, high-gap rather than low-gap data raised the teacher's score from 29.0% to 38.6% and the student's from 28.8% to 39.1%. Theorem 1 likewise treats teacher improvement as insufficient by itself: the student must absorb enough of that improvement. See Table 5B and Appendix E.3.

Follow-up on the contribution and teacher strength

We thank the reviewer. We agree that the score is simple and the distillation objective is standard. We keep the objective and both training budgets fixed to compare where selection is applied.

W1 / W2. What the paper contributes. We compare improving the teacher through its training data with choosing student prompts for a fixed teacher. On the same test set, applying the gap rule to student prompts scored 39.5%, not clearly different from 39.9% with uniform prompts. Applying it to teacher data raised the final student's score to 45.7%, an absolute gain of 6.17% over student-prompt selection (Table 5A and Appendix E.2). Theorem 1 gives conditions on teacher improvement and student training under which teacher-data selection beats even the best choice of student prompts with the baseline teacher.

Q1. Could a stronger teacher hurt the student? We did not test whether deliberately keeping the teacher closer to the student would help. On the separate test set in Section 4.4, the gains over uniform teacher data were 5.9% for the teacher and 5.7% for the student. The gain transferred in that comparison, which does not show that a stronger teacher always helps.

Robustness checks. The revised paper tests placement on previously unused prompts from outside the training corpus. The gain holds with a minimum answer length and in a separate blinded check that counts a pass only when the full answer completes the task (Section 4.5 and Appendix E.2). We would be glad to address any remaining concerns about the design or evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.