acceptodds
Under review as a conference paper at ICLR 2027

StyleOPD: Selecting the Generation Prompt for On-Policy Distillation

Abstract

On-policy distillation (OPD) supervises a student on its own rollouts. Recent work improves that supervision by changing what the teacher is conditioned on or which tokens receive loss, while the prompt under which the student generates its rollouts is left at the task default, even though those rollouts determine what supervision exists at all. We treat the generation prompt as a selection variable and choose it by measurement. Reusing the criterion behind teachability-based token selection (corrective disagreement the student can still act on), we turn a per-token gate into a per-prompt score, computed once on the untrained student with no labels, no training and no change to the rest of the OPD recipe. In a controlled study distilling Qwen3.5-27B into Qwen3.5-9B, the choice of prompt moves accuracy by more than the prompt-free OPD gain over the initial student, every added prompt improves on vanilla OPD averaged over training, and the selected prompt beats vanilla OPD on every seed. Disagreement is what decides: every disagreement-based signal we measured, including our score, selects the best prompt, while every rule built on agreement or accuracy selects a worse one, and the best training prompt is the one under which the untrained student is least accurate. The selected prompt outperforms a sequence-level distillation baseline that needs extra teacher data, and it composes with token selection: the prompt helps on every seed with or without token selection, and more with it over training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, knowledge distillation, large language models, prompt selection, mathematical reasoning, teachability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aAoMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a teachability-aware OPD based label-free and training-free prompt selection score. The score is a combination of (1) teacher-student disagreement in distribution, which quantifies the possible corrective signal, and (2) the teacher probability mass among the student’s top-k tokens, approximating the ease of the student to absorb that signal. Initial student rollouts and teacher-forced grading are used to score candidate prompts, without training separate models. The experiments on math reasoning benchmarks show PSV prompt improves over vanilla OPD by 1.92 AVG5 points at the best checkpoint on an average of 3 seeds. The combining prompt selection with teachability-based token selection is at 72.09 AVG5. Also the disagreement-based metrics predict useful prompts more reliably than initial model accuracy, and support weighting may improve ranking among closely matched candidates.

Critical strengths

(1). The paper presents interesting and practical motivation. It highlights an underexplored aspect of on-policy distillation: how the generation prompt affects the supervision available to the student. The proposed training-free, label-free selection approach is computationally efficient and can be useful in practice.

(2). The paper evaluates multiple prompts across full training runs and repeat the main comparisons over three seeds. The selected prompt consistently improves over vanilla OPD and also works well with token selection. These results support the importance of prompt choice, even if the advantage of the specific scoring function over simpler alternatives is less convincing.

Critical weaknesses

(1) The novelty of this paper is limited. The paper added a simple disagreement metric as KL divergence and select the same winning prompt as the proposed score. The additional benefit of support weighting is demonstrated is through a small, single-seed comparison with a tiny difference (0.51 points, see table 1 Canonical Freeform vs Tabular-CoT curve-arg score). this is within reported seed variability. Also, at the best checkpoint, Tabular-CoT performs better than Canonical Freeform (see table 1 AVG5). So the key methodological contribution is not convincing.

(2). The evidence is insufficient to support the mechanism. The same prompt is given to the student and the teacher, changing both the student's sampled trajectories and the teacher's supervision. it is unclear that wether the improvements come from more teachable student trajectories or just better teacher responses under the selected prompt, since there is no ablations separating these effects. This weakens the paper's explanation of why the proposed approach works.

Questions for authors
  1. The experiments use one teacher–student pair and focus on math reasoning. Do you have evidence that the method works across other models or task domains?

  2. please see weakness above.

Reviewer_yVMUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

OPD consists in the following steps: sample the student model on prompts x to get sequences y. This create a batch of (x,y) Perform knowledge distillation on this batch with a teacher and computing gradient only on the generated part of the dataset y. Variant of OPD, TA-OPD proposes to optimize only on a subset of tokens selected with pertinent metrics on the teacher and student token level distribution. Another variant of OPD proposes to add a specific task-preserving preprompt p after x to create the batch of (x,p,y) and then to train on y. More precisely, the tested preprompts constrains the reasoning style inside <think> without changing the problem or final-answer format. The authors propose a compute efficient method to select the best p among a given pool of preprompt. To do this the author proposes a score J(p) that is computed by generating student traces on a small sample of (x, p) and then to teacher-force the teacher on this scoring dataset (x,p,y). At a step t of y, the local score is an aggregation of 2 score, the disagreement on top_k D_k and the top_k support overlapping C_k. K is the union of top k tokens of teacher and student Dk is the KL between the student distribution over K and the teacher distribution over K. It measures the disagreement between teacher and student while neglecting the tail disagreement. Ck: is the mass the teacher prob gives to the top k of the student. It measures the extent the student must do a radical change to its logic or if the student and teacher are globally agreeing on the head of their distrib. The token level score is then N(Ck)*N(Dk) and the global score is averaged over the whole sentence and over the samples. Where N is quantile normalisation.
The method is evaluated with Qwen 3.5-27B as teacher and Qwen 3.5-9B as student with a math OPD dataset and math evaluation. The pool of preprompt evaluated is a selection preprompt from the literature.

Critical strengths

The idea is interesting and promising. The experimental setup is coherent. The selection criteria is compute efficient.

Critical weaknesses

The readability is very poor: the text is complex compared to the simplicity of the method, very long phrases, hard to follow. The figure 1 could easily be made more readable. A change in my score could not be considered without a rewriting of the paper. Line 255 - 257 “Each prediction is checked as a decision: a high-score group should outperform a low-score group, and the chosen prompt should have low regret. The pool, score direction and group boundaries are fixed for all audited signals, and every candidate is reported.” -> What does it mean ?

Questions for authors
  1. Clarification: In table 1 the J order is not preserved in the downstream final metric, which if I understood well weakens the methods proposed by the paper of using J as a selection of p_star.

  2. Suggestion: The method seems to saturate on C_k so why not take a smaller k where k=16 comes from ? Also since C_16 is almost 99% everywhere there is almost no signal in it but this signal is stretched by the normalisation method. How is k chosen?

  3. Suggested revision: The missing checkpoint and evaluation also weaken the results, are the mean computed only on the intersection of the existing checkpoint. Why not rerun the xp ?

Reviewer_3tCbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Introduces: generation prompt for on-policy distillation with a label-free, training-free score J(p). J -> per-token teacher-student KL on the top-k union, multiplied by the teacher mass on the student’s top-k, quantile-normalized and averaged over calibration rollouts from the untrained student. They only test it on Qwen3.5-27B down to 9B on DAPO-Math-17K. The method beats vanilla OPD by 1.92 AVG5 at the best checkpoint. The text is clearly AI generated.

Critical strengths
  • Good presentation, clearly communicated, admitted what weaknesses (but not all of them)
  • Fairly counterintuitive finding: the best prompt is the one under which the untrained student is the least good
Critical weaknesses
  • Base ODP barely works (see table 1+3, line ~272, ~277, ~381)
  • Best checkpoints are selected on the test benchmarks, with no held-out test split (?). minor:
  • table readability should be improved, highlighting should be used here
Questions for authors

Clarifying the base ODP issue as well as testing splits mentioned above.

Comments on the paper

1 comment

Dear Reviewers,

The authors have posted responses. As the discussion period is ending soon, could you please take a look and reply shortly, or at least acknowledge that you have read it?

Thank you!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.