acceptodds
Under review as a conference paper at ICLR 2027

EFFICIENT ON-POLICY DISTILLATION OF LONG REASONING VIA OFF-POLICY PREFIX DISTILLATION

25 views

Abstract

On-policy distillation (OPD) trains a student on its own reasoning with dense teacher feedback. However, the teacher must supervise long reasoning traces that it would not have written itself, which makes the student learn slowly. We observe that the accuracy of the student greatly increases when it continues from the beginning of the reasoning generated by the teachers, and vice-versa the performance of the teacher is degraded when it has to start from a partial reasoning traces written by the student. This large effect of the reasoning prefix motivates our proposed OPD variant, **Prefix-Supervised On-Policy Distillation (PS-OPD)**. At each training step, the student continues a reasoning prefix generated offline by the teacher, and it is trained by supervised fine-tuning on the prefix and OPD on its own continuation. In this way, the student will generate higher quality answers and receive more informative supervision from the teacher, as demonstrated by faster convergence and better final performance of PS-OPD compared to OPD. For example, on AIME24, AIME25 and AIME26 with a 7B math teacher and 1.5B student, PS-OPD reaches in training steps the accuracy OPD attains after steps, ends points higher, and needs about fewer GPU-hours for training, including the generation of teacher traces. The gains carry over to a teacher fine-tuned by reinforcement learning from the student itself: PS-OPD leads OPD over the first ten steps and ends points higher. PS-OPD is also more robust to the choice of distillation loss: across four forward- and reverse-KL losses, its final accuracy spans and points with the two teachers, against and for OPD. It also curbs OPD's response-length explosion: with the RL-finetuned teacher under the main loss, up to of OPD's training rollouts reach the length cap, against at most of PS-OPD's.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, knowledge distillation, large language models, long chain-of-thought reasoning, teacher prefix, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mBTKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes PS-OPD, a variant of OPD in which the student continues a teacher-written reasoning prefix. The method is motivated by a two-way experiment: a teacher prefix lifts the student while a student prefix drops the teacher. The evaluation on AIME24/25/26, PS-OPD reaches OPD's step-50 accuracy in 7 steps and uses about 40% fewer GPU-hours, including the cost of generating teacher traces.

Critical strengths

1.The prefix experiment directly motivates the design, and the gains in speed and GPU-hours hold even after counting the one-time cost of teacher traces. 2.PS-OPD curbs OPD's length explosion and is less sensitive to the choice of loss.

Critical weaknesses

1.Teacher text as rollout context (Relay-OPD[1], Prefix-GRPO[2], ReOPD[3]) and SFT combined with OPD (TrOPD[4]) both already exist. The paper offers no theory or analysis of why this particular combination is the right one, so the contribution is an empirical recipe rather than a new idea. 2.The method is heuristic and depends on tuning. Performance hinges on hand-set knobs: λ = 0.3, a cut ratio of r = 50%, an 8,192-token prefix cap, and dropping the ~15% of questions without a usable trace.

[1]Xu, H., Xu, X., Hong, H., Ni, Z., Li, H., Qiu, Y., ... & Shen, Y. (2026). Pass the baton: Trajectory-relayed on-policy distillation. arXiv preprint arXiv:2607.26057. [2]Wang, Yihan, et al. "From Trajectories to Prefixes: Reusing Teacher Trajectories via Replayed Prefixes and Online Continuation." arXiv preprint arXiv:2607.19395 (2026). [3]Liao, Baohao, et al. "Multi-Turn On-Policy Distillation with Prefix Replay." arXiv preprint arXiv:2607.04763 (2026). [4]Xing, Xingrun, et al. "Trust Region On-Policy Distillation." arXiv preprint arXiv:2606.01249 (2026).

Questions for authors

1.What distinguishes PS-OPD from prior prefix-based OPD? The RH (λ = 0) arm uses the teacher prefix only as rollout context, which looks essentially like Relay-OPD and ReOPD. If so, the only new ingredient is the added SFT term. 2.Does any of the advantage survive outside the 89 tuning problems? λ = 0.3 was selected on the same 89 AIME problems used for every reported number, including the confirmatory fresh-seed test. Do you have results where λ is chosen on a held-out set, or evaluated on benchmarks not used for selection?

3 comments

Thank you for the detailed review. In the following we address your concerns about novelty of our method and its dependence on particular, dataset-specific parameters.

W1 (theory). Why is this combination the right one?

While we do not have a complete theory of why our approach improves over the baselines, our method is motivated by several empirical observations and supported by the experimental results, where it outperforms the baselines. We note that also the baselines are similarly based on heuristics rather than rigorous theoretical proofs of optimality.

We agree that the single ingredients exist, but our main contribution is how to combine them in a simple way which leads to faster convergence and better accuracy. There is a simple reason to combine them: the student's gap to the teacher has two main parts, and PS-OPD puts one term on each.

(1) The experiment in Figure 2 motivates this combination. The start of the reasoning has a large effect on the final answer. A teacher prefix lifts the student: when the student continues the first 75% of the teacher's reasoning, its accuracy rises from 19.6% to 45.3%. A student prefix drops the teacher: when the teacher continues the first 75% of the student's reasoning, its accuracy falls from 59.9% to 31.8%. So the student should learn to write a good opening, not only to continue one. The prefix SFT term teaches this opening. So the two terms fit together: both move the student toward the teacher's next-token distribution, SFT on the teacher's opening and OPD on the student's continuation.

(2) Table 26 gives the numbers behind Figure 2 (frozen models, on AIME). It splits the gap into the two main parts: the opening and the continuation. Without a prefix, the teacher scores 59.9% and the student 19.6%, a gap of 40.3 points. At the 50% cut, 16.0 points come from the opening: the student gains 16.0 points when it continues the teacher's opening instead of writing its own. Another 23.3 points come from the continuation: when both continue a teacher opening, the student still scores 23.3 points below the teacher. The last 1.0 point is lost when the teacher continues its own trace, cut at 50%. The OPD term trains the continuation. The prefix SFT term trains the opening. On prefixed questions, it is the only loss on the opening. RH, our baseline that uses the prefix only as context, puts no loss on the opening. The opening part grows with the cut: 7.8, 16.0 and 25.7 points at 25%, 50% and 75%.

This is why PS-OPD has both terms. Our decoupled baseline also has both terms, but its OPD starts from the question alone. So whether OPD should continue the same prefix is a separate question. W1 / Q1 below answers it.

Q2. Is λ tuned on the test problems, and do the gains hold on unseen problems?

Yes, we chose λ = 0.3 on the 89 AIME problems but (1) λ = 0.3 achieves the best performance on all three individual AIME test sets (AIME24/25/26) and (2) the gains hold a new test set of 210 unseen problems, as we detail below. Therefore, this choice generalizes well across datasets and it is not the reason for which PS-OPD over the baselines.

Check 1: each AIME year alone. λ = 0.3 is the best weight on each AIME year separately. We compare at training step 50, the checkpoint we used to choose λ. Table 20 gives these values in the column "50" under each year, averaged over seeds 1–3. With λ = 0.3, accuracy is 35.7% on AIME24, 27.3% on AIME25 and 23.9% on AIME26. The best of λ = 0, 1 and 3 reaches 34.2%, 26.9% and 21.8%. So if we choose λ on any two years, we still pick 0.3, and it is also the best on the third year. Most of these per-year margins are within seed variation. If we choose λ on a random half of the 89 problems instead, we pick 0.3 in about 99% of the splits.

Check 2: a new held-out test. Table A in item 1 of our general response (and repeated below) gives the results on 210 new contest problems, which we never used to choose λ or any other setting. With λ = 0.3 unchanged, PS-OPD has higher accuracy than OPD for all step ranges (steps 1–10, steps 20–50 and step 50) with both teachers, and it reaches OPD's step-50 accuracy at step 7 (Skywork) and at step 20 (JustRL).

Table A. Accuracy on the new test set. Values in % (avg@16), averaged over the six contests and seeds 1–3. For each teacher, the columns give the mean over training steps 1–10, the mean over steps 20–50 (checkpoints 20, 30, 40 and 50), and the accuracy at step 50. Δ\Delta is computed before rounding.

MethodSkywork 1–10Skywork 20–50Skywork 50JustRL 1–10JustRL 20–50JustRL 50
OPD15.919.319.718.614.724.9
PS-OPD (ours)17.521.121.221.027.828.0
Δ\Delta (PS-OPD - OPD)+1.66+1.85+1.49+2.39+13.04+3.05

W2. Does PS-OPD depend on hand-tuned settings, and are questions dropped?

PS-OPD's gains over OPD, in speed and in accuracy, do not depend on hand-tuned parameters (exact values of λ or the cut ratio). We did not tune the prefix cap, and no questions are dropped.

λ = 0.3. This is a stable choice (see Q2). At step 50 it is the best weight on each AIME year. We fixed a single value, chosen on one teacher, then we used the same value with the second teacher (JustRL), with all four distillation losses and on the held-out test. With it, PS-OPD's step-50 accuracy, averaged over the three AIME years, is above OPD's for both teachers and all four losses (Table 2).

The gains also do not need exactly λ = 0.3. With λ = 0.3, 1 and 0, the student reaches OPD's step-50 accuracy at step 7, 8 and 9. OPD needs 40 steps (Table 9). Only λ = 3 is slower, at step 30. Every weight we tried (λ = 0, 0.3, 1 and 3) also has higher accuracy than OPD over steps 1–10 and over steps 20–50 (Table 20).

Cut ratio r = 50%. We ran cut ratios 0.125, 0.5 and 0.75, each at λ = 0 and 1, with one seed each (Table 22). All six runs average 21.2–24.2% over steps 1–10, against 20.9% for OPD on that seed (Table 8).

8,192-token prefix cap. The cap only shortens the prefixes of the longest teacher traces. These are traces whose reasoning is longer than 16,384 tokens, twice the cap. On average, the cap lowers the prefix from 50% to 49.1% of the reasoning span (Appendix G). So it changes the prefixes very little, and we did not tune it, therefore it is not expected to affect the results.

Questions without a usable trace. None are dropped. They still train with plain OPD (Appendix A). We will clarify this in Section 3.2.

W1 / Q1. How is PS-OPD different from prefix-based methods, and is the SFT term the only new part?

Compared with methods that use teacher text only as context, yes, the SFT term is the new part. Compared with methods that also train on teacher text but start the student's rollout from the question alone, the new part is that the rollout continues the same prefix that SFT trains. Our baselines show that both parts help. Therefore, PS-OPD has significant differences to each of the existing baselines: it is simpler than existing methods and reaches OPD's step-50 accuracy much earlier, with about 40% fewer GPU-hours to get there (Table 9), and at step 50 it is 2.12 points above OPD (Table 6).

We also note that all three are arXiv preprints, not peer-reviewed papers: TrOPD (May 31 2026), ReOPD (July 6 2026) and Relay-OPD (28 July 2026, seven weeks before the deadline). We still discuss all of them, and we compare with TrOPD in experiments. Prefix-GRPO, which you also cite, has been withdrawn by its authors, and it does not anyway involve OPD-variants and thus it is not directly relevant to our work.

Item 2 of our general response compares PS-OPD with TrOPD, ReOPD and Relay-OPD in one table.

Prior methods.

  • Relay-OPD (arXiv:2607.26057): the teacher takes over briefly inside the student's live rollout.
  • ReOPD (arXiv:2607.04763): replays teacher histories in multi-turn tool use and trains selected student actions.
  • TrOPD (arXiv:2606.01249) is the closest. In its paper (Sections 4.2–4.3, Eq. 8), the student continues a teacher prefix. The maximum prefix length starts at the maximum training sequence length and is cosine-annealed to zero, so training ends fully on-policy. The prefix gets a forward-KL loss with a small weight (0.001). Each student token is kept in a "trust region" with probability min(1, p_teacher / p_student) and gets the usual OPD loss. The other ("outlier") tokens get a separate top-k forward-KL loss. Its released code, which we used for our baseline, works differently. There, the student starts from the question only, and the teacher traces are trained as separate sequences, not as the start of the student's rollout.

PS-OPD. We cut one offline teacher trace at one fixed point: half of the reasoning, at most 8,192 tokens, the same at every step. The student learns this prefix with plain SFT, with weight λ = 0.3. It continues the prefix with standard OPD on all its own tokens. There is no live teacher generation, no prefix schedule and no token filter, so there are far fewer settings to tune.

Our baselines. RH, our baseline with λ = 0 (Table 20), is the context-only case. The teacher prefix is context, and the loss is on student tokens only, as in ReOPD. Unlike in Relay-OPD, the teacher does not write inside the rollout. Our decoupled baseline has the same structure as TrOPD's released code. It keeps prefix SFT but starts OPD from the question.

  • Adding prefix SFT to RH gives +1.85 points over steps 1–10 and +0.73 over steps 20–50, in all three seeds (Table 6).
  • The decoupled baseline gets most of the early gain. PS-OPD leads it by only 0.42 points over steps 1–10, within seed variation. But over steps 20–50, PS-OPD is 1.36 points higher (Table 25), and 1.75 points higher on three new seeds (4–6). Over steps 20–50, PS-OPD leads in every seed of both sets.

 

Overall, we believe that these clarifications and additional experiments clearly show the novelty and effectiveness of PS-OPD, as weIl as the fact that its advantage does not really on unfairly tuned parameters. Therefore, we ask the reviewer to reconsider their initial score. We are happy to answer any additional questions.

Reviewer_EfMUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the limited effectiveness of teacher guidance on student-generated reasoning trajectories, the high cost of long-sequence generation, and training instability in on-policy distillation. It proposes Prefix-Supervised On-Policy Distillation (PS-OPD): the student continues from a teacher-generated reasoning prefix to complete the response, while training jointly applies a supervised fine-tuning loss to the teacher prefix and an on-policy distillation loss to the student’s continuation. Its main contribution is to combine prefix conditioning with direct prefix supervision to improve distillation performance, convergence speed, and training stability. The authors evaluate the method’s effectiveness and investigate the sources of its gains through comparative experiments and ablation studies.

Critical strengths
  1. The paper presents a simple and effective improvement to address the high cost of student generation and limited learning efficiency in long-reasoning distillation. Through teacher-prefix conditioning and direct supervision, PS-OPD reaches standard OPD's accuracy at step 50 more quickly. The computational cost analysis also accounts for the overhead of pre-generating teacher reasoning trajectories, providing evidence of its efficiency gains.

  2. To address training instability in on-policy distillation, PS-OPD mitigates the response-length explosion and accuracy degradation observed in standard OPD under certain configurations. Additionally, the method maintains strong performance across different KL directions and top-k token selection strategies, demonstrating robust adaptability to the chosen distillation objective.

Critical weaknesses
  1. The paper's main methodological contribution is a coupled design: the student uses a teacher-generated reasoning prefix as context for continuation, learns the prefix via SFT, and applies OPD supervision to the subsequent student-generated continuation. However, related ideas have precedent. As a baseline, TrOPD combines on-policy and off-policy distillation, and its paper also discusses gradually annealing the teacher prefix during training. The main incremental contribution of PS-OPD is using the same teacher prefix both as the continuation context and as an SFT supervision target, while jointly optimizing prefix supervision and OPD on the subsequent generation. Therefore, its methodological innovation remains incremental.

  2. The main text does not present component ablation results that directly examine the core coupled design and therefore does not clearly clarify the teacher prefix's contributions as context for continuation and as an SFT supervision target, or the additional gains from combining the two. Although the appendix provides relevant control experiments, the decoupled control reproduces most of the early performance gains of PS-OPD over OPD, suggesting that additional prefix supervision alone may explain a substantial portion of the gains. Furthermore, the later-stage comparison changes both the continuation context and the generation budget, making it difficult to clearly attribute PS-OPD's performance advantage to the coupling of the two roles of the prefix.

Questions for authors
  1. The authors state that they changed TrOPD’s learning rate and weight decay from the released configuration. The original TrOPD paper reports 32.50% on AIME25 with the same teacher–student model pair, compared with 26.0% here, although the training and evaluation configurations differ. Does PS-OPD retain its advantage when TrOPD uses its recommended hyperparameters under the same computational budget and evaluation protocol?

  2. Could the authors provide any available results comparing PS-OPD with Decoupled control under the same student generation budget to clarify whether the coupling design provides additional gains beyond auxiliary prefix supervision?

3 comments

Thank you for the careful review and noting that PS-OPD (1) is simple and effective, (2) reduces OPD's response-length explosion, and (3) reaches standard OPD's step-50 accuracy faster, even after counting the cost of the teacher traces. We ran both experiments you asked for in Q1 and Q2.

Q1. Does PS-OPD keep its advantage when TrOPD uses its recommended settings?

Yes. With TrOPD's released learning rate and weight decay, PS-OPD still yields higher accuracy than TrOPD in each of the three seeds: over steps 1–10, over steps 20–50, at step 50 and at equal training GPU-hours.

In our submission, we changed two settings of TrOPD (Trust Region On-Policy Distillation) to the values of our other methods. The learning rate went from 5e-6 to 1e-5, and the weight decay from 0.1 to 0.01. We kept TrOPD's 200-step prefix schedule, so our 50-step run covers only its first quarter (Appendix K.4).

For your question, we reran TrOPD with its released learning rate and weight decay. We used our training setup and evaluation, with seeds 1–3. We fixed the analysis before seeing any result. We ran two versions: one keeps TrOPD's 200-step schedule, and one fits the schedule to our 50 steps.

Accuracy (%) of PS-OPD and TrOPD. Avg. accuracy (the mean of AIME24, AIME25 and AIME26, avg@16), averaged over seeds 1–3. The steps 20–50 column is the mean of checkpoints 20, 30, 40 and 50. Differences in the text are computed before rounding.

MethodSteps 1–10Steps 20–50Step 50
TrOPD, released settings, 200-step schedule (new)21.0525.7626.09
TrOPD, released settings, 50-step schedule (new)21.3125.4025.75
TrOPD, our submission's settings (Table 6)22.8126.2326.28
PS-OPD (ours)23.2528.1128.95

All gaps between PS-OPD and the two new versions in this table are beyond seed variation (seed-paired test of Table 6). At step 50, PS-OPD is 2.86 and 3.19 points higher. PS-OPD is also higher at equal training GPU-hours. TrOPD's 50 training steps take about 30 GPU-hours, and PS-OPD uses the same GPU-hours by about step 34, where it scores about 28.0%. This is 1.89 and 2.21 points above the two new versions at step 50, and 1.67 points above our submitted TrOPD runs, all beyond seed variation.

The released settings do not help TrOPD here. Over steps 1–10, the two new versions score 1.75 and 1.50 points below our submitted TrOPD runs. After step 10, they are within seed variation of those runs.

With our submission's settings, PS-OPD is also above TrOPD on the AMD cluster over steps 20–50, with both teachers. The gap is 1.85 points with the Skywork teacher. It is 2.31 points with the JustRL teacher, whose runs we evaluate at temperature 0.6 and top-p 0.95.

You also mention the gap between 32.50% in the TrOPD paper and 26.0% in ours. The main difference is probably the training data. TrOPD trains on math prompts from OpenThoughts3. We train every OPD-type method on DAPO-Math-17k, including our TrOPD baseline. TrOPD's full set of teacher responses is not public, so our TrOPD baseline uses our own Skywork traces. The training schedule also differs. TrOPD runs 200 steps of 128 prompts × 4 rollouts, with an 8K-token length limit. We run 50 steps of 32 prompts × 8 rollouts, with a 27,648-token length limit. The evaluation differs too. TrOPD reports avg@32, and its paper does not state the temperature, top-p or length limit. We report avg@16 at temperature 1.0 and top-p 1.0, with a 31,744-token length limit. Even the untrained student scores differently on AIME25: 24.2% in the TrOPD paper and 21.9% in ours. So 32.50% and 26.0% are not directly comparable.

In the revision, we will keep our original TrOPD row and add the two new rows.

W2 / Q2. Does coupling add anything beyond prefix SFT, at the same length limit?

Yes, later in training: beyond prefix SFT, coupling adds a gain over steps 20–50, but no clear gain over steps 1–10.

PS-OPD couples two roles of the same teacher prefix: context for the student's rollout, and an SFT target. We separate the two roles with three baselines, each run with seeds 1–3:

  • RH is our baseline that uses the prefix only as context. It sets the SFT weight λ to 0, so it has no prefix SFT (Tables 6 and 20).
  • The decoupled baseline uses the prefix only as an SFT target (Table 25). The student learns the prefix with SFT, but its OPD rollouts start from the question. As you note, this changes both the context and the length limit. The student gets the full 27,648 tokens.
  • The length-matched decoupled baseline is new, for your Q2. It is the decoupled baseline with exactly PS-OPD's length limit for every question: 27,648 tokens minus that question's prefix. We also reran the original decoupled baseline with the same code. We fixed the analysis before seeing any result.

Accuracy (%) of PS-OPD and the baselines that separate the two roles of the prefix, measured as in the table in Q1.

MethodSteps 1–10Steps 20–50Step 50
RH (prefix as context only)21.4027.3827.14
Decoupled (prefix as SFT target only)22.8426.7527.35
Length-matched decoupled (new)23.0426.2026.50
PS-OPD (ours, prefix as both)23.2528.1128.95

Prefix SFT gives most of the early gain, as you point out in W2. Over steps 1–10, PS-OPD is 1.85 points above RH, which has no prefix SFT. It is only 0.42 and 0.21 points above the two decoupled baselines, which have prefix SFT, and these two gaps are within seed variation.

Coupling adds a later gain. Over steps 20–50, PS-OPD is 1.36 points above the decoupled baseline, beyond seed variation. On three new seeds (4–6), which we did not use to choose λ, the gap is 1.75 points, so this later gain is confirmed. PS-OPD also reaches OPD's step-50 accuracy at step 7, and the decoupled baseline only at step 40 (Table 9).

The length limit does not explain this later gain. Over steps 20–50, the length-matched baseline, which has exactly PS-OPD's length limit, is still 1.91 points below PS-OPD, beyond seed variation. It is level with the same-code rerun of the decoupled baseline (−0.21 points), and only 0.6% of its rollouts reach its limit. This test uses seeds 1–3, on which we chose λ = 0.3, so it is exploratory.

We will add this comparison to the main paper.

W1. How is PS-OPD different from TrOPD?

PS-OPD differs from TrOPD in two important ways. First, PS-OPD is much simpler. The TrOPD paper combines a changing prefix length, a small-weight forward-KL loss on the prefix, trust-region selection and an outlier loss. PS-OPD uses one fixed cut, SFT on the prefix and standard OPD on the continuation. So there are fewer hyperparameters to tune.

Second, the two designs prepare the student differently for test time, where neither method uses a teacher prefix. TrOPD gradually shrinks the teacher prefix to zero during training, like the mixing weight in DAgger (Ross et al., 2011), so its training ends on the student's own states. PS-OPD keeps the cut fixed and never removes the prefix. Instead, its SFT term trains the student to write the teacher's opening itself, which is what the student must do at test time.

However, the released TrOPD code (https://github.com/Xingrun-Xing2/TrOPD), which we used for our baseline, works differently from the paper. Its student rollouts start from the question only. It adds the teacher's trace for each question as a separate training sequence, so the student never continues a teacher prefix.

Notably, in our experiments, PS-OPD is both faster and more accurate. It reaches OPD's step-50 accuracy at step 7, while OPD needs 40 steps, so it needs about 40% fewer GPU-hours to get there, including the teacher traces (Table 9). At step 50, it is 2.12 points above OPD and 2.67 points above TrOPD (Table 6).

In the revision, we will add more details on how PS-OPD is novel compared with previous methods, including the comparison table in item 2 of the general response, which also covers ReOPD and Relay-OPD. We will also describe the TrOPD paper and its released code separately, in the Baselines paragraph of Section 4.1 and in Section 5 (Related Work). Appendix K.4 already does this.

Finally, as you note, the TrOPD paper by Xing et al. (2026) appeared on May 31 2026, so it can be considered concurrent, non-peer-reviewed work.

 

We hope these answers address the reviewer's concerns, and that the reviewer could reconsider the score. We are also happy to answer any further questions.

Reviewer_3YyxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper first discuss an obvservation that the accuracy of the student greatly increases when it continues from the beginning of the reasoning generated by the teachers, and vice-versa. So this paper propose an OPD variant, Prefix-Supervised On-Policy Distillation (PS-OPD). At each training step, the student continues a reasoning prefix generated offline by the teacher, and it is trained by supervised fine-tuning on the prefix and OPD on its own continuation. It shows clear improvements in empirical experiments.

Critical strengths
  1. It shows clear improvements in empirical experiments.
  2. This paper focus on an interesting and important problem.
  3. The method is simple but useful.
  4. The ablation study and related analyses provide useful insights into the effectiveness and robustness of the proposed approach, particularly its ability to mitigate the response-length explosion observed in standard OPD.
Critical weaknesses
  1. In Figure 2, the authors show that student accuracy improves as the length of the teacher prefix increases. This paper attribute this improvement to the importance of high-quality reasoning prefixes. They use this as the main motivation of the method. But a longer teacher prefix reduces the amount of reasoning the student needs to perform so it actually make the task easier. So I don't think the results fully support the conclusion.
  2. Also Figure2, even when the teacher model fails to recover from a flawed student prefix, its local token-level distributions may still provide informative learning signals. Therefore, the results in Figure 2 do not directly support that lower teacher continuation accuracy leads to less effective distillation.
  3. All main experiments are conducted on math using a single student model, DeepSeek-R1-Distill-Qwen-1.5B. I think more eval can be helpful
  4. I think an important baseline is missing which is Teacher Prefix + Student OPD.
  5. The core idea of using teacher-generated reasoning prefixes to guide student rollouts has been explored in several prior works. The methodological novelty relative to closely related approaches such as Relay-OPD or ReOPD remains somewhat unclear to me.
Questions for authors
  1. The full-trace SFT baseline looks pretty strong. Given the additional computation cost of PS-OPD, can you clarify whether its advantage over full-trace SFT remains significant under matched training token budgets, computational costs?
4 comments

Thank you for noting that PS-OPD is simple and useful, gives clear gains, and reduces OPD's response-length explosion. Below we answer your questions and concerns not addressed in the general response.

W1. Does a longer teacher prefix only make the task easier?

No. What matters is what the prefix says, not only how much work it leaves.

To test this, we extended the own-prefix test of Table 26 to three cuts. Each model continues its own prefix, cut at 25%, 50% or 75% of its reasoning. We compare this with the other model's prefix at the same cut. When the student continues, its own prefix does not help compared with no prefix: −0.6, −0.7 and −0.6 points, all within noise. The teacher's prefix helps a lot: +8.4, +16.7 and +26.2 points over the student's own prefix. The student also writes a longer continuation after a teacher prefix than after its own: +702, +896 and +609 tokens. The gain stays when we compare continuations of similar length: +16.6 points. It also stays when we remove prefixes that already contain the answer: +16.5 points at the 50% cut. When the teacher continues, a student prefix leaves it less work but lowers its accuracy. The teacher's own prefix does not lower it. We will add these tests to Section 3.1.

W2. Does lower teacher accuracy mean less effective distillation?

We agree that Figure 2 alone does not show this. Instead, it shows the importance of the initial part of the reasoning traces on the final accuracy. Consequently, this motivates us to use teacher prefixes (50% length) in our PS-OPD.

In particular, Figure 2 shows that a teacher prefix is a much better starting point for the student: with 75% of the teacher's reasoning, the student's accuracy rises from 19.6% to 45.3%. In the other direction, starting the teacher from 75% of the student's reasoning lowers its accuracy from 59.9% to 31.8%. This is why our method, PS-OPD, starts the student from the teacher's reasoning and also trains it to write that reasoning. Figure 2 checks only final answers. It does not measure the teacher's token-level feedback (Appendix L.1).

You suggest that the teacher's token-level feedback is still useful after the student has written a long, possibly wrong part. If so, removing that feedback should make training worse. Standard OPD gives the student the teacher's feedback on its whole answer. Our Forward-Horizon (FH) baseline keeps this feedback only on the first 4,096 tokens of each answer. The student still writes the full answer. Removing the feedback on later tokens does not hurt (Appendix K.3). Over steps 20–50, FH is even slightly above OPD in all three seeds, by 0.66 points on average. At step 50, the two are about the same. So in our setting, this late feedback adds no measurable learning. Xie et al. (2026, arXiv:2606.22600) find the same with other models.

W3. One student and math only: is the evaluation enough?

First, we highlight that the paper already rests on a very large set of experiments. In total, it reports:

  • more than 200 training runs, with more than 9,000 training steps;
  • more than 1,700 evaluated checkpoints;
  • about 2.5 million sampled answers, 16 per problem;
  • more than 8,000 GPU-hours.

These runs cover many settings:

  • Two kinds of teacher of different sizes: Skywork-OR1-Math-7B, a stronger math model, and JustRL-DeepSeek-1.5B, RL-finetuned from the student itself.
  • Four distillation losses: forward or reverse KL, over the teacher's or the student's top-128 tokens (Table 2).
  • Three benchmarks: AIME24, AIME25 and AIME26.
  • Four baselines: OPD, FH, TrOPD and off-policy distillation. FH gives teacher feedback only on the start of each answer. TrOPD mixes on- and off-policy distillation. Off-policy distillation is SFT on full teacher traces.
  • Three seeds for every main run. A test fixed in advance confirms that PS-OPD leads OPD over steps 20–50 on three new seeds (Appendix K.2).
  • Ablations: SFT weight λ of 0, 0.3, 1 and 3; cut ratio of 0.125, 0.5 and 0.75; a ten times smaller learning rate. The cut ratio is the share of teacher reasoning kept as prefix.
  • Three baselines separate the prefix's two roles: context and SFT target (Appendix D). RH uses the prefix only as context, with no SFT. The decoupled baseline keeps prefix SFT, but the student's answers start from the question alone. SFT→OPD does SFT first, then OPD.

For the rebuttal, we added a held-out test beyond AIME presented in Table A in the general response, which confirms the advantage of PS-OPD. We believe that all these experiments already strongly support our method in an important and challenging domain. We agree that one more student would be a useful, but not necessary, extra check, we will try to add it for the final version.

Moreover, we are adding a second student-teacher pair and different evaluation tasks with the same settings used in the paper. We will report it in the revision.

W4. The "teacher prefix + student OPD" baseline.

If we understand your comment correctly, this baseline is in the paper as RH, presented in Table 20 of Appendix D. In RH, the student runs OPD from the same teacher prefix as PS-OPD, with the same length limit. Thus, RH is equivalent to PS-OPD with λ=0\lambda=0, i.e. no SFT loss on the teacher prefix. PS-OPD achieves higher accuracy than RH in all three seeds, with average improvement of 1.85% over steps 1–10 and 0.73% over steps 20–50 (Table 6). We agree that this is a relevant baselines: we will move RH into Section 4 and add it to Figure 6 as the λ = 0 curve, next to λ = 0.3, 1 and 3, for completeness.

W5. Novelty compared with Relay-OPD and ReOPD.

Please see the answer to this in item 2 of the general response, where we compare in detail PS-OPD to Relay-OPD, ReOPD and TrOPD in one table.

Q1. Does PS-OPD's advantage over full-trace SFT remain at equal training tokens and equal compute?

Yes at equal training tokens, though one of the two gaps is within seed variation. No at equal compute, measured in GPU-hours. Full-trace SFT trains the student on complete teacher traces (in the paper we call it "off-policy distillation"). Importantly, our goal is not to beat SFT in every setting, but to make OPD, which learns from the student's own reasoning, both faster and more accurate. PS-OPD reaches OPD's step-50 accuracy at step 7, and it is 2.12 points above OPD at step 50 (Table 6).

  • Equal training tokens. A PS-OPD step trains on 2.47M tokens and an SFT step on 2.17M (Table 10). PS-OPD's 50 steps use 123.5M tokens. At that point, PS-OPD is 2.37 points above SFT, in all three seeds. SFT's 50 steps use 108.5M tokens. At that point, PS-OPD is 1.78 points higher, which is within seed variation. At equal steps, PS-OPD is 2.15 points higher at step 50, again in all three seeds (Table 6).
  • Equal GPU-hours. An SFT step is about 24 times cheaper than a PS-OPD step: 0.037 vs 0.886 GPU-hours (Table 10). So SFT is ahead early, by 4.2–9.3 points between 14.2 and 18.5 GPU-hours. For both methods, these GPU-hours include the cost of the teacher traces. Our SFT runs end at about 19.9 GPU-hours. There, the two are about equal: 28.3% vs 28.1%. So our 40% GPU-hour saving is relative to standard on-policy distillation (OPD), not to SFT.

Other work also finds that whether SFT or OPD works better depends on the teacher and the task. Zhao et al. (2026) find SFT better with one teacher and OPD better with another. Agarwal et al. (2024) find that on-policy distillation can beat supervised distillation on arithmetic reasoning, while Yang et al. (2026) find SFT ahead of OPD on a visual geometry task.

 

We hope these answers address the reviewer's concerns, and that the reviewer could reconsider the score. We are also happy to answer any further questions.

Thank you for the response. My main concern remains the comparison with full-trace SFT. The reported gains at matched training-token budgets is unclear to me. For my understanding, the computational cost per token of PS-OPD is much higher than SFT. And the matched GPU-hour comparison shows no clear improvement over full-trace SFT. It's even worse at the beginning. I do not think comparison with standard OPD alone is sufficient.

Also, I still think the novelty is a bit limited.

I will maintain my score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (1/2)

We thank all reviewers for their comments and feedback. We are glad that they appreciated that our method PS-OPD is simple and effective (3Yyx, EfMU), reaches OPD's step-50 accuracy much faster (at training step 7 instead of step 40 with the Skywork teacher, and with about 40% fewer GPU-hours, Table 9), even after counting the cost of the teacher traces (EfMU, mBTK), and mitigates OPD's response-length explosion (3Yyx, EfMU, mBTK). Below we answer the main questions common to multiple reviewers, and address the remaining concerns in the individual replies.

  1. Do the gains hold on new problems that were never used to choose λ? (mBTK Q2, 3Yyx W3)

Yes. To test how well the parameter choice λ=0.3\lambda=0.3 generalizes to unseen datasets, we build a new test set of 210 math problems (from six 2025–2026 contests, i.e., HMMT Feb 2025, Nov 2025 and Feb 2026; BRUMO, CMIMC and SMT 2025), which is disjoint from the datasets used in the paper. We checked every problem against the training set (DAPO-Math-17k) and the AIME test sets: none of them appears there, and we removed 6 problems that are similar to a training problem.

In the table below, we report the accuracy (avg@16) on this new test set achieved by the models fine-tuned with either OPD or PS-OPD and reported in the main evaluation in the paper (Tables 1 and 2 and Figure 3, forward KL over the teacher's top-128). To test convergence speed, we report average accuracy over training steps 1-10, 20-50, and final accuracy (step 50).

For both teachers (Skywork and JustRL), PS-OPD yields higher accuracy than OPD for all step ranges. Moreover, PS-OPD reaches OPD's step-50 accuracy at step 7 with Skywork teacher and at step 20 with JustRL teacher. In particular, the response-length explosion suffered by OPD with the JustRL teacher, observed in the paper, leads to the OPD student to hit the token limit in 55.5% of answers over steps 20-50 and thus lower accuracy. PS-OPD largely mitigates response-length explosion (4.2% token limit hits in the same setting). OPD's length explosion is also why the JustRL gap over steps 20-50 is so large in Table A: OPD's accuracy falls to 6.5% at step 20 and 5.9% at step 30, then recovers to 24.9% at step 50.

This shows that our configuration of PS-OPD, with λ=0.3\lambda=0.3 tuned on the AIME test sets, generalizes well to new problems, and even in this case significantly outperforms OPD.

Table A. Accuracy on the new test set. Values in % (avg@16), averaged over the six contests and seeds 1–3. For each teacher, the columns give the mean over training steps 1–10, the mean over steps 20–50 (checkpoints 20, 30, 40 and 50), and the accuracy at step 50. Δ\Delta is computed before rounding.

MethodSkywork 1–10Skywork 20–50Skywork 50JustRL 1–10JustRL 20–50JustRL 50
OPD15.919.319.718.614.724.9
PS-OPD (ours)17.521.121.221.027.828.0
Δ\Delta (PS-OPD - OPD)+1.66+1.85+1.49+2.39+13.04+3.05
AuthorsOpenReview ↗

General response (2/2)

  1. What is new compared with prior prefix-based methods (TrOPD, Relay-OPD, ReOPD)? (EfMU W1, mBTK W1/Q1, 3Yyx W5)

We would like to note that TrOPD, ReOPD and Relay-OPD are concurrent arXiv preprints (May–July 2026), not peer-reviewed papers. Under the ICLR 2027 Reviewer Guidelines, authors are not required to compare with papers that are only on arXiv, and a missing comparison "cannot be a basis for rejection" (https://iclr.cc/Conferences/2027/ReviewerGuidelines). In our manuscript, we directly compared the performance of PS-OPD to TrOPD, with PS-OPD achieving better results. In the table below we provide a detailed comparison of the four methods. For TrOPD, we found that there are a few discrepancies between the method described in the paper and the one implemented in the official code (github.com/Xingrun-Xing2/TrOPD), and therefore we discuss both.

PS-OPD (ours)TrOPDReOPDRelay-OPD
SettingSingle-turn long math reasoningSingle-turn reasoningMulti-turn tool-use agentsSingle-turn math
Teacher text (source)One offline trace per problemOne offline trace per problemOffline teacher trajectoriesShort segments generated live during the student's rollout
Prefix (teacher text from which the student starts)First 50% of the trace's reasoning, fixed at every stepPaper: first part of the teacher response, annealed to zero. Code: none (the student starts from the question)Teacher history up to a randomly sampled stepNone at the start; teacher segments are inserted when a trigger fires
Loss on teacher textSFT (cross entropy) on the prefix, λ = 0.3Paper: forward KL on the prefix, β = 0.001 (imitation). Code: the opposite of imitation: teacher responses get a PPO advantage 0.001⋅(logpS−logpT)0.001·(log p_S − log p_T), which widens the student–teacher gap, pushing the student away from the teacherNone (the prefix is context only)Same PPO reverse-KL loss as student tokens
Coupling (student continues the trained prefix)Coupled at every step; tested against decoupledPaper: coupled, until the prefix is annealed away. Code: decoupled (teacher responses are separate rows)No loss on the prefix, so no couplingCoupled, only when the trigger fires
On-policy lossForward / reverse KL (no PPO clipping)Reverse KL in trust region + forward KL on outliers; PPO clipping (code)Full-vocabulary reverse KL; step-decay samplingSampled-token reverse KL, PPO clipping, handoff trigger, speculative decoding

This direct comparison shows that our PS-OPD provides a much simpler method than the baselines to improve both efficiency and results of OPD. Unlike the baselines, PS-OPD needs one offline teacher trace per problem, one fixed cut that gives the prefix, and one cross-entropy term on the prefix. It needs no live teacher generation, no prefix schedule, no trust-region filter and no verifier. We are happy to add this discussion to the final version of the paper.

  1. Which use of the teacher prefix helps: as context, as an SFT target, or both? (EfMU W2/Q2, mBTK Q1, 3Yyx W4)

Both help, at different times. SFT on the prefix gives most of the early gain. Letting the student continue the prefix in its OPD rollouts adds a later gain.

In fact, RH (equivalent to PS-OPD with λ=0\lambda=0) is the baseline that uses the teacher prefix only as context, with no SFT on it. It is the "teacher prefix + student OPD" baseline that 3Yyx asked for (Table 20). PS-OPD is 1.85 points above RH over steps 1–10, and 0.73 points above it over steps 20–50.

Moreover, the decoupled baseline shown in Table 25 keeps SFT on the prefix, but starts OPD from the question. It keeps most of the early gain. But over steps 20–50, PS-OPD is 1.75 points higher on three new seeds that we did not use to choose λ.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.