- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes PS-OPD, a variant of OPD in which the student continues a teacher-written reasoning prefix. The method is motivated by a two-way experiment: a teacher prefix lifts the student while a student prefix drops the teacher. The evaluation on AIME24/25/26, PS-OPD reaches OPD's step-50 accuracy in 7 steps and uses about 40% fewer GPU-hours, including the cost of generating teacher traces.
Critical strengths
1.The prefix experiment directly motivates the design, and the gains in speed and GPU-hours hold even after counting the one-time cost of teacher traces. 2.PS-OPD curbs OPD's length explosion and is less sensitive to the choice of loss.
Critical weaknesses
1.Teacher text as rollout context (Relay-OPD[1], Prefix-GRPO[2], ReOPD[3]) and SFT combined with OPD (TrOPD[4]) both already exist. The paper offers no theory or analysis of why this particular combination is the right one, so the contribution is an empirical recipe rather than a new idea. 2.The method is heuristic and depends on tuning. Performance hinges on hand-set knobs: λ = 0.3, a cut ratio of r = 50%, an 8,192-token prefix cap, and dropping the ~15% of questions without a usable trace.
[1]Xu, H., Xu, X., Hong, H., Ni, Z., Li, H., Qiu, Y., ... & Shen, Y. (2026). Pass the baton: Trajectory-relayed on-policy distillation. arXiv preprint arXiv:2607.26057. [2]Wang, Yihan, et al. "From Trajectories to Prefixes: Reusing Teacher Trajectories via Replayed Prefixes and Online Continuation." arXiv preprint arXiv:2607.19395 (2026). [3]Liao, Baohao, et al. "Multi-Turn On-Policy Distillation with Prefix Replay." arXiv preprint arXiv:2607.04763 (2026). [4]Xing, Xingrun, et al. "Trust Region On-Policy Distillation." arXiv preprint arXiv:2606.01249 (2026).
Questions for authors
1.What distinguishes PS-OPD from prior prefix-based OPD? The RH (λ = 0) arm uses the teacher prefix only as rollout context, which looks essentially like Relay-OPD and ReOPD. If so, the only new ingredient is the added SFT term. 2.Does any of the advantage survive outside the 89 tuning problems? λ = 0.3 was selected on the same 89 AIME problems used for every reported number, including the confirmatory fresh-seed test. Do you have results where λ is chosen on a held-out set, or evaluated on benchmarks not used for selection?
3 comments
Q2. Is λ tuned on the test problems, and do the gains hold on unseen problems?
Yes, we chose λ = 0.3 on the 89 AIME problems but (1) λ = 0.3 achieves the best performance on all three individual AIME test sets (AIME24/25/26) and (2) the gains hold a new test set of 210 unseen problems, as we detail below. Therefore, this choice generalizes well across datasets and it is not the reason for which PS-OPD over the baselines.
Check 1: each AIME year alone. λ = 0.3 is the best weight on each AIME year separately. We compare at training step 50, the checkpoint we used to choose λ. Table 20 gives these values in the column "50" under each year, averaged over seeds 1–3. With λ = 0.3, accuracy is 35.7% on AIME24, 27.3% on AIME25 and 23.9% on AIME26. The best of λ = 0, 1 and 3 reaches 34.2%, 26.9% and 21.8%. So if we choose λ on any two years, we still pick 0.3, and it is also the best on the third year. Most of these per-year margins are within seed variation. If we choose λ on a random half of the 89 problems instead, we pick 0.3 in about 99% of the splits.
Check 2: a new held-out test. Table A in item 1 of our general response (and repeated below) gives the results on 210 new contest problems, which we never used to choose λ or any other setting. With λ = 0.3 unchanged, PS-OPD has higher accuracy than OPD for all step ranges (steps 1–10, steps 20–50 and step 50) with both teachers, and it reaches OPD's step-50 accuracy at step 7 (Skywork) and at step 20 (JustRL).
Table A. Accuracy on the new test set. Values in % (avg@16), averaged over the six contests and seeds 1–3. For each teacher, the columns give the mean over training steps 1–10, the mean over steps 20–50 (checkpoints 20, 30, 40 and 50), and the accuracy at step 50. is computed before rounding.
| Method | Skywork 1–10 | Skywork 20–50 | Skywork 50 | JustRL 1–10 | JustRL 20–50 | JustRL 50 |
|---|---|---|---|---|---|---|
| OPD | 15.9 | 19.3 | 19.7 | 18.6 | 14.7 | 24.9 |
| PS-OPD (ours) | 17.5 | 21.1 | 21.2 | 21.0 | 27.8 | 28.0 |
| (PS-OPD - OPD) | +1.66 | +1.85 | +1.49 | +2.39 | +13.04 | +3.05 |
W2. Does PS-OPD depend on hand-tuned settings, and are questions dropped?
PS-OPD's gains over OPD, in speed and in accuracy, do not depend on hand-tuned parameters (exact values of λ or the cut ratio). We did not tune the prefix cap, and no questions are dropped.
λ = 0.3. This is a stable choice (see Q2). At step 50 it is the best weight on each AIME year. We fixed a single value, chosen on one teacher, then we used the same value with the second teacher (JustRL), with all four distillation losses and on the held-out test. With it, PS-OPD's step-50 accuracy, averaged over the three AIME years, is above OPD's for both teachers and all four losses (Table 2).
The gains also do not need exactly λ = 0.3. With λ = 0.3, 1 and 0, the student reaches OPD's step-50 accuracy at step 7, 8 and 9. OPD needs 40 steps (Table 9). Only λ = 3 is slower, at step 30. Every weight we tried (λ = 0, 0.3, 1 and 3) also has higher accuracy than OPD over steps 1–10 and over steps 20–50 (Table 20).
Cut ratio r = 50%. We ran cut ratios 0.125, 0.5 and 0.75, each at λ = 0 and 1, with one seed each (Table 22). All six runs average 21.2–24.2% over steps 1–10, against 20.9% for OPD on that seed (Table 8).
8,192-token prefix cap. The cap only shortens the prefixes of the longest teacher traces. These are traces whose reasoning is longer than 16,384 tokens, twice the cap. On average, the cap lowers the prefix from 50% to 49.1% of the reasoning span (Appendix G). So it changes the prefixes very little, and we did not tune it, therefore it is not expected to affect the results.
Questions without a usable trace. None are dropped. They still train with plain OPD (Appendix A). We will clarify this in Section 3.2.
W1 / Q1. How is PS-OPD different from prefix-based methods, and is the SFT term the only new part?
Compared with methods that use teacher text only as context, yes, the SFT term is the new part. Compared with methods that also train on teacher text but start the student's rollout from the question alone, the new part is that the rollout continues the same prefix that SFT trains. Our baselines show that both parts help. Therefore, PS-OPD has significant differences to each of the existing baselines: it is simpler than existing methods and reaches OPD's step-50 accuracy much earlier, with about 40% fewer GPU-hours to get there (Table 9), and at step 50 it is 2.12 points above OPD (Table 6).
We also note that all three are arXiv preprints, not peer-reviewed papers: TrOPD (May 31 2026), ReOPD (July 6 2026) and Relay-OPD (28 July 2026, seven weeks before the deadline). We still discuss all of them, and we compare with TrOPD in experiments. Prefix-GRPO, which you also cite, has been withdrawn by its authors, and it does not anyway involve OPD-variants and thus it is not directly relevant to our work.
Item 2 of our general response compares PS-OPD with TrOPD, ReOPD and Relay-OPD in one table.
Prior methods.
- Relay-OPD (arXiv:2607.26057): the teacher takes over briefly inside the student's live rollout.
- ReOPD (arXiv:2607.04763): replays teacher histories in multi-turn tool use and trains selected student actions.
- TrOPD (arXiv:2606.01249) is the closest. In its paper (Sections 4.2–4.3, Eq. 8), the student continues a teacher prefix. The maximum prefix length starts at the maximum training sequence length and is cosine-annealed to zero, so training ends fully on-policy. The prefix gets a forward-KL loss with a small weight (0.001). Each student token is kept in a "trust region" with probability min(1, p_teacher / p_student) and gets the usual OPD loss. The other ("outlier") tokens get a separate top-k forward-KL loss. Its released code, which we used for our baseline, works differently. There, the student starts from the question only, and the teacher traces are trained as separate sequences, not as the start of the student's rollout.
PS-OPD. We cut one offline teacher trace at one fixed point: half of the reasoning, at most 8,192 tokens, the same at every step. The student learns this prefix with plain SFT, with weight λ = 0.3. It continues the prefix with standard OPD on all its own tokens. There is no live teacher generation, no prefix schedule and no token filter, so there are far fewer settings to tune.
Our baselines. RH, our baseline with λ = 0 (Table 20), is the context-only case. The teacher prefix is context, and the loss is on student tokens only, as in ReOPD. Unlike in Relay-OPD, the teacher does not write inside the rollout. Our decoupled baseline has the same structure as TrOPD's released code. It keeps prefix SFT but starts OPD from the question.
- Adding prefix SFT to RH gives +1.85 points over steps 1–10 and +0.73 over steps 20–50, in all three seeds (Table 6).
- The decoupled baseline gets most of the early gain. PS-OPD leads it by only 0.42 points over steps 1–10, within seed variation. But over steps 20–50, PS-OPD is 1.36 points higher (Table 25), and 1.75 points higher on three new seeds (4–6). Over steps 20–50, PS-OPD leads in every seed of both sets.
Overall, we believe that these clarifications and additional experiments clearly show the novelty and effectiveness of PS-OPD, as weIl as the fact that its advantage does not really on unfairly tuned parameters. Therefore, we ask the reviewer to reconsider their initial score. We are happy to answer any additional questions.
Thank you for the detailed review. In the following we address your concerns about novelty of our method and its dependence on particular, dataset-specific parameters.
W1 (theory). Why is this combination the right one?
While we do not have a complete theory of why our approach improves over the baselines, our method is motivated by several empirical observations and supported by the experimental results, where it outperforms the baselines. We note that also the baselines are similarly based on heuristics rather than rigorous theoretical proofs of optimality.
We agree that the single ingredients exist, but our main contribution is how to combine them in a simple way which leads to faster convergence and better accuracy. There is a simple reason to combine them: the student's gap to the teacher has two main parts, and PS-OPD puts one term on each.
(1) The experiment in Figure 2 motivates this combination. The start of the reasoning has a large effect on the final answer. A teacher prefix lifts the student: when the student continues the first 75% of the teacher's reasoning, its accuracy rises from 19.6% to 45.3%. A student prefix drops the teacher: when the teacher continues the first 75% of the student's reasoning, its accuracy falls from 59.9% to 31.8%. So the student should learn to write a good opening, not only to continue one. The prefix SFT term teaches this opening. So the two terms fit together: both move the student toward the teacher's next-token distribution, SFT on the teacher's opening and OPD on the student's continuation.
(2) Table 26 gives the numbers behind Figure 2 (frozen models, on AIME). It splits the gap into the two main parts: the opening and the continuation. Without a prefix, the teacher scores 59.9% and the student 19.6%, a gap of 40.3 points. At the 50% cut, 16.0 points come from the opening: the student gains 16.0 points when it continues the teacher's opening instead of writing its own. Another 23.3 points come from the continuation: when both continue a teacher opening, the student still scores 23.3 points below the teacher. The last 1.0 point is lost when the teacher continues its own trace, cut at 50%. The OPD term trains the continuation. The prefix SFT term trains the opening. On prefixed questions, it is the only loss on the opening. RH, our baseline that uses the prefix only as context, puts no loss on the opening. The opening part grows with the cut: 7.8, 16.0 and 25.7 points at 25%, 50% and 75%.
This is why PS-OPD has both terms. Our decoupled baseline also has both terms, but its OPD starts from the question alone. So whether OPD should continue the same prefix is a separate question. W1 / Q1 below answers it.