- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Student-Prefix Teacher Adaptation (SPTA), a teacher-side intervention for on-policy distillation (OPD). Standard OPD already queries a frozen teacher on student-generated prefixes, but does not explicitly train the teacher to handle those prefixes. SPTA adds a pre-distillation adaptation stage: the initial student generates trajectories, randomly truncated prefixes become teacher inputs, and the teacher is optimized with GRPO using final-answer correctness as the reward. The adapted teacher is then frozen and used for otherwise unchanged OPD.
Critical strengths
-
The idea is reasonable and well-motivated. Comparing original and adapted teachers on identical student prefixes is quite clear to demonstrate target issue.
-
The proposed SPTA prepares the teacher models to provide useful guidance to students without modifying the downstream distillation objectives, which also makes the method straightforward to understand and potentially compatible with other OPD improvements.
-
Experimental results show the effectiveness of the proposed method.
Critical weaknesses
-
The clean-prompt RLVR control tests whether prefix conditioning helps, but does not establish whether student-sourced prefixes are essential. Adaptation on teacher-generated or unrelated-model prefixes, matched approximately for length and difficulty, could also improve continuation robustness. Likewise, the decline in recovery with increasing student-prefix ratio does not independently identify a teacher-side distribution mismatch: prefix errors, context length, and remaining completion difficulty change together. The current evidence supports the effectiveness of the proposed training recipe more strongly than its student-specific explanation.
-
SPTA optimizes final-answer success after a prefix, whereas OPD consumes next-token feedback along student trajectories. These are related but distinct quantities. Successful teacher continuations can move onto states different from those the student subsequently visits, and improved outcome accuracy does not establish better local calibration or more useful sampled-token advantages. The manuscript acknowledges this distinction, but the explanation of why SPTA improves distillation remains indirect. A targeted analysis of the actual OPD feedback before and after adaptation—particularly at erroneous student states—would strengthen the proposed mechanism.
Questions for authors
-
Self-continuation baseline: Can you directly evaluate student recovery on the exact prefixes used in Table 2, and revise the constant-reference and “no information” claims accordingly?
-
Statistical robustness: Can you provide independent training repeats for the principal comparisons, especially the smaller gains over Tea-RL-OPD and in configuration (3)? Problem-level paired confidence intervals would also help characterize evaluation uncertainty.
-
Prefix-source specificity: Does adaptation on approximately length-matched teacher-generated prefixes provide similar downstream gains? This would help distinguish student-specific adaptation from general prefix-conditioned continuation training.
-
Mechanism and cost: Can you provide a focused analysis of changes in the actual sampled-token OPD feedback, together with measured adaptation and distillation costs? Please also clarify whether AIME25 validation influenced checkpoint selection or hyperparameter decisions, since AIME25 is included in the reported evaluation average.
2 comments
Response to Reviewer 8z1M (Part 2/2)
W2: SPTA optimizes final-answer success after a prefix, whereas OPD consumes next-token feedback along student trajectories. These are related but distinct quantities. A targeted analysis of the actual OPD feedback before and after adaptation—particularly at erroneous student states—would strengthen the proposed mechanism.
We agree: outcome accuracy after a prefix and token-level OPD feedback are distinct quantities. We ran a direct analysis of the feedback itself.
Setup. We fix one set of student rollouts (256 rollouts, 264,505 sampled tokens, from the first OPD step of configuration (1), so the sampling policy is exactly Qwen3-4B-Base) and score the same tokens with all five teachers. This removes any difference in the states being evaluated. For each sampled token, we compute the quantity OPD actually consumes, with .
Averages turn out to be the wrong statistic here, so we also condition on the states that matter. We call a token confidently wrong if inside a rollout that ends with a wrong answer (145k tokens); these are the states OPD must correct, and a teacher that agrees there provides no corrective signal. We report how often the teacher effectively endorses such a token (), how often it objects strongly ($ A_t < -2$), and what it proposes instead.
| Teacher | Prefix source | Mean | Mean | Endorse % @ conf.-wrong | Strong-obj. % @ conf.-wrong | Self-corr. top-1 % |
|---|---|---|---|---|---|---|
| Original | – | −1.53 | 0.53 | 86.4 | 2.16 | 2.9 |
| Tea-RL | – (clean prompt) | −1.49 | 0.48 | 81.2 | 2.33 | 3.7 |
| Tea-Self-Prefix | teacher itself | −1.49 | 0.48 | 82.3 | 2.27 | 2.9 |
| Tea-Other-Prefix | unrelated model | −1.49 | 0.49 | 83.0 | 2.08 | 3.5 |
| SPTA | target student | −1.81 | 0.79 | 75.3 | 4.45 | 12.9 |
The last column is measured on the strong-objection tokens of incorrect rollouts: it is the fraction whose teacher top-1 continuation is a self-correction marker (Wait, But, However, Hmm, 但是, ...).
Three findings.
- The original teacher is silent exactly where correction is needed. It endorses 86.4% of confidently wrong tokens, and its strong-objection rate is the same for rollouts that succeed and rollouts that fail (ratio 1.01). SPTA cuts endorsement to 75.3%, doubles the strong-objection rate, and makes it rollout-dependent (ratio 1.22).
- Where SPTA objects, it points somewhere useful: its top-1 continuation is a self-correction marker 4.4× more often than the original teacher's (12.9% vs. 2.9%).
- All three prefix-source controls are indistinguishable from each other and close to the original teacher on every one of these quantities. The token-level feedback therefore separates the methods in the same order as downstream accuracy, and it does so only for target-student prefixes. This also strengthens our answer to your W1.
Case study at an erroneous student state. A representative example: the same token scored by both teachers. After an arithmetic slip gives , the student writes ### Final Answer. At the token Final, the original teacher agrees (, ); the SPTA teacher assigns it () and puts 0.86 on Wait. The original teacher rubber-stamps the student's premature commitment to a wrong answer, which is precisely the failure OPD should penalize.
This makes the mechanism direct rather than inferred: SPTA changes the teacher's conditional at student states, and the OPD signal is read off exactly that conditional.
In the revised PDF: we add the analysis and results in §7.3 with Table 3.
Q4: Mechanism and cost
(a) OPD feedback analysis. Please see our reply to W2 above.
(b) Cost. For configuration (1), Stage I (teacher adaptation) runs on 64 GPUs and Stage II (student OPD) on 72 GPUs, identically for every method in the comparison.
(c) AIME25. Thank you for catching this. To be transparent: yes, AIME25 was used for periodic validation (every 10 steps) and for checkpoint selection, in all baselines. Hyperparameters were not tuned on it. Since the selection protocol is identical across methods, the comparison stays fair, but AIME25 should not be read as a clean held-out benchmark. The ranking does not depend on it. Excluding AIME25 and averaging the remaining four benchmarks:
| Pairing | OPD | Tea-RL-OPD | SPTA-OPD | SPTA − best baseline |
|---|---|---|---|---|
| (1) | 32.89 | 33.14 | 35.74 | +2.60 |
| (2) | 31.82 | 33.57 | 35.23 | +1.66 |
| (3) | 43.61 (Tea-RL) | – | 44.84 | +1.23 |
On the same single seed, the gains over Tea-RL-OPD are in fact larger without AIME25 (+1.66 and +1.23, vs. the +0.87 and +0.82 reported in the submission).
Thank you once again for the insightful comments that help us improve our paper.
Response to Reviewer 8z1M (Part 1/2)
Thanks for your positive review and constructive feedback. We address each point below.
Thank you for this precise point. We ran two new controls.
(a) Are student-sourced prefixes essential? We repeated the adaptation with prefixes from other sources, keeping everything else identical (same truncation procedure, same RL schedule). Configuration (1), teacher Qwen3-4B, target student Qwen3-4B-Base.
Gains over OPD: teacher's own prefixes +0.10, unrelated-model prefixes +1.17, target-student prefixes +3.78. Generic prefix-conditioned continuation training helps somewhat, but the target student's prefixes help more than 3× as much.
(b) Is the decline teacher-side? We measured the student's own continuation accuracy on the exact prefixes of Table 2 (see Q1). It stays nearly flat across ratios (12.49 → 12.41 and 29.58 → 28.89), while the teacher drops by 38.6 and 10.1 points on those same prefixes. Context length and remaining difficulty are identical in both cases. So these prefixes are not intrinsically harder to finish; they are harder for the teacher. This is the asymmetry our claim rests on. Due to the time limitation, we do not isolate every factor (e.g., prefix errors vs. style), and we leave it for further work.
Yes. Following your suggestion, we measure the student's own continuation accuracy on the exact prefixes and revise Table 2.
Yes. We repeated all main comparisons with 3 independent training seeds under the same evaluation protocol. Summary of the 5-benchmark average (mean ± std). The results show that SPTA outperforms the task-level RL teacher in all three pairings, which demonstrates the superiority of our method.
In (3), per-benchmark results: (Results for the other two model pairs are presented in Table 7 of the paper.)
JustRL-DeepSeek-1.5B → DeepSeek-R1-Distill-Qwen-1.5BNo. Teacher-generated prefixes give a much smaller gain. This is the table from W1(a) (configuration (1)):
Prefixes from the teacher itself are the weakest control, which is what the mismatch predicts: those states are already in the teacher's own distribution, so there is nothing to adapt to. Prefixes from an unrelated model help more (+1.17), since they are at least off-distribution. In summary, general prefix-conditioned continuation training does not substitute for student-specific adaptation.