acceptodds
Under review as a conference paper at ICLR 2027

Meet Students Where They Are: Student-Prefix Teacher Adaptation for Reliable On-Policy Distillation

13 views

Abstract

On-policy distillation (OPD) mitigates the student's exposure bias by optimizing on student-generated trajectories. However, student-generated prefixes may contain reasoning errors or follow patterns unfamiliar to the teacher, undermining the reliability of its token-level supervision. This degradation becomes more pronounced as prefixes grow longer. To address this teacher-side distribution mismatch, we propose **Student-Prefix Teacher Adaptation** (SPTA), which adapts the teacher to the states on which it is queried during distillation. Specifically, before OPD starts, we collect student rollout prefixes with random lengths and continually train the teacher to continue from the prefixes using reinforcement learning with verifiable rewards (RLVR). By treating each prefix as context and rewarding correct final answers, the teacher learns to recover from student-induced states rather than imitate the student's reasoning. Then, the adapted teacher is used for standard OPD, with no changes to the distillation objective. Experiments on mathematical reasoning benchmarks show that our SPTA improves average student performance over standard OPD and OPD with task-level RLVR-adapted teachers. These results highlight the importance of adapting teachers not only to solve problems independently, but also to provide reliable guidance on the states their students actually visit.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Distribution Mismatch, Student-Prefix Teacher Adaptation, Reliable Supervision

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_8z1MOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Student-Prefix Teacher Adaptation (SPTA), a teacher-side intervention for on-policy distillation (OPD). Standard OPD already queries a frozen teacher on student-generated prefixes, but does not explicitly train the teacher to handle those prefixes. SPTA adds a pre-distillation adaptation stage: the initial student generates trajectories, randomly truncated prefixes become teacher inputs, and the teacher is optimized with GRPO using final-answer correctness as the reward. The adapted teacher is then frozen and used for otherwise unchanged OPD.

Critical strengths
  1. The idea is reasonable and well-motivated. Comparing original and adapted teachers on identical student prefixes is quite clear to demonstrate target issue.

  2. The proposed SPTA prepares the teacher models to provide useful guidance to students without modifying the downstream distillation objectives, which also makes the method straightforward to understand and potentially compatible with other OPD improvements.

  3. Experimental results show the effectiveness of the proposed method.

Critical weaknesses
  1. The clean-prompt RLVR control tests whether prefix conditioning helps, but does not establish whether student-sourced prefixes are essential. Adaptation on teacher-generated or unrelated-model prefixes, matched approximately for length and difficulty, could also improve continuation robustness. Likewise, the decline in recovery with increasing student-prefix ratio does not independently identify a teacher-side distribution mismatch: prefix errors, context length, and remaining completion difficulty change together. The current evidence supports the effectiveness of the proposed training recipe more strongly than its student-specific explanation.

  2. SPTA optimizes final-answer success after a prefix, whereas OPD consumes next-token feedback along student trajectories. These are related but distinct quantities. Successful teacher continuations can move onto states different from those the student subsequently visits, and improved outcome accuracy does not establish better local calibration or more useful sampled-token advantages. The manuscript acknowledges this distinction, but the explanation of why SPTA improves distillation remains indirect. A targeted analysis of the actual OPD feedback before and after adaptation—particularly at erroneous student states—would strengthen the proposed mechanism.

Questions for authors
  1. Self-continuation baseline: Can you directly evaluate student recovery on the exact prefixes used in Table 2, and revise the constant-reference and “no information” claims accordingly?

  2. Statistical robustness: Can you provide independent training repeats for the principal comparisons, especially the smaller gains over Tea-RL-OPD and in configuration (3)? Problem-level paired confidence intervals would also help characterize evaluation uncertainty.

  3. Prefix-source specificity: Does adaptation on approximately length-matched teacher-generated prefixes provide similar downstream gains? This would help distinguish student-specific adaptation from general prefix-conditioned continuation training.

  4. Mechanism and cost: Can you provide a focused analysis of changes in the actual sampled-token OPD feedback, together with measured adaptation and distillation costs? Please also clarify whether AIME25 validation influenced checkpoint selection or hyperparameter decisions, since AIME25 is included in the reported evaluation average.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 8z1M (Part 1/2)

Thanks for your positive review and constructive feedback. We address each point below.


W1: The clean-prompt RLVR control tests whether prefix conditioning helps, but does not establish whether student-sourced prefixes are essential.

Thank you for this precise point. We ran two new controls.

(a) Are student-sourced prefixes essential? We repeated the adaptation with prefixes from other sources, keeping everything else identical (same truncation procedure, same RL schedule). Configuration (1), teacher Qwen3-4B, target student Qwen3-4B-Base.

MethodPrefix sourceAIME24AIME25AMC23HMMT24HMMT25Avg.
OPD–33.5426.0465.5116.0416.4631.52
Tea-RL-OPD– (clean prompt)33.1227.2965.8916.6716.8831.97
Tea-Self-Prefix-RL-OPDteacher itself33.7528.3366.8714.7914.3731.62
Tea-Other-Prefix-RL-OPDDeepSeek-R1-Distill-1.5B31.6730.6368.6716.0416.4632.69
SPTA-OPDtarget student37.7133.5471.9116.0417.2935.30

Gains over OPD: teacher's own prefixes +0.10, unrelated-model prefixes +1.17, target-student prefixes +3.78. Generic prefix-conditioned continuation training helps somewhat, but the target student's prefixes help more than 3× as much.

(b) Is the decline teacher-side? We measured the student's own continuation accuracy on the exact prefixes of Table 2 (see Q1). It stays nearly flat across ratios (12.49 → 12.41 and 29.58 → 28.89), while the teacher drops by 38.6 and 10.1 points on those same prefixes. Context length and remaining difficulty are identical in both cases. So these prefixes are not intrinsically harder to finish; they are harder for the teacher. This is the asymmetry our claim rests on. Due to the time limitation, we do not isolate every factor (e.g., prefix errors vs. style), and we leave it for further work.


Q1: Self-continuation baseline: Can you directly evaluate student recovery on the exact prefixes used in Table 2, and revise the constant-reference and “no information” claims accordingly?

Yes. Following your suggestion, we measure the student's own continuation accuracy on the exact prefixes and revise Table 2.


Q2: Statistical robustness: Can you provide independent training repeats for the principal comparisons, especially the smaller gains over Tea-RL-OPD and in configuration (3)?

Yes. We repeated all main comparisons with 3 independent training seeds under the same evaluation protocol. Summary of the 5-benchmark average (mean ± std). The results show that SPTA outperforms the task-level RL teacher in all three pairings, which demonstrates the superiority of our method.

PairingOPDTea-RL-OPDSPTA-OPDSPTA − Tea-RL-OPD
(1) Qwen3-4B → Qwen3-4B-Base31.24 (±0.27)31.91 (±0.48)35.31 (±0.30)+3.40
(2) R1-Distill-7B → R1-Distill-1.5B30.08 (±0.14)32.46 (±0.08)33.31 (±0.08)+0.85
(3) JustRL-1.5B → R1-Distill-1.5B–41.86 (±0.14)42.82 (±0.33)+0.96

In (3), per-benchmark results: (Results for the other two model pairs are presented in Table 7 of the paper.)

  • JustRL-DeepSeek-1.5B → DeepSeek-R1-Distill-Qwen-1.5B
MethodAIME24AIME25AMC23HMMT24HMMT25Avg.
OPD (Tea-RL-OPD)49.93 (±1.74)34.10 (±1.48)81.20 (±3.70)22.57 (±3.31)21.53 (±0.67)41.86 (±0.14)
SPTA-OPD51.74 (±0.73)35.21 (±0.91)80.22 (±0.42)24.58 (±0.72)22.36 (±0.96)42.82 (±0.33)

Q3: Prefix-source specificity: Does adaptation on approximately length-matched teacher-generated prefixes provide similar downstream gains? This would help distinguish student-specific adaptation from general prefix-conditioned continuation training.

No. Teacher-generated prefixes give a much smaller gain. This is the table from W1(a) (configuration (1)):

MethodPrefix sourceAvg.Gain over OPD
OPD–31.52–
Tea-RL-OPD– (clean prompt)31.97+0.45
Tea-Self-Prefix-RL-OPDteacher itself31.62+0.10
Tea-Other-Prefix-RL-OPDDeepSeek-R1-Distill-1.5B32.69+1.17
SPTA-OPDtarget student35.30+3.78

Prefixes from the teacher itself are the weakest control, which is what the mismatch predicts: those states are already in the teacher's own distribution, so there is nothing to adapt to. Prefixes from an unrelated model help more (+1.17), since they are at least off-distribution. In summary, general prefix-conditioned continuation training does not substitute for student-specific adaptation.

AuthorsOpenReview ↗

Response to Reviewer 8z1M (Part 2/2)

W2: SPTA optimizes final-answer success after a prefix, whereas OPD consumes next-token feedback along student trajectories. These are related but distinct quantities. A targeted analysis of the actual OPD feedback before and after adaptation—particularly at erroneous student states—would strengthen the proposed mechanism.

We agree: outcome accuracy after a prefix and token-level OPD feedback are distinct quantities. We ran a direct analysis of the feedback itself.

Setup. We fix one set of student rollouts (256 rollouts, 264,505 sampled tokens, from the first OPD step of configuration (1), so the sampling policy is exactly Qwen3-4B-Base) and score the same tokens with all five teachers. This removes any difference in the states being evaluated. For each sampled token, we compute the quantity OPD actually consumes, dt=log⁡πS(yt)−log⁡πT(yt)d_t = \log\pi_S(y_t) - \log\pi_T(y_t) with At=−clip(dt,−10,10)A_t = -\mathrm{clip}(d_t, -10, 10).

Averages turn out to be the wrong statistic here, so we also condition on the states that matter. We call a token confidently wrong if πS(yt)>0.9\pi_S(y_t) > 0.9 inside a rollout that ends with a wrong answer (145k tokens); these are the states OPD must correct, and a teacher that agrees there provides no corrective signal. We report how often the teacher effectively endorses such a token (∣At∣<0.05\lvert A_t \rvert < 0.05), how often it objects strongly ($ A_t < -2$), and what it proposes instead.

TeacherPrefix sourceMean log⁡πT\log\pi_TMean ∣At∣\lvert A_t \rvertEndorse % @ conf.-wrongStrong-obj. % @ conf.-wrongSelf-corr. top-1 %
Original–−1.530.5386.42.162.9
Tea-RL– (clean prompt)−1.490.4881.22.333.7
Tea-Self-Prefixteacher itself−1.490.4882.32.272.9
Tea-Other-Prefixunrelated model−1.490.4983.02.083.5
SPTAtarget student−1.810.7975.34.4512.9

The last column is measured on the strong-objection tokens of incorrect rollouts: it is the fraction whose teacher top-1 continuation is a self-correction marker (Wait, But, However, Hmm, 但是, ...).

Three findings.

  1. The original teacher is silent exactly where correction is needed. It endorses 86.4% of confidently wrong tokens, and its strong-objection rate is the same for rollouts that succeed and rollouts that fail (ratio 1.01). SPTA cuts endorsement to 75.3%, doubles the strong-objection rate, and makes it rollout-dependent (ratio 1.22).
  2. Where SPTA objects, it points somewhere useful: its top-1 continuation is a self-correction marker 4.4× more often than the original teacher's (12.9% vs. 2.9%).
  3. All three prefix-source controls are indistinguishable from each other and close to the original teacher on every one of these quantities. The token-level feedback therefore separates the methods in the same order as downstream accuracy, and it does so only for target-student prefixes. This also strengthens our answer to your W1.

Case study at an erroneous student state. A representative example: the same token scored by both teachers. After an arithmetic slip gives 294+384=678294+384=678, the student writes ### Final Answer. At the token Final, the original teacher agrees (πT=0.80\pi_T = 0.80, At=+0.02A_t = +0.02); the SPTA teacher assigns it e−11.65e^{-11.65} (At=−10.00A_t = -10.00) and puts 0.86 on Wait. The original teacher rubber-stamps the student's premature commitment to a wrong answer, which is precisely the failure OPD should penalize.

This makes the mechanism direct rather than inferred: SPTA changes the teacher's conditional at student states, and the OPD signal is read off exactly that conditional.

In the revised PDF: we add the analysis and results in §7.3 with Table 3.


Q4: Mechanism and cost

(a) OPD feedback analysis. Please see our reply to W2 above.

(b) Cost. For configuration (1), Stage I (teacher adaptation) runs on 64 GPUs and Stage II (student OPD) on 72 GPUs, identically for every method in the comparison.

(c) AIME25. Thank you for catching this. To be transparent: yes, AIME25 was used for periodic validation (every 10 steps) and for checkpoint selection, in all baselines. Hyperparameters were not tuned on it. Since the selection protocol is identical across methods, the comparison stays fair, but AIME25 should not be read as a clean held-out benchmark. The ranking does not depend on it. Excluding AIME25 and averaging the remaining four benchmarks:

PairingOPDTea-RL-OPDSPTA-OPDSPTA − best baseline
(1)32.8933.1435.74+2.60
(2)31.8233.5735.23+1.66
(3)43.61 (Tea-RL)–44.84+1.23

On the same single seed, the gains over Tea-RL-OPD are in fact larger without AIME25 (+1.66 and +1.23, vs. the +0.87 and +0.82 reported in the submission).


Thank you once again for the insightful comments that help us improve our paper.

Reviewer_AUd3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SPTA adapts a teacher using reinforcement learning on student-generated prefixes, then freezes it for on-policy distillation. The teacher is first adapted using reinforcement learning with verifiable rewards to continue student-generated prefixes, then frozen for standard student distillation. Experiments on five math benchmarks show average improvements across three teacher–student pairings.

Critical strengths
  • A simple, well-motivated method that preserves the downstream distillation procedure.
  • Positive average gains across three pairings, including comparisons against clean-prompt teacher adaptation.
Critical weaknesses
  • The contribution appears incremental. Improvements over stronger controls are sometimes small and uneven across benchmarks: configuration (2) gains only 0.87 points over task-level teacher RL and underperforms it on three benchmarks.
  • No repeated training runs or confidence intervals establish whether the smaller gains are reliable. Missing alternative-prefix controls also leave the importance of using the target student’s prefixes unclear.
Questions for authors
  • Does adaptation using target-student prefixes outperform adaptation using comparable-length teacher or other-model prefixes?
1 comment
AuthorsOpenReview ↗

Response to Reviewer AUd3

Thank you for the careful review and constructive feedback. We address each point below.


W1: The contribution appears incremental; gains over stronger controls are small and uneven (e.g., +0.87 in configuration (2)).

The gains are consistent across seeds. We repeated all main comparisons with 3 independent training seeds (5-benchmark average, mean ± std). The results show that SPTA outperforms the task-level RL teacher in all three pairings, which demonstrates the superiority of our method. (Detailed results are presented in Table 7 of the revised paper.)

PairingOPDTea-RL-OPDSPTA-OPDSPTA − Tea-RL-OPD
(1) Qwen3-4B → Qwen3-4B-Base31.24 (±0.27)31.91 (±0.48)35.31 (±0.30)+3.40
(2) R1-Distill-7B → R1-Distill-1.5B30.08 (±0.14)32.46 (±0.08)33.31 (±0.08)+0.85
(3) JustRL-1.5B → R1-Distill-1.5B–41.86 (±0.14)42.82 (±0.33)+0.96

Per-benchmark variation. These are small competition sets, so a single problem can move a benchmark score by several points and single-benchmark differences are noisy. This is why we use the 5-benchmark average as the main metric. Per-benchmark results for configuration (2) (3 seeds):

MethodAIME24AIME25AMC23HMMT24HMMT25Avg.
OPD31.11 (±0.12)23.47 (±0.48)64.21 (±0.61)15.90 (±1.73)15.70 (±1.68)30.08 (±0.14)
Tea-RL-OPD32.64 (±0.43)27.29 (±0.21)69.23 (±0.49)17.29 (±0.75)15.83 (±0.21)32.46 (±0.08)
SPTA-OPD38.40 (±0.48)26.25 (±0.95)68.15 (±0.20)16.46 (±0.21)17.29 (±0.42)33.31 (±0.08)

Contribution. SPTA is not just "more teacher RL". We evaluated the Tea-RL teacher on the same student prefixes (config (1)). Tea-RL uses the same RL algorithm, hyperparameters, update count, and context budget as SPTA. Task-level RL lowers recovery on student prefixes at every ratio, while SPTA raises it by 17.7–25.3 points. The key factor is where the teacher is optimized, not how much.

Teacherr=0r=25%r=50%r=75%r=90%Student Avg.
Original60.9932.7628.9124.8222.3631.52
Tea-RL52.0428.8125.2420.9818.6531.97
SPTA54.3950.4649.8848.5647.6835.30

W2: No repeated runs; missing alternative-prefix controls.

Repeated runs are reported above. For alternative prefixes, see Q1.


Q1: Does adaptation on target-student prefixes outperform adaptation on comparable-length teacher or other-model prefixes?

Yes. We added two prefix-source controls in configuration (1) (teacher Qwen3-4B, target student Qwen3-4B-Base). Everything is identical to SPTA except where the prefixes come from, including the random truncation procedure and the RL schedule:

  • Tea-Self-Prefix: prefixes from the teacher's own rollouts.
  • Tea-Other-Prefix: prefixes from an unrelated model, DeepSeek-R1-Distill-Qwen-1.5B.
MethodPrefix sourceAIME24AIME25AMC23HMMT24HMMT25Avg.
OPD–33.5426.0465.5116.0416.4631.52
Tea-RL-OPD– (clean prompt)33.1227.2965.8916.6716.8831.97
Tea-Self-Prefix-RL-OPDteacher itself33.7528.3366.8714.7914.3731.62
Tea-Other-Prefix-RL-OPDunrelated model31.6730.6368.6716.0416.4632.69
SPTA-OPDtarget student37.7133.5471.9116.0417.2935.30

Gains over OPD: teacher's own prefixes +0.10, unrelated-model prefixes +1.17, target-student prefixes +3.78.

Prefix-conditioned training in general gives only a small gain. Using the target student's prefixes gives more than 3× that gain. The teacher's own prefixes help least, as expected: they are already in its training distribution, so there is no mismatch to fix. This answers the question directly: the benefit comes from adapting the teacher to the states its own student visits, not from prefix conditioning as such.


Thank you once again for the insightful comments that help us improve our paper. We sincerely hope you can reconsider the contributions of our work.

Reviewer_6ssSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies teacher-side distribution mismatch in on-policy distillation (OPD) for mathematical reasoning models: although sampling trajectories from the student eliminates student-side exposure bias, student-generated prefixes may contain reasoning errors or stylistic patterns unfamiliar to the teacher, degrading the teacher's token-level supervision πT(⋅∣x,y<tS)\pi_T(\cdot \mid x, y^S_{<t}).

Critical strengths
  1. Clean, orthogonal teacher-side perspective on OPD reliability. Most recent OPD refinements treat unreliable teacher feedback on drifted prefixes as fixed and filter/reweight tokens on the student side. Adapting the teacher via outcome-rewarded RL on student prefixes is conceptually clean: using verifiable final-answer rewards encourages error recovery rather than imitating flawed student prefixes, while keeping reference answers out of the conditioning context avoids the train–test mismatch of privileged self-distillation.
  2. Insightful empirical decoupling between standalone capability and prefix recovery. Figure 2 and Table 2 clearly demonstrate that strong clean-prompt (r=0%) accuracy does not imply reliable continuation from student prefixes. Particularly striking is Pairing (1) (Qwen3-4B -> Qwen3-4B-Base), where SPTA reduces the teacher's clean-prompt accuracy by 6.60 points (60.99% -> 54.39%) while boosting recovery at r=90% by +25.32 points (22.36% -> 47.68%), ultimately producing a +3.78-point stronger student. Benchmarking teacher recovery against the student's flat self-continuation break-even accuracy (Section 7.2) also provides an intuitive diagnostic for why standard OPD stalls in Pairing (2) (+0.62 points over S0S_0).
  3. High clarity and scientific candor. The manuscript is well-organized and refreshingly precise about what its experiments do and do not establish (e.g., clearly documenting the exact clamped sampled-token k1 implementation in Appendix A and explicitly discussing several scope boundaries in Section 9).
Critical weaknesses
  1. Unjustified compute/supervision trade-off (missing Student-RLVR and multi-epoch OPD baselines): Unlike standard OPD, SPTA requires ground-truth verifiers on DAPO-Math-17k and spends >10x more rollouts adapting the larger teacher sequences up to 16,384 tokens) than distilling the student (1 epoch x 4 rollouts = 4∣D∣4|\mathcal{D}| sequences up to 8,192 tokens. Given ground-truth rewards and this compute budget, the paper omits (a) spending comparable RLVR compute directly on the student (notably, in Pairing 3, JustRL-1.5B—trained via direct GRPO from the same 1.5B student—yields 41.97% even before SPTA, vs. 33.23% in Pairing 2), and (b) running standard OPD / Tea-RL-OPD for more than 1 epoch.
  2. Marginal, noisy gains over Tea-RL-OPD in format-aligned pairings vs. format confounding in Pairing (1): Where teacher and student already share the R1 <think> format, SPTA beats Tea-RL-OPD by only +0.87 (Pairing 2, losing on 3/5 benchmarks) and +0.82 points (Pairing 3, -5.27 on AMC23) over 160 total problems with a single seed. The only large margin over Tea-RL-OPD (+3.33, Pairing 1) pairs <think>-post-trained Qwen3-4B with raw Qwen3-4B-Base, where original teacher accuracy plunges -28.23 points by r=25%—pointing to chat-template/<think> mismatch, yet template handling is unspecified and no lightweight student SFT warmup baseline is tested.
  3. Incomplete mechanistic controls: Tea-RL is omitted from the teacher recovery evaluation (Figure 2 / Table 2) despite checkpoints already existing. Moreover, Stage I caps generation at 9,216 tokens for both methods, giving SPTA up to 7,168 + 9,216 = 16,384 total reasoning tokens vs. 9,216 for Tea-RL, and no teacher-own-prefix RLVR control (y^T_{1:k}) is included to separate student-prefix adaptation from generic mid-trajectory continuation.
Questions for authors

NA

1 comment
AuthorsOpenReview ↗

Response to Reviewer 6ssS

Thank you for the careful review and constructive feedback. We address each point below.

W1: Compute/supervision trade-off (missing Student-RLVR and multi-epoch OPD baselines).

  • SPTA and Tea-RL-OPD have the same compute. They use the same RL algorithm, hyperparameters, update count, and a 16,384-token total budget. Only the start state differs. This is the controlled comparison that isolates prefix conditioning. SPTA actually generates fewer tokens per sample (≤ 9,216 vs. ≤ 15,360 for Tea-RL), because the prefix is given, not generated.
  • Config (3) already covers Student-RLVR. JustRL-1.5B is obtained by GRPO directly on the same 1.5B student, using a full student-side RLVR run that is far longer than one SPTA round. To clarify, 41.86% is the score of the student distilled from JustRL, not of JustRL itself. Adapting this strong RL model with SPTA still adds +0.96 (3 seeds). So SPTA is complementary to student RLVR, not a replacement for it.
  • Teacher training is already a standard step. In real pipelines, the teacher is usually RL-trained before distillation anyway. SPTA does not add a new stage. It changes how this existing stage is run, which helps the trained teacher become a better distillation teacher.

W2: Small, noisy gains in format-aligned pairings vs. format confounding in Pairing (1).

Noise. We repeated all main comparisons with 3 independent training seeds (5-benchmark average, mean ± std). The results demonstrate the advantages of our method.

PairingOPDTea-RL-OPDSPTA-OPDSPTA − Tea-RL-OPD
(1) Qwen3-4B → Qwen3-4B-Base31.24 (±0.27)31.91 (±0.48)35.31 (±0.30)+3.40
(2) R1-Distill-7B → R1-Distill-1.5B30.08 (±0.14)32.46 (±0.08)33.31 (±0.08)+0.85
(3) JustRL-1.5B → R1-Distill-1.5B–41.86 (±0.14)42.82 (±0.33)+0.96

Format confounding. We added a format-aligned variant of (1). We first SFT Qwen3-4B-Base on teacher-generated responses and then use it as the student for all methods. After format alignment, SPTA still beats Tea-RL-OPD by +2.34 and OPD by +3.32, and it is best on all five benchmarks. So the large gain in (1) is not just a <think>-format fix.

MethodAIME24AIME25AMC23HMMT24HMMT25Avg.
Student (SFT)19.5818.7548.1210.0010.0021.29
OPD32.9229.3861.9718.1218.5432.19
Tea-RL-OPD32.7130.8369.2016.4616.6733.17
SPTA-OPD37.0832.5070.0418.5419.3835.51

W3: Incomplete mechanistic controls.

(a) Tea-RL in Table 2. We added the Tea-RL teacher on the same student prefixes (config (1)). Task-level RL lowers recovery at every ratio (−3.7 to −4.0 points at r > 0). SPTA raises it by +17.7 to +25.3 points.

Teacherr=0r=25%r=50%r=75%r=90%Student Avg.
Original60.9932.7628.9124.8222.3631.52
Tea-RL52.0428.8125.2420.9818.6531.97
SPTA54.3950.4649.8848.5647.6835.30

(b) Token budget. This is a reporting gap in our appendix, and we apologize for it. Tea-RL is not capped at 9,216 tokens. It uses 1,024 input + 15,360 generation tokens = 16,384, the same total as SPTA (7,168 + 9,216). Tea-RL therefore has the larger generation budget. This is now corrected in Table 5 (Appendix A) of the revised PDF, both in the row entries and in an explicit note in the caption.

(c) Teacher-own-prefix control. We ran RLVR on prefixes from the teacher's own rollouts, using the same truncation procedure (Tea-Self-Prefix-RL-OPD). The results show that own-prefix adaptation gives almost no gain (+0.10 over OPD). Generic mid-trajectory continuation therefore does not explain SPTA. The gain requires student prefixes.

MethodAIME24AIME25AMC23HMMT24HMMT25Avg.
OPD33.5426.0465.5116.0416.4631.52
Tea-Self-Prefix-RL-OPD33.7528.3366.8714.7914.3731.62
Tea-RL-OPD33.1227.2965.8916.6716.8831.97
SPTA-OPD37.7133.5471.9116.0417.2935.30
Reviewer_cnn7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Student-Prefix Teacher Adaptation (SPTA), which trains a teacher with verifiable rewards to complete randomly truncated student-generated prefixes, then freezes it for standard on-policy distillation. Experiments on five mathematical reasoning benchmarks across three teacher–student pairings report improved teacher recovery and downstream student accuracy, including gains over task-level teacher RL in the two configurations with that control.

Critical strengths
  1. The paper addresses a relevant issue in on-policy distillation: a teacher with strong standalone performance may still struggle to provide useful guidance on student-generated prefixes. The recovery analysis offers a useful perspective on evaluating teachers in the contexts where their supervision is actually needed.

  2. SPTA is a simple and well-motivated approach to this problem. It adapts the teacher through prefix-conditioned RLVR without requiring error annotations or modifying the OPD objective. The reported improvements across three teacher–student pairings, together with the task-level teacher-RL controls, provide encouraging evidence for this approach.

Critical weaknesses

The additional contribution relative to existing teacher-adaptation work could be better established. Prior work has already discussed student-prefix-induced teacher mismatch and the benefits of RLVR-adapted teachers[1]. SPTA makes a useful distinction by conditioning teacher training on student prefixes, although the methodological extension appears relatively focused. Further evidence explaining when this adaptation is particularly beneficial would help establish its significance beyond the existing teacher-RL approach.

The reliability and conditions of the gains remain unclear. The most convincing reported advantage is the 3.33-point improvement over Tea-RL-OPD in the Qwen3-4B pairing. The improvements over the corresponding reported baselines in the other two pairings are only 0.87 and 0.82 points. It is difficult to distinguish these smaller gains from experimental fluctuations and the additional computation may be harder to justify. The variation across pairings also leaves open when student-prefix adaptation is most beneficial.

[1] Zhu S, Ye X, Lu H, et al. The many faces of on-policy distillation: Pitfalls, mechanisms, and fixes[J]. arXiv preprint arXiv:2605.11182, 2026.

Questions for authors
  1. Could the authors provide stronger evidence that the 0.87- and 0.82-point gains are reliable? An analysis of when gains are larger—for example, whether they depend on student initialization or prior distillation—would also help clarify the method’s practical value.

  2. Could the authors provide additional experimental evidence on both other model families, such as GLM, and domains beyond mathematical reasoning, such as coding or agent tasks, to assess generalization along more dimensions?

1 comment
AuthorsOpenReview ↗

Response to Reviewer cnn7

Thank you for the careful review and constructive feedback. We address each point below.


W1: The additional contribution relative to existing teacher-adaptation work could be better established.

We agree that [1] identifies student-prefix mismatch and shows that RLVR-adapted teachers help OPD. SPTA differs in the states on which the teacher is optimized:

  • Task-level teacher RL ([1] and our Tea-RL) starts from the question only.
  • SPTA starts from the question plus a randomly truncated student rollout. The prefix is only context; the RL loss applies only to the teacher's continuation.

This difference matters in practice. We evaluated the Tea-RL teacher on the same student prefixes as in Table 2 (config (1)). Tea-RL uses the same RL algorithm, hyperparameters, update count, and context budget as SPTA. The results show that Task-level RL does not fix the mismatch: it lowers recovery at every prefix ratio. SPTA raises recovery by 17.7–25.3 points.

Teacherr=0r=25%r=50%r=75%r=90%Student Avg.
Original60.9932.7628.9124.8222.3631.52
Tea-RL52.0428.8125.2420.9818.6531.97
SPTA54.3950.4649.8848.5647.6835.30

We also ran the same RLVR on prefixes from the teacher's own rollouts (Tea-Self-Prefix), with the same truncation procedure (config (1), student Avg.). The results show that own-prefix adaptation gives almost no gain. So the benefit comes from training on student prefixes, not from generic mid-trajectory continuation.

OPDTea-Self-Prefix-RL-OPDTea-RL-OPDSPTA-OPD
31.5231.6231.9735.30

W2 & Q1: Reliability of the 0.87/0.82-point gains, and when gains are larger.

Reliability. We repeated all main comparisons with 3 independent training seeds under the same evaluation protocol. The 5-benchmark averages are reported as mean (± std):

PairingOPDTea-RL-OPDSPTA-OPDSPTA − Tea-RL-OPD
(1) Qwen3-4B → Qwen3-4B-Base31.24 (±0.27)31.91 (±0.48)35.31 (±0.30)+3.40
(2) R1-Distill-7B → R1-Distill-1.5B30.08 (±0.14)32.46 (±0.08)33.31 (±0.08)+0.85
(3) JustRL-1.5B → R1-Distill-1.5B–41.86 (±0.14)42.82 (±0.33)+0.96

Per-benchmark results: (Results for the other two model pairs are presented in Table 7 of the paper.)

  • Qwen3-4B → Qwen3-4B-Base
MethodAIME24AIME25AMC23HMMT24HMMT25Avg.
OPD33.33 (±1.16)25.97 (±0.12)65.59 (±0.13)15.83 (±1.16)15.48 (±1.05)31.24 (±0.27)
Tea-RL-OPD34.44 (±1.77)27.01 (±1.07)66.42 (±1.04)16.32 (±0.44)15.35 (±1.57)31.91 (±0.48)
SPTA-OPD37.22 (±1.03)32.36 (±1.39)72.34 (±0.50)16.46 (±0.42)18.19 (±1.56)35.31 (±0.30)

When gains are larger. The gain over OPD with the original teacher grows with the initial teacher–student distribution gap:

  • (1) +4.07. The teacher is fully post-trained; the student is a base model with no post-training. This is the largest gap: the original teacher drops 38.6 points from r=0 to r=90%.
  • (2) +3.23. Both models are distilled from R1, but they still differ in scale and behavior. The teacher drops 10.1 points.
  • (3) +0.96. The teacher is RL-trained from the student itself, so they start from the same distribution. This is the smallest gap.

A new experiment supports this trend. We first SFT the Qwen3-4B-Base student to reduce the gap, then run each method (single seed). With a closer student, the gain becomes smaller but stays clear. In short, SPTA helps most when the student is far from the teacher, e.g., base or lightly trained students. This is a common case in practice.

Student init.OPDTea-RL-OPDSPTA-OPDSPTA − Tea-RL-OPD
Qwen3-4B-Base31.5231.9735.30+3.33
Qwen3-4B-Base + SFT32.1933.1735.51+2.34

In summary, these results demonstrate the necessity and effectiveness of our method.

Q2: Other model families (e.g., GLM) and domains (coding, agents).

We agree this would strengthen the paper. Our current scope is mathematical reasoning with Qwen3 and DeepSeek-R1-Distill teachers. SPTA only needs a verifiable outcome reward, so it applies directly to coding and agent tasks. Given the emergency review and limited response time, we leave other families and domains for future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We thank all reviewers for their insightful and constructive comments. Our work identifies a teacher-side distribution mismatch in on-policy distillation (OPD) and proposes Student-Prefix Teacher Adaptation (SPTA), which adapts the teacher with RLVR on student-generated prefixes before standard OPD. We are glad that the reviewers recognize the following strengths:

  • An important and relevant problem: a teacher with strong standalone performance may still give unreliable guidance on student prefixes (Reviewers cnn7, 6ssS, 8z1M).
  • A simple, well-motivated, and clean method: SPTA needs no error annotations and keeps the OPD objective unchanged, so it is compatible with other OPD improvements (Reviewers cnn7, 6ssS, AUd3, 8z1M).
  • An insightful analysis: separating standalone accuracy from prefix recovery on identical student prefixes offers a useful way to evaluate teachers (Reviewers cnn7, 6ssS, 8z1M).
  • Consistent gains across three teacher–student pairings, including over task-level teacher RL (Reviewers cnn7, AUd3, 8z1M).
  • Clear writing and candid scope (Reviewer 6ssS).

The main concerns were the reliability of the smaller gains, whether student prefixes are essential, and the mechanism. We addressed them with new experiments, all of which are already incorporated into the revised PDF and marked in blue:

  1. Reliability of gains (cnn7, 6ssS, AUd3, 8z1M). We ran 3 independent training seeds for all main comparisons. SPTA outperforms task-level teacher RL in all three pairings: +3.40 / +0.85 / +0.96 points. (§6, Appendix C)

  2. Student prefixes are essential (6ssS, AUd3, 8z1M). We added two prefix-source controls, which demonstrate that the gain comes from the target student's states, not from generic prefix-conditioned training. (§6, Appendix D)

  3. Mechanism (8z1M, 6ssS). We conducted a token-level analysis of the OPD feedback on the same student rollouts, which shows that the adapted teacher stops rubber-stamping confidently wrong student states (endorsement 86.4% → 75.3%, strong-objection rate 2.16% → 4.45%) and, where it objects, points to self-correction 4.4× more often (2.9% → 12.9%). (new §7.3 and Table 3; Appendix E)

To sum up, SPTA is a simple and effective way to make teachers reliable on the states their students actually visit, and the new experiments confirm that its gains are robust and come specifically from student prefixes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.