acceptodds
Under review as a conference paper at ICLR 2027

When Teachers Conflict with Verifiers: Support-Preserving On-Policy Distillation

Abstract

On-policy distillation (OPD) removes state mismatch by querying the teacher on student trajectories, yet it still treats the teacher distribution as valid at every visited state. In verifiable reasoning, an outcome verifier can accept a trajectory even when the teacher assigns one of its sampled tokens less probability than the behavior policy. We call this a teacher–verifier conflict: OPD’s reverse-KL update suppresses behavior from a correct trajectory, and EOPD’s forward-KL term can repeat the same error. We introduce Support-Preserving On-Policy Distillation (SP-OPD). On correct trajectories, SP-OPD requires the target probability of each sampled token to remain at least its behavior-policy probability. KL projection produces the closest feasible target in closed form, changing only a conflict position and preserving the teacher’s pairwise probability ratios among alternatives. The projected target corrects both EOPD loss terms, while sequence-balanced reduction aligns the sampled-token loss with trajectory-level verification. Conflicts cover 41.57% of sampled tokens on correct trajectories from a trained EOPD policy and persist throughout training. Across 0.6B, 1.7B, and 4B Qwen3-Base students, SP-OPD improves both Avg@8 and Pass@8 at every scale. At 4B it reaches 43.92% Avg@8 and 64.91% Pass@8, gains of 1.26 and 4.20 points over EOPD.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, knowledge distillation, large language models, verifiable reasoning, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_k5JwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates conflicts between teacher supervision and verifier feedback in On-Policy Distillation (OPD) and proposes Support-Preserving On-Policy Distillation (SP-OPD). The method uses KL projection to correct the teacher's probability distribution, ensuring consistent supervision across both reverse-KL and forward-KL objectives. It also introduces sequence-balanced reduction to mitigate trajectory-length bias in loss aggregation. Experimental results based on three Qwen3 student models of varying parameter scales and five mathematical reasoning benchmarks demonstrate that SP-OPD surpasses EOPD in both Macro average Avg@8 and Pass@8 metrics.

Critical strengths
  1. The KL projection method proposed in this paper can provide a closed-form solution (analytical solution) for the target distribution under given probability constraints, and gradient analysis further substantiates the method's validity under both reverse KL and forward KL optimization objectives, thus offering a theoretical foundation for the proposed target adjustment mechanism.

  2. Beyond the final benchmark results, this paper evaluates the effectiveness of the proposed mechanism from multiple perspectives by analyzing the prevalence of conflicts between teacher supervision signals and verifier feedback, examining gradient behaviors, and conducting ablation studies. The experimental results demonstrate that SP-OPD enhances the Macro average Avg@8 and Pass@8 metrics across student models of three different scales, with the improvement in Pass@8 being particularly pronounced. Together, these analyses and experimental findings provide substantial empirical support for the effectiveness of the proposed approach.

Critical weaknesses
  1. Recent research on verifier-aware OPD has discussed the teacher–verifier conflict issue. Although this paper employs a different objective rectification mechanism, the substantive differences between KL projection and verifier-aware distillation methods have not been adequately clarified. Moreover, there is a lack of sufficient analysis or experimental evidence to demonstrate its unique advantages in mitigating this conflict, so the incremental innovative value of the method itself remains to be further defined.

  2. The existing ablation experiments mainly use a cumulative design, which fails to independently assess the impact of sequence-balance reduction on model performance. Particularly in experiments with the 4B model, a significant portion of the performance improvement in Pass@8 occurred after introducing this component. Therefore, it is difficult to accurately distinguish the respective contributions of sequence-balance reduction and KL projection, and it is also insufficient to determine whether KL projection can provide stable and significant performance improvements without relying on this component.

Questions for authors
  1. Although SP-OPD achieves positive macro-average performance improvements across different model scales, some of the gains are concentrated on relatively smaller evaluation benchmarks, and the results may be affected by evaluation fluctuations. In addition, the paper has not reported repeated experimental results under different independent training random seeds, making it difficult to determine whether the observed performance improvements are statistically stable. Could the authors provide experimental results with multiple random seeds and the corresponding statistical analysis to further validate the reliability of the reported performance gains?

  2. Currently, all experiments are limited to the Qwen3 model family, and it is unclear whether the observed performance improvements depend on specific attributes of this model family or the existing teacher–student setup. Could the authors further provide experimental results on other model families or with different teacher–student combinations to verify whether the effectiveness of SP-OPD can generalize beyond the current Qwen3 experimental settings?

6 comments

Response to Q1

Recent research on verifier-aware OPD has discussed the teacher–verifier conflict issue. Although this paper employs a different objective rectification mechanism, the substantive differences between KL projection and verifier-aware distillation methods have not been adequately clarified. Moreover, there is a lack of sufficient analysis or experimental evidence to demonstrate its unique advantages in mitigating this conflict, so the incremental innovative value of the method itself remains to be further defined.

SG-OPD [1] adjusts teacher supervision according to agreement between teacher and verifier signals, and uses verified teacher responses during early training. RG-OPD [2] gates teacher supervision using reward advantages and teacher–student trajectory likelihood differences. SPOT [3] probes alternative tokens at selected positions and uses verified student continuations to adjust its target.

SP-OPD directly corrects the teacher distribution. For a response with a verified correct final answer, when the teacher assigns less probability to a sampled token than the student that generated it, the projection raises its target probability to the student's value. It proportionally rescales the other tokens, preserving their relative probabilities. Positions without a conflict retain the original teacher distribution. Both the clipped reverse-KL and forward-KL terms use the projected target.

The specific contribution is a closed-form distribution correction that addresses conflicting teacher targets in both loss terms while preserving the teacher's relative preferences among alternative tokens. In the paper's 4B ablation, adding projection to masking increases Macro Avg@8 from 42.29% to 43.49% and Macro Pass@8 from 61.20% to 62.00%.

We added comparisons with SG-OPD, RG-OPD, and SPOT using Qwen3-1.7B-Base and Qwen3-4B-Base students with a Qwen3-8B teacher. G1 and G8 denote one and eight primary student responses per training question. Additional teacher responses in SG-OPD and branch continuations in SPOT are separate from this count. G1 is the paper's main setting; the additional G8 setting examines training with multiple responses per question. We also newly trained EOPD and SP-OPD for G8. Both settings evaluate eight answers per question.

The training configuration and dataset are consistent with those in the paper. Benchmark results are reported as Macro Avg@8 and Macro Pass@8 (%). Bold values identify the highest scores within each student and training setting.

StudentSettingMethodMacro Avg@8 (%)Macro Pass@8 (%)
1.7BG1EOPD30.6346.96
1.7BG1SPOT30.1149.56
1.7BG1SP-OPD31.2050.24
4BG1EOPD42.6660.71
4BG1SPOT41.1564.39
4BG1SP-OPD43.9264.91
1.7BG8EOPD29.0949.29
1.7BG8SG-OPD28.8847.51
1.7BG8RG-OPD30.2149.52
1.7BG8SP-OPD29.5351.50
4BG8EOPD40.3457.48
4BG8SG-OPD39.5456.98
4BG8RG-OPD41.0160.18
4BG8SP-OPD42.0560.49

The direct comparisons demonstrate the competitiveness of SP-OPD against existing verifier-aware distillation methods. Across G1 and G8, SP-OPD achieves the highest Macro Pass@8 in all four student–setting groups and the highest Macro Avg@8 in three of four. These results support the effectiveness of target-level correction alongside existing verifier-aware approaches. Unlike methods that primarily gate or reweight teacher supervision, SP-OPD constructs a closed-form projected distribution and uses it consistently in both KL loss terms. This correction does not require additional candidate continuations. The comparisons therefore provide empirical support for a distinct way of incorporating verifier feedback into on-policy distillation.

Response to Q2

The existing ablation experiments mainly use a cumulative design, which fails to independently assess the impact of sequence-balance reduction on model performance. Particularly in experiments with the 4B model, a significant portion of the performance improvement in Pass@8 occurred after introducing this component. Therefore, it is difficult to accurately distinguish the respective contributions of sequence-balance reduction and KL projection, and it is also insufficient to determine whether KL projection can provide stable and significant performance improvements without relying on this component.

We added separate component controls comparing EOPD, EOPD with projection, EOPD with sequence balancing, and SP-OPD. All variants use G1 sampling. In these controls, balancing changes the clipped reverse-KL reduction while the forward-KL reduction remains fixed.

Evaluation samples eight answers per question on the five paper benchmarks, with equal benchmark weights. Values are reported as mean ± SD over three independent stochastic evaluations of the same trained checkpoint for each method

StudentMethodMacro Avg@8 (%)Macro Pass@8 (%)
1.7BEOPD29.76 ± 0.2049.94 ± 0.83
1.7BEOPD with balancing29.10 ± 0.8550.49 ± 0.82
1.7BEOPD with projection30.09 ± 0.4749.12 ± 0.56
1.7BSP-OPD30.56 ± 0.5650.24 ± 0.01
4BEOPD43.33 ± 0.5463.18 ± 2.10
4BEOPD with balancing42.04 ± 0.5661.53 ± 1.25
4BEOPD with projection43.15 ± 0.2662.79 ± 1.91
4BSP-OPD43.47 ± 0.3964.48 ± 1.51

These results show that the contributions of projection and sequence balancing are not simply interchangeable. Their individual effects vary across scales and metrics, while their combination yields the strongest Avg@8 at both scales and the strongest accuracy–coverage pair at 4B. Together with the paper's masking-versus-projection ablation, these controls support the role of correcting the teacher target rather than attributing the full improvement solely to sequence balancing.

Response to Q3

Although SP-OPD achieves positive macro-average performance improvements across different model scales, some of the gains are concentrated on relatively smaller evaluation benchmarks, and the results may be affected by evaluation fluctuations. In addition, the paper has not reported repeated experimental results under different independent training random seeds, making it difficult to determine whether the observed performance improvements are statistically stable. Could the authors provide experimental results with multiple random seeds and the corresponding statistical analysis to further validate the reliability of the reported performance gains?

We report the paper results alongside additional EOPD and SP-OPD models trained with seeds 43 and 44. The training configuration and dataset are consistent with those in the paper. Each cell reports Macro Avg@8 / Macro Pass@8 (%).

StudentMethodPaper resultTraining seed 43Training seed 44Macro Avg@8, mean ± SD (%)Macro Pass@8, mean ± SD (%)
1.7BEOPD30.63 / 46.9629.70 / 51.1530.86 / 49.7930.40 ± 0.6149.30 ± 2.14
1.7BSP-OPD31.20 / 50.2430.57 / 50.5930.70 / 50.6530.82 ± 0.3350.49 ± 0.22
4BEOPD42.66 / 60.7135.27 / 57.9339.80 / 58.5239.24 ± 3.7359.05 ± 1.47
4BSP-OPD43.92 / 64.9137.49 / 58.2937.90 / 57.7739.77 ± 3.6060.32 ± 3.98

Across the three reported results, SP-OPD achieves higher mean Macro Avg@8 and Pass@8 than EOPD at both student scales. The improvements are 0.43 and 1.20 percentage points at 1.7B, and 0.53 and 1.27 points at 4B, respectively. For each metric, SP-OPD scores higher in two of the three comparisons at each scale. These results support improvements in both overall answer accuracy and solution coverage across the two student sizes.

Benchmark-level results are reported as mean ± SD over the same three training-run evaluations shown above.

StudentBenchmark (questions)EOPD Avg@8 / Pass@8 (%)SP-OPD Avg@8 / Pass@8 (%)
1.7BMATH-500 (500)68.02 ± 0.43 / 87.60 ± 0.3567.17 ± 1.00 / 86.20 ± 1.04
1.7BOlympiadBench (675)30.80 ± 0.19 / 53.33 ± 1.2131.00 ± 0.31 / 55.16 ± 1.07
4BMATH-500 (500)75.05 ± 5.65 / 92.13 ± 0.6172.88 ± 5.55 / 91.87 ± 0.90
4BOlympiadBench (675)40.71 ± 3.61 / 63.95 ± 0.7339.94 ± 3.13 / 62.52 ± 1.18

The 1.7B results also show improved solution coverage on OlympiadBench, which contains 675 questions. Across the same three results, mean Pass@8 rises from 53.33% to 55.16% (+1.83 percentage points), while mean Avg@8 increases from 30.80% to 31.00%. SP-OPD achieves higher Pass@8 in each of the three comparisons on this benchmark, showing that the coverage gain extends to a larger evaluation set across all three reported results.

Reviewer_Fe4vOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SP-OPD to address cases where teacher-guided distillation suppresses sampled tokens from verifier-accepted trajectories. It projects the teacher distribution onto probability constraints that preserve the behavior-policy probabilities of these tokens, applies the projected targets to both EOPD loss terms, and introduces sequence-balanced reduction. Experiments across three Qwen3 student scales show improvements in average accuracy and Pass@8 over EOPD.

Critical strengths
  1. The paper is well organized and easy to follow, with clear explanations of the motivation, mathematical formulation, and experimental results.
  2. The method offers a simple, closed-form target correction that preserves the teacher’s relative preferences among alternative tokens. Applying this correction consistently to both distillation terms is technically well motivated.
Critical weaknesses
  1. A correct answer does not justify preserving every sampled token’s probability. The teacher may reasonably favor more reliable or efficient reasoning paths. The proposed lower bound is therefore a conservative design choice rather than a necessary consequence of verification. Please distinguish harmful suppression from beneficial distribution correction and establish which drives the gains.
  2. Effectiveness in thinking mode remains unclear. Correct long reasoning trajectories can contain failed attempts, backtracking, and self-correction. Protecting every sampled token may preserve both useful exploration and local errors. Evaluation in thinking mode is needed to assess accuracy and reasoning efficiency.
  3. The ablations do not isolate the contribution of projection. Sequence balancing produces the largest marginal Pass@8 gain in Figure 2 (+2.91), yet Table 5 omits EOPD with sequence-balanced reduction alone. Please add this control and report uncertainty to separate the benefits of normalization, KL projection, and their interaction.
  4. Sensitivity to teacher strength is unexplored. All experiments use Qwen3-8B in non-thinking mode as the teacher. It is unclear whether SP-OPD mainly compensates for limited teacher capability or remains beneficial with stronger teachers, where preserving student probabilities could restrict useful knowledge transfer. Please evaluate different teacher strengths and their effects on conflict rates and performance gains.
Questions for authors

See Weaknesses

5 comments

Response to Q1

A correct answer does not justify preserving every sampled token’s probability. The teacher may reasonably favor more reliable or efficient reasoning paths. The proposed lower bound is therefore a conservative design choice rather than a necessary consequence of verification. Please distinguish harmful suppression from beneficial distribution correction and establish which drives the gains.

SP-OPD uses the verification result for the whole response as the basis for its training constraint, without assuming that every intermediate token is independently correct or optimal. For verified correct responses, we impose a lower bound on the sampled token's target probability to avoid suppressing it solely because the teacher disagrees. At conflicting positions, the projection raises the target probability only to the probability assigned by the student that generated the token. It proportionally rescales the other tokens, preserving the teacher's relative probabilities among them. Positions without a conflict retain the original teacher distribution. Among distributions satisfying the constraint, the projection minimizes KL divergence to the original teacher distribution and makes the smallest adjustment needed to meet the constraint.

The corrected target is used in both loss terms. In the paper's 4B ablation, projection improves Macro Avg@8 from 42.29% to 43.49% and Macro Pass@8 from 61.20% to 62.00% compared with masking negative advantages alone. These results show that correcting the conflicting teacher target improves answer accuracy and question coverage beyond masking the reverse-KL update.

Response to Q2

Effectiveness in thinking mode remains unclear. Correct long reasoning trajectories can contain failed attempts, backtracking, and self-correction. Protecting every sampled token may preserve both useful exploration and local errors. Evaluation in thinking mode is needed to assess accuracy and reasoning efficiency.

We added a comparison with Qwen3-1.7B-Base as the initial student and Qwen3-8B as the teacher. Student rollouts during training and responses during evaluation use the Qwen3 chat template with enable_thinking=True. The teacher provides token probabilities on the same student-generated prefixes. Both EOPD and SP-OPD are trained on MATH-lighteval for 174 iterations and evaluated using identical prompt and sampling settings.

ModeMethodMacro Avg@8 (%)Macro Pass@8 (%)
NonthinkingEOPD30.6346.96
NonthinkingSP-OPD31.2050.24
ThinkingEOPD22.7342.04
ThinkingSP-OPD24.2142.79

In thinking mode, SP-OPD achieves Macro Avg@8 and Macro Pass@8 of 24.21 and 42.79, compared with EOPD's 22.73 and 42.04. With equal weights across the five benchmarks, its mean generated response length is about 1,009 tokens shorter.

Response to Q3

The ablations do not isolate the contribution of projection. Sequence balancing produces the largest marginal Pass@8 gain in Figure 2 (+2.91), yet Table 5 omits EOPD with sequence-balanced reduction alone. Please add this control and report uncertainty to separate the benefits of normalization, KL projection, and their interaction.

We added EOPD with balancing as a separate component control. The students are Qwen3-1.7B-Base and Qwen3-4B-Base, with a Qwen3-8B teacher. Balancing gives each response equal weight in the clipped reverse-KL reduction; the forward-KL reduction stays fixed.

All rows use one student's answer for each training question (G1). The training configuration and dataset are consistent with those in the paper. Values are reported as mean ± SD over three independent stochastic evaluations of the same trained checkpoint for each method

Evaluation samples eight answers per question on the five paper benchmarks, with equal benchmark weights.

StudentMethodMacro Avg@8 (%)Macro Pass@8 (%)
1.7BEOPD29.76 ± 0.2049.94 ± 0.83
1.7BEOPD with balancing29.10 ± 0.8550.49 ± 0.82
1.7BEOPD with projection30.09 ± 0.4749.12 ± 0.56
1.7BSP-OPD30.56 ± 0.5650.24 ± 0.01
4BEOPD43.33 ± 0.5463.18 ± 2.10
4BEOPD with balancing42.04 ± 0.5661.53 ± 1.25
4BEOPD with projection43.15 ± 0.2662.79 ± 1.91
4BSP-OPD43.47 ± 0.3964.48 ± 1.51

SP-OPD improves both metrics over EOPD in the paper's G1 experiments and has the highest Macro Avg@8 at both student sizes in this table. It also has the highest Macro Pass@8 at 4B.

Response to Q4

Sensitivity to teacher strength is unexplored. All experiments use Qwen3-8B in non-thinking mode as the teacher. It is unclear whether SP-OPD mainly compensates for limited teacher capability or remains beneficial with stronger teachers, where preserving student probabilities could restrict useful knowledge transfer. Please evaluate different teacher strengths and their effects on conflict rates and performance gains.

We compare 4B, 8B, and 32B Qwen3 teachers with the 1.7B student. The 8B-teacher student scores use the paper's G1 results; the student scores with the 4B and 32B teachers come from the added teacher-strength experiments. The 8B teacher scores come from Table 2 of the paper. A conflict occurs when the teacher assigns less probability to a sampled token than the student that generated it. Conflict rates are measured separately on correct, completed responses from the teacher-strength training controls.

TeacherTeacher Macro Avg@8 / Pass@8 (%)EOPD student Macro Avg@8 / Pass@8 (%)SP-OPD student Macro Avg@8 / Pass@8 (%)Conflict rate, EOPD / SP-OPD (%)
4B48.04 / 62.9129.03 / 48.9330.45 / 50.8626.70 / 21.79
8B49.74 / 68.3830.63 / 46.9631.20 / 50.2427.95 / 23.48
32B51.65 / 68.3629.16 / 49.6829.70 / 50.4941.36 / 38.01

SP-OPD improves both Macro Avg@8 and Macro Pass@8 with the 4B, 8B, and 32B teachers in this comparison. SP-OPD has lower conflict rates with all three teachers.


We hope these responses and the new experiments address the reviewer's concerns, and we would be grateful if the reviewer would consider raising the score. We are happy to answer any further questions.

Dear Reviewer Fe4v,

As the rebuttal period is coming to a close, we sincerely appreciate your time and thoughtful feedback. We hope our additional experiments and detailed responses have addressed your concerns, and would be grateful if you could reconsider your rating.

We look forward to further discussion and appreciate your consideration!

Reviewer_m7pkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a variant of on policy distillation (OPD) for when there are verified rewards. For such cases, it differentiates between instances when the student gets the correct answer and when it does not. When the student is correct, the paper aims to change the OPD loss at certain tokens - specifically, those where the teacher's probability of the student token is smaller than the student's probability for that token. Vanilla OPD would have resulted in decreased probability for that token, which is considered to be a bad thing by this paper. So for those tokens only it changes the teacher to a slightly different distribution so as to leave that token's probability the same between teacher and student. Comparisons are mainly to EOPD, a prior work that is blind to final reward.

Critical strengths

An interesting and concrete target correction. The closed-form projection provides a principled way to protect successful student behavior while preserving the teacher’s relative probabilities among alternative tokens.

A useful analysis of the two loss paths. The paper explains why masking negative sampled-token advantages alone can leave conflicting forward-KL supervision intact, and includes ablations that test this distinction.

Consistent aggregate improvements over EOPD. These appear at all three student sizes, with particularly encouraging improvements in solution coverage measured by Pass@8. (However, EOPD is blind to trajectory success or failure)

Critical weaknesses

A couple very relevant baseline comparisons are missing; the paper compares to EOPD but this does not use student success to assess teacher guidance. Consequently, improvements over EOPD do not establish the advantage over existing verifier-aware approaches. A direct comparison with SG-OPD is particularly important; comparisons with RG-OPD and SPOT would also substantially strengthen the evaluation, subject to their timing relative to submission.

The masking ablation does not replace these comparisons. Masking negative advantages on successful trajectories is a useful component control, but it does not reproduce the complete algorithms of SG-OPD or RG-OPD. The current experiments therefore leave uncertainty about whether target projection improves over competing ways of incorporating verifier feedback.

The preservation principle deserves qualification. Outcome verification establishes that a response reaches a correct answer, but does not certify every intermediate token. The projection is mathematically optimal under the chosen constraint; this does not by itself establish that preserving every sampled token is the best training policy.

Questions for authors

I think a principled apples-to-apples comparison against SG-OPD and RG-OPD (i.e. same student, same teacher, same verified-reward setting etc.) should be possible to accomplish. All of these papers make intuitively appealing but slightly different suggestions, so a direct comparison could be of value.

4 comments

Response to Q1

I think a principled apples-to-apples comparison against SG-OPD and RG-OPD (i.e. same student, same teacher, same verified-reward setting etc.) should be possible to accomplish. All of these papers make intuitively appealing but slightly different suggestions, so a direct comparison could be of value.

We added SG-OPD[1], RG-OPD[2], and SPOT[3] comparisons for the 1.7B and 4B students using the paper's models and evaluation settings.

G1 and G8 denote one and eight primary student responses sampled per question during training. SG-OPD's teacher rollouts and SPOT's branch continuations are additional samples outside this count. G1 is the paper's main setting and is used for the SPOT comparison. We add G8 to compare with SG-OPD and RG-OPD, which use verifier feedback across several responses to the same question. For the G8 comparison, we additionally trained EOPD and SP-OPD with eight student responses per question. Evaluation samples eight answers per question in both settings.

The students are Qwen3-1.7B-Base and Qwen3-4B-Base, with Qwen3-8B as the teacher. The training configuration and dataset are consistent with those in the paper. In each G8 iteration, 16 questions are sampled, with eight primary student responses for each question.

Benchmark columns report Avg@8 / Pass@8 (%), with equal weight for the five benchmarks in the Macro columns.

StudentSettingMethodMATH-500AMC23OlympiadBenchAIME24AIME25Macro Avg@8 (%)Macro Pass@8 (%)
1.7BG1EOPD68.50 / 87.8040.00 / 65.0030.89 / 52.008.33 / 16.675.42 / 13.3330.6346.96
1.7BG1SPOT66.27 / 85.2039.06 / 70.0030.22 / 52.599.58 / 23.335.42 / 16.6730.1149.56
1.7BG1SP-OPD68.13 / 85.6040.31 / 75.0031.30 / 53.9310.83 / 20.005.42 / 16.6731.2050.24
1.7BG8EOPD64.50 / 86.8039.38 / 67.5028.67 / 52.1510.00 / 30.002.92 / 10.0029.0949.29
1.7BG8SG-OPD65.77 / 84.6035.00 / 65.0029.46 / 51.268.75 / 23.335.42 / 13.3328.8847.51
1.7BG8RG-OPD67.60 / 86.0037.50 / 62.5030.96 / 52.447.92 / 20.007.08 / 26.6730.2149.52
1.7BG8SP-OPD65.58 / 86.8040.00 / 77.5029.15 / 53.197.50 / 16.675.42 / 23.3329.5351.50
StudentSettingMethodMATH-500AMC23OlympiadBenchAIME24AIME25Macro Avg@8 (%)Macro Pass@8 (%)
4BG1EOPD79.45 / 91.6056.25 / 77.5043.44 / 64.4418.33 / 36.6715.83 / 33.3342.6660.71
4BG1SPOT78.50 / 92.4053.44 / 92.5042.13 / 63.7016.67 / 33.3315.00 / 40.0041.1564.39
4BG1SP-OPD79.23 / 92.8060.63 / 85.0043.52 / 63.4117.92 / 36.6718.33 / 46.6743.9264.91
4BG8EOPD77.25 / 90.6052.50 / 82.5041.93 / 64.3016.25 / 23.3313.75 / 26.6740.3457.48
4BG8SG-OPD76.88 / 90.0052.50 / 77.5039.98 / 60.7415.00 / 26.6713.33 / 30.0039.5456.98
4BG8RG-OPD76.25 / 91.4052.50 / 85.0041.70 / 61.1917.08 / 30.0017.50 / 33.3341.0160.18
4BG8SP-OPD78.30 / 91.8055.62 / 80.0042.59 / 64.0016.67 / 26.6717.08 / 40.0042.0560.49

SP-OPD achieves the highest Macro Avg@8 and Pass@8 on both Qwen3 base models (1.7B and 4B) in G1. Its strong performance extends to G8: at 1.7B, Macro Avg@8 is close to RG-OPD and Macro Pass@8 is the highest; at 4B, SP-OPD leads both metrics. These results demonstrate SP-OPD’s strong overall performance across both base models and both training settings.

The added baselines follow their published algorithms. SG-OPD adjusts teacher supervision according to whether the teacher and verifier signals agree, and uses verified teacher responses during early training. RG-OPD uses verifier rewards across answers to the same question to determine how teacher supervision is applied to each response. SPOT samples alternative teacher tokens, generates student continuations, and uses their verified outcomes to adjust the training target. These experiments directly compare different ways of using verifier feedback in distillation.

[1] SG-OPD: https://arxiv.org/abs/2606.09304

[2] RG-OPD: https://arxiv.org/abs/2607.04037

[3] SPOT: https://arxiv.org/abs/2608.04419

Response to Q2

The masking ablation does not replace these comparisons. Masking negative advantages on successful trajectories is a useful component control, but it does not reproduce the complete algorithms of SG-OPD or RG-OPD. The current experiments therefore leave uncertainty about whether target projection improves over competing ways of incorporating verifier feedback.

The masking ablation isolates the conflict between teacher supervision and verifier feedback, rather than approximating SG-OPD or RG-OPD. Masking negative advantages on correct responses removes the conflicting reverse-KL update, but leaves the original teacher distribution as the forward-KL target, which can still suppress the sampled token. SP-OPD projects the teacher distribution and uses the corrected target in both loss terms. In the paper's 4B ablation, projection improves Macro Avg@8 from 42.29% to 43.49% and Macro Pass@8 from 61.20% to 62.00% compared with masking alone.

The new direct comparisons in Q1 further show that this mechanism is competitive with other verifier-aware methods. In the Q1 tables, SP-OPD achieves the highest Macro Avg@8 in G1 at both student scales and in 4B G8; its Macro Avg@8 in 1.7B G8 is close to RG-OPD. SP-OPD also achieves the highest Macro Pass@8 in all four comparisons.

Response to Q3

The preservation principle deserves qualification. Outcome verification establishes that a response reaches a correct answer, but does not certify every intermediate token. The projection is mathematically optimal under the chosen constraint; this does not by itself establish that preserving every sampled token is the best training policy.

SP-OPD uses the verification result for the whole response as the basis for its training constraint, without assuming that every intermediate token is independently correct or optimal. For verified correct responses, we impose a lower bound on the sampled token's target probability to avoid suppressing it solely because the teacher disagrees. When the teacher assigns the sampled token less probability than the student that generated it, the projection raises the target probability to the student's value and proportionally rescales the probabilities of the other tokens, preserving the teacher's relative probabilities among them. Where no conflict occurs, the teacher distribution remains unchanged. Among distributions satisfying the constraint, the projection minimizes KL divergence to the original teacher distribution and therefore makes the smallest adjustment needed to meet the constraint. The projection ablation and the added comparisons in Q1 support the effectiveness of this training rule.


We hope these responses and the new experiments address the reviewer's concerns, and we would be grateful if the reviewer would consider raising the score. We are happy to answer any further questions.

Dear Reviewer m7pk,

As the rebuttal period is coming to a close, we sincerely appreciate your time and thoughtful feedback. We hope our additional experiments and detailed responses have addressed your concerns, and would be grateful if you could reconsider your rating.

We look forward to further discussion and appreciate your consideration!

Dear Reviewer m7pk,

As the rebuttal period is coming to a close, we sincerely appreciate your time and thoughtful feedback. We hope our additional experiments and detailed responses have addressed your concerns, and would be grateful if you could reconsider your rating.

We look forward to further discussion and appreciate your consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.