- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates conflicts between teacher supervision and verifier feedback in On-Policy Distillation (OPD) and proposes Support-Preserving On-Policy Distillation (SP-OPD). The method uses KL projection to correct the teacher's probability distribution, ensuring consistent supervision across both reverse-KL and forward-KL objectives. It also introduces sequence-balanced reduction to mitigate trajectory-length bias in loss aggregation. Experimental results based on three Qwen3 student models of varying parameter scales and five mathematical reasoning benchmarks demonstrate that SP-OPD surpasses EOPD in both Macro average Avg@8 and Pass@8 metrics.
Critical strengths
-
The KL projection method proposed in this paper can provide a closed-form solution (analytical solution) for the target distribution under given probability constraints, and gradient analysis further substantiates the method's validity under both reverse KL and forward KL optimization objectives, thus offering a theoretical foundation for the proposed target adjustment mechanism.
-
Beyond the final benchmark results, this paper evaluates the effectiveness of the proposed mechanism from multiple perspectives by analyzing the prevalence of conflicts between teacher supervision signals and verifier feedback, examining gradient behaviors, and conducting ablation studies. The experimental results demonstrate that SP-OPD enhances the Macro average Avg@8 and Pass@8 metrics across student models of three different scales, with the improvement in Pass@8 being particularly pronounced. Together, these analyses and experimental findings provide substantial empirical support for the effectiveness of the proposed approach.
Critical weaknesses
-
Recent research on verifier-aware OPD has discussed the teacher–verifier conflict issue. Although this paper employs a different objective rectification mechanism, the substantive differences between KL projection and verifier-aware distillation methods have not been adequately clarified. Moreover, there is a lack of sufficient analysis or experimental evidence to demonstrate its unique advantages in mitigating this conflict, so the incremental innovative value of the method itself remains to be further defined.
-
The existing ablation experiments mainly use a cumulative design, which fails to independently assess the impact of sequence-balance reduction on model performance. Particularly in experiments with the 4B model, a significant portion of the performance improvement in Pass@8 occurred after introducing this component. Therefore, it is difficult to accurately distinguish the respective contributions of sequence-balance reduction and KL projection, and it is also insufficient to determine whether KL projection can provide stable and significant performance improvements without relying on this component.
Questions for authors
-
Although SP-OPD achieves positive macro-average performance improvements across different model scales, some of the gains are concentrated on relatively smaller evaluation benchmarks, and the results may be affected by evaluation fluctuations. In addition, the paper has not reported repeated experimental results under different independent training random seeds, making it difficult to determine whether the observed performance improvements are statistically stable. Could the authors provide experimental results with multiple random seeds and the corresponding statistical analysis to further validate the reliability of the reported performance gains?
-
Currently, all experiments are limited to the Qwen3 model family, and it is unclear whether the observed performance improvements depend on specific attributes of this model family or the existing teacher–student setup. Could the authors further provide experimental results on other model families or with different teacher–student combinations to verify whether the effectiveness of SP-OPD can generalize beyond the current Qwen3 experimental settings?
6 comments
Response to Q2
The existing ablation experiments mainly use a cumulative design, which fails to independently assess the impact of sequence-balance reduction on model performance. Particularly in experiments with the 4B model, a significant portion of the performance improvement in Pass@8 occurred after introducing this component. Therefore, it is difficult to accurately distinguish the respective contributions of sequence-balance reduction and KL projection, and it is also insufficient to determine whether KL projection can provide stable and significant performance improvements without relying on this component.
We added separate component controls comparing EOPD, EOPD with projection, EOPD with sequence balancing, and SP-OPD. All variants use G1 sampling. In these controls, balancing changes the clipped reverse-KL reduction while the forward-KL reduction remains fixed.
Evaluation samples eight answers per question on the five paper benchmarks, with equal benchmark weights. Values are reported as mean ± SD over three independent stochastic evaluations of the same trained checkpoint for each method
| Student | Method | Macro Avg@8 (%) | Macro Pass@8 (%) |
|---|---|---|---|
| 1.7B | EOPD | 29.76 ± 0.20 | 49.94 ± 0.83 |
| 1.7B | EOPD with balancing | 29.10 ± 0.85 | 50.49 ± 0.82 |
| 1.7B | EOPD with projection | 30.09 ± 0.47 | 49.12 ± 0.56 |
| 1.7B | SP-OPD | 30.56 ± 0.56 | 50.24 ± 0.01 |
| 4B | EOPD | 43.33 ± 0.54 | 63.18 ± 2.10 |
| 4B | EOPD with balancing | 42.04 ± 0.56 | 61.53 ± 1.25 |
| 4B | EOPD with projection | 43.15 ± 0.26 | 62.79 ± 1.91 |
| 4B | SP-OPD | 43.47 ± 0.39 | 64.48 ± 1.51 |
These results show that the contributions of projection and sequence balancing are not simply interchangeable. Their individual effects vary across scales and metrics, while their combination yields the strongest Avg@8 at both scales and the strongest accuracy–coverage pair at 4B. Together with the paper's masking-versus-projection ablation, these controls support the role of correcting the teacher target rather than attributing the full improvement solely to sequence balancing.
Response to Q3
Although SP-OPD achieves positive macro-average performance improvements across different model scales, some of the gains are concentrated on relatively smaller evaluation benchmarks, and the results may be affected by evaluation fluctuations. In addition, the paper has not reported repeated experimental results under different independent training random seeds, making it difficult to determine whether the observed performance improvements are statistically stable. Could the authors provide experimental results with multiple random seeds and the corresponding statistical analysis to further validate the reliability of the reported performance gains?
We report the paper results alongside additional EOPD and SP-OPD models trained with seeds 43 and 44. The training configuration and dataset are consistent with those in the paper. Each cell reports Macro Avg@8 / Macro Pass@8 (%).
| Student | Method | Paper result | Training seed 43 | Training seed 44 | Macro Avg@8, mean ± SD (%) | Macro Pass@8, mean ± SD (%) |
|---|---|---|---|---|---|---|
| 1.7B | EOPD | 30.63 / 46.96 | 29.70 / 51.15 | 30.86 / 49.79 | 30.40 ± 0.61 | 49.30 ± 2.14 |
| 1.7B | SP-OPD | 31.20 / 50.24 | 30.57 / 50.59 | 30.70 / 50.65 | 30.82 ± 0.33 | 50.49 ± 0.22 |
| 4B | EOPD | 42.66 / 60.71 | 35.27 / 57.93 | 39.80 / 58.52 | 39.24 ± 3.73 | 59.05 ± 1.47 |
| 4B | SP-OPD | 43.92 / 64.91 | 37.49 / 58.29 | 37.90 / 57.77 | 39.77 ± 3.60 | 60.32 ± 3.98 |
Across the three reported results, SP-OPD achieves higher mean Macro Avg@8 and Pass@8 than EOPD at both student scales. The improvements are 0.43 and 1.20 percentage points at 1.7B, and 0.53 and 1.27 points at 4B, respectively. For each metric, SP-OPD scores higher in two of the three comparisons at each scale. These results support improvements in both overall answer accuracy and solution coverage across the two student sizes.
Benchmark-level results are reported as mean ± SD over the same three training-run evaluations shown above.
| Student | Benchmark (questions) | EOPD Avg@8 / Pass@8 (%) | SP-OPD Avg@8 / Pass@8 (%) |
|---|---|---|---|
| 1.7B | MATH-500 (500) | 68.02 ± 0.43 / 87.60 ± 0.35 | 67.17 ± 1.00 / 86.20 ± 1.04 |
| 1.7B | OlympiadBench (675) | 30.80 ± 0.19 / 53.33 ± 1.21 | 31.00 ± 0.31 / 55.16 ± 1.07 |
| 4B | MATH-500 (500) | 75.05 ± 5.65 / 92.13 ± 0.61 | 72.88 ± 5.55 / 91.87 ± 0.90 |
| 4B | OlympiadBench (675) | 40.71 ± 3.61 / 63.95 ± 0.73 | 39.94 ± 3.13 / 62.52 ± 1.18 |
The 1.7B results also show improved solution coverage on OlympiadBench, which contains 675 questions. Across the same three results, mean Pass@8 rises from 53.33% to 55.16% (+1.83 percentage points), while mean Avg@8 increases from 30.80% to 31.00%. SP-OPD achieves higher Pass@8 in each of the three comparisons on this benchmark, showing that the coverage gain extends to a larger evaluation set across all three reported results.
Response to Q1
SG-OPD [1] adjusts teacher supervision according to agreement between teacher and verifier signals, and uses verified teacher responses during early training. RG-OPD [2] gates teacher supervision using reward advantages and teacher–student trajectory likelihood differences. SPOT [3] probes alternative tokens at selected positions and uses verified student continuations to adjust its target.
SP-OPD directly corrects the teacher distribution. For a response with a verified correct final answer, when the teacher assigns less probability to a sampled token than the student that generated it, the projection raises its target probability to the student's value. It proportionally rescales the other tokens, preserving their relative probabilities. Positions without a conflict retain the original teacher distribution. Both the clipped reverse-KL and forward-KL terms use the projected target.
The specific contribution is a closed-form distribution correction that addresses conflicting teacher targets in both loss terms while preserving the teacher's relative preferences among alternative tokens. In the paper's 4B ablation, adding projection to masking increases Macro Avg@8 from 42.29% to 43.49% and Macro Pass@8 from 61.20% to 62.00%.
We added comparisons with SG-OPD, RG-OPD, and SPOT using Qwen3-1.7B-Base and Qwen3-4B-Base students with a Qwen3-8B teacher. G1 and G8 denote one and eight primary student responses per training question. Additional teacher responses in SG-OPD and branch continuations in SPOT are separate from this count. G1 is the paper's main setting; the additional G8 setting examines training with multiple responses per question. We also newly trained EOPD and SP-OPD for G8. Both settings evaluate eight answers per question.
The training configuration and dataset are consistent with those in the paper. Benchmark results are reported as Macro Avg@8 and Macro Pass@8 (%). Bold values identify the highest scores within each student and training setting.
The direct comparisons demonstrate the competitiveness of SP-OPD against existing verifier-aware distillation methods. Across G1 and G8, SP-OPD achieves the highest Macro Pass@8 in all four student–setting groups and the highest Macro Avg@8 in three of four. These results support the effectiveness of target-level correction alongside existing verifier-aware approaches. Unlike methods that primarily gate or reweight teacher supervision, SP-OPD constructs a closed-form projected distribution and uses it consistently in both KL loss terms. This correction does not require additional candidate continuations. The comparisons therefore provide empirical support for a distinct way of incorporating verifier feedback into on-policy distillation.