- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how improvements from GRPO-trained reasoning teachers transfer to smaller students through on-policy distillation. It proposes TS-OPD, which categorizes problems by teacher–student success rates, applies forward KL to knowledge gaps and reverse KL to precision gaps, and introduces token-level gating. Experiments across three student sizes and eight mathematical reasoning benchmarks show improvements in average correctness, although the effectiveness of the proposed routing mechanism is not fully isolated.
Critical strengths
- This paper introduces a useful idea to use joint teacher–student success to distinguish knowledge acquisition from precision improvement. The KL-direction analysis in Eq. 4 provides a reasonable motivation for the proposed routing.
- The evaluation covers three student sizes, two teacher checkpoints, and multiple benchmarks. In particular, Table 3 compares routing against uniform KL objectives on the same selected problems, providing evidence that the improvements are not simply caused by filtering.
Critical weaknesses
- Table 3 shows that routing outperforms uniform forward or reverse KL, but this does not establish that assigning forward KL specifically to knowledge gaps and reverse KL to precision gaps is necessary. The improvement could arise from combining both objectives rather than matching them to the proposed strata. A swapped-routing control or a fixed forward/reverse KL mixture on the same training problems would distinguish these explanations.
- The A/C/P/N labels are determined from only eight samples per model and remain fixed throughout training. Such estimates may misclassify low-success problems, particularly those assigned to the knowledge-gap stratum. Also, Section 5.3 assumes that Direct GRPO receives no reward on A problems, although zero success during calibration does not imply zero reward during training. Moreover, most results use a single training seed, while several improvements over competing objectives are relatively small (Table 2). This makes it difficult to assess whether the reported advantages are robust to sampling and training variability.
Questions for authors
- Would TS-OPD still outperform a fixed forward/reverse KL mixture or a reversed A/C routing strategy under the same training budget?
- How stable are the problem assignments under different calibration samples or larger values of ? Does this affect the final performance?
- Could the authors report multiple training seeds for the main TS-OPD versus routing-only and uniform-KL comparisons, particularly where the improvements are small?
Thank you for the suggestions on routing assignment, calibration stability, and multi-seed comparisons. We address each point below.
1. Is matched routing better than simply mixing the two KL objectives?
Matched routing outperforms both random and reversed routing in avg@8 and pass@8, supporting the value of matching the KL direction to each problem's state.
We use the Qwen3-4B student and GRPO-Qwen3-8B teacher. The three routing variants share the same retained training problems and training budget, with token selection and the entropy brake disabled. Random routing preserves the number of problems assigned to each KL objective while shuffling the problem-to-objective assignment; reversed routing swaps the KL directions assigned to A and C.
Results are below. All values are avg@8 and pass@8 macro averages over seven benchmarks from the same single-run ablation, excluding MATH-test (owing to the limited rebuttal time, and because MATH-test scores are already high and have little effect on the conclusions), in percent.
2. Few calibration samples, fixed labels, and GRPO rewards on A problems
Most A/C labels remain unchanged with more calibration samples, while a small fraction of A problems move to C; zero successes in eight calibration samples also does not imply that no reward is ever received during GRPO training.
We fix the initial models and decoding settings and compare nested prefixes of the same set of responses at K=8, 16, and 32. Problems that move from A to C remain in training, but their KL assignment changes. We will correct the statement that Direct GRPO receives no reward on A problems.
Details are in the two tables below: the first gives each stratum's share of all training problems, and the second gives per-problem retention rates from K=8 to K=32.
Stratum proportions under different calibration budgets (%):
Label transitions from K=8 to K=32:
3. Multiple training seeds for full, routing-only, and uniform KL
We have expanded the experiments to three training seeds: with the post-GRPO teacher, full achieves the highest mean avg@8 at every student size.
The complete table below covers both teacher checkpoints, all methods, and all three student sizes. Values are mean ± standard deviation. avg@8 and pass@8 are equally weighted macro averages over seven benchmarks, excluding MATH-test (owing to the limited rebuttal time, and because MATH-test scores are already high and have little effect on the conclusions), in percent; transfer gap is in percentage points. Base and GRPO denote the teacher's pre- and post-GRPO checkpoints.
Uniform OPD in the main table uses all training problems, whereas TS uses the retained training pool, so the table compares overall training recipes.