- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper examines whether improvements from on-policy distillation (OPD) persist as the test-time sampling budget increases. Across mathematics, code generation, and factual question answering, the authors observe both sustained pass@K improvements and cases where better pass@1 accompanies worse large-budget performance. They derive a sufficient condition for this reversal and construct a reverse-KL example in which it occurs despite the teacher being more accurate on every problem. The paper also proposes TAS@K, a pre-training teacher-selection score based on teachers’ relative likelihood preferences for correct versus incorrect student responses, weighted by the target sampling budget. TAS@K agrees with the observed distilled-student ordering in 148 of 177 benchmark-budget comparisons across four teacher pairs.
Critical strengths
1.The empirical study establishes that OPD can have materially different effects at different sampling budgets. The paper covers eight primary student–teacher configurations across three domains and examines several alternative OPD formulations. Some reversals are substantial: in the Qwen3-1.7B-Base setting on AIME2024, pass@1 increases by approximately 7.8 percentage points while pass@1024 decreases by 16.7 points. The inclusion of Code and Fact settings with persistent gains makes the conclusion appropriately qualified. This is consequential because it changes how improvements from OPD should be evaluated. 2.The reverse-KL counterexample gives a precise explanation of why teacher accuracy is insufficient. Appendix A.4 shows that a teacher with higher correctness probability can nevertheless induce a negative student accuracy derivative. Replacing the teacher’s distribution over incorrect responses, while preserving its accuracy, reverses that derivative. This is a useful distinction between teacher performance and the supervision actually delivered to the student. The authors also carefully separate finite-training behavior from exact teacher matching.
Critical weaknesses
1.The teacher-selection experiments do not establish an advantage over simple alternatives. The headline 83.6% agreement pools correlated budget points from only four teacher pairs. No comparison is provided against selecting by teacher pass@1, teacher pass@K, or the same preference score without budget weighting. Table 6 makes this omission consequential: selecting the teacher with higher teacher pass@K would identify the better distilled student in 33 of the 36 Code comparisons, whereas TAS@K succeeds in 28. The RL-Code teacher produces the better student in all 36 comparisons. These results leave open whether TAS@K provides useful additional predictive information, particularly about changes in teacher preference as K increases. 2.Reliability in the rare-success regime remains insufficiently tested. TAS@K has no directional signal when the sampled base responses contain no correct answer. Appendix A.6 acknowledges this limitation and reports usable correct/incorrect comparisons for only 213 of 800 Fact problems and 35 of 175 LiveCodeBench v6 problems. Such missing information matters at large K, where gains on initially difficult problems can determine the final ordering. Meanwhile, evaluation uses seed 0, with no reported repeated-run variability or uncertainty intervals. At K=n, the pass@K estimator becomes an indicator of whether any sampled response succeeds; on AIME, a 3.3-point endpoint difference corresponds to one problem. The larger reversals are convincing observations, but the stability of the finer teacher rankings and the pooled selection accuracy is harder to assess.
Questions for authors
The following questions concern evidence needed to assess the current claims. 1.Selection baselines: How does TAS@K compare across all four teacher pairs with selecting by teacher pass@1 or teacher pass@K, and with removing the budget weight from TAS? Please report results by teacher pair and the performance lost through incorrect selections, in addition to pooled agreement. The Code comparison above suggests these baselines could materially change the interpretation. 2.Sampling protocol and missing information: What response count N is used to compute TAS@K on each benchmark, and are the same base responses reused in the pass@K evaluation? How sensitive are the selections to resampling or reducing N? What fraction of the final student performance difference comes from problems that supply no TAS signal? 3.Uncertainty: Which reversals and changes in teacher ordering remain stable across independent sampling pools or training runs? How would the reported agreement change if statistically unresolved student orderings were treated as ties rather than counted as definite selection outcomes?
9 comments
Response to reviewer qPyg (Part 2/5)
Thank you for your insightful questions.
W1: These results leave open whether TAS@K provides useful additional predictive information, particularly about changes in teacher preference as K increases.
Following your suggestion, we examined the five teacher-benchmark combinations where the preferred teacher changes with . The preferred teacher is the one whose OPD-trained model has higher pass@K. The table compares this ordering with the choices made by teacher pass@K, UTAS@K, and TAS@K across budgets. Mean loss is averaged over all 11 evaluated budgets for each benchmark.
| Teacher pair | Benchmark | Preferred teacher | Teacher pass@K | UTAS@K | TAS@K | |
|---|---|---|---|---|---|---|
| Qwen-Math | AMC2023 | 1-64 | B | B | B | B |
| 128 | B | A | B | B | ||
| 256-1024 | A | A | B | B | ||
| Agreement | 10/11 | 8/11 | 8/11 | |||
| Mean loss (pp) | 0.012 | 0.527 | 0.527 | |||
| Qwen-Math | AIME2026 | 1-512 | B | B | B | B |
| 1024 | A | B | B | B | ||
| Agreement | 10/11 | 10/11 | 10/11 | |||
| Mean loss (pp) | 0.303 | 0.303 | 0.303 | |||
| DeepSeek-Math | AIME2024 | 1-256 | B | A | B | B |
| 512-1024 | A | A | B | B | ||
| Agreement | 2/11 | 9/11 | 9/11 | |||
| Mean loss (pp) | 4.965 | 0.436 | 0.436 | |||
| Qwen-Fact | HotpotQA | 1-8 | B | B | A | A |
| 16-128 | A | B | A | A | ||
| 256-512 | B | B | A | A | ||
| 1024 | B | B | A | B | ||
| Agreement | 7/11 | 4/11 | 5/11 | |||
| Mean loss (pp) | 0.130 | 0.308 | 0.217 | |||
| Qwen-Fact | TriviaQA | 1 | B | B | B | B |
| 2 | A | B | B | B | ||
| 4 | A | B | B | A | ||
| 8-128 | A | A | B | A | ||
| 256-1024 | A | A | B | B | ||
| Agreement | 9/11 | 1/11 | 7/11 | |||
| Mean loss (pp) | 0.122 | 1.247 | 0.406 | |||
| Overall agreement | 38/55 | 32/55 | 39/55 | |||
| Overall mean loss (pp) | 1.106 | 0.564 | 0.378 |
A/B denote Qwen3-8B / RL-Math for Qwen-Math, Skywork / JustRL for DeepSeek-Math, and Binary-RAR / SESA for Qwen-Fact.
As shown in the table, TAS@K matches 39/55 student rankings, compared with 32/55 for UTAS@K and 38/55 for teacher pass@K, with the lowest mean selection loss of 0.378 pp. For example, on HotpotQA, the preferred teacher changes from A at to B at . TAS@K selects A and B at these respective budgets, whereas UTAS@K selects A at both. These results show that TAS@K provides useful additional predictive information about budget-dependent teacher preferences in this subset. We will add these budget-wise teacher-choice and selection-loss comparisons to the updated manuscript.
Q3: Uncertainty: Which reversals and changes in teacher ordering remain stable across independent sampling pools or training runs?
To test teacher selection under another training condition, we trained two additional DeepSeek-Math students on Hendrycks MATH and evaluated the same TAS@K teacher choices against their pass@K ordering. The pre-OPD base model and the two teachers, Skywork and JustRL, remain unchanged. The table compares TAS@K agreement (%) [95% CI] under the two training datasets, with 11 budgets per benchmark.
| Benchmark | DAPO-Math-17K | Hendrycks MATH |
|---|---|---|
| AMC2023 | 100.0 [63.6, 100.0] | 100.0 [63.6, 100.0] |
| AIME2024 | 81.8 [36.4, 90.9] | 90.9 [54.5, 100.0] |
| AIME2025 | 90.9 [81.8, 100.0] | 90.9 [90.9, 100.0] |
| AIME2026 | 81.8 [54.5, 100.0] | 81.8 [54.5, 100.0] |
| Overall | 88.6 [70.5, 93.2] | 90.9 [77.3, 97.7] |
TAS@K matches 40/44 student rankings after changing the training dataset, compared with 39/44 under DAPO-Math-17K. All six teacher-benchmark combinations showing higher pass@1 but lower pass@1024 under DAPO-Math-17K retain this reversal under Hendrycks MATH, with one additional combination exhibiting a reversal. We will add this training-data comparison to the updated manuscript.
Response to reviewer qPyg (Part 3/5)
Thank you for your constructive comments.
Q2: Sampling protocol and missing information: What response count N is used to compute TAS@K on each benchmark, and are the same base responses reused in the pass@K evaluation?
We agree that the response count, response reuse, and sensitivity to sampling should be made explicit. We would like to clarify that is the number of responses sampled per problem from the pre-OPD base model, while is the target sampling budget. In our evaluation, for each benchmark. We use all responses to estimate the correctness probability and teacher preference gap at every target , changing only the budget weight.
| Domain | Benchmark | ||
|---|---|---|---|
| Math | AMC2023, AIME2024, AIME2025, AIME2026 | 1024 | 1024 |
| Math | MATH500 | 256 | 256 |
| Code | HumanEval+, MBPP+ | 512 | 512 |
| Code | LiveCodeBench v5, LiveCodeBench v6-only | 128 | 128 |
| Fact | HotpotQA, TriviaQA, PopQA, 2WikiMultiHopQA | 1024 | 1024 |
The same Base response pool is used to compute TAS@K and estimate the pre-OPD base model's pass@K. Each OPD-trained model is evaluated using its own response pool.
Q2: How sensitive are the selections to resampling or reducing N?
To assess this sensitivity, we recomputed TAS@K using smaller subsets of Base responses and full- resampling, with 1,000 repetitions per design. Smaller subsets are sampled without replacement, while full- samples are drawn with replacement. Both teachers use the same sampled responses. The target budgets and student evaluation results remain fixed. Entries show mean selection agreement (%) [95% resampling interval] across all 177 comparisons. The last column reports changes from the original full- result.
| Sampling design | Agreement [95% interval] | Change |
|---|---|---|
| Original full | 83.6 | 0.0 pp |
| 80.8 [75.1, 85.9] | -2.8 pp | |
| 78.8 [71.2, 85.9] | -4.8 pp | |
| 76.9 [68.1, 84.7] | -6.7 pp | |
| Full- bootstrap | 81.1 [75.7, 85.9] | -2.5 pp |
As shown in the table, TAS@K retains 80.8% mean agreement using half the Base responses, 2.8 pp below the full- result, and 76.9% using one eighth. Full- resampling yields 81.1% mean agreement. These results indicate that TAS@K remains informative with smaller response pools, while using the full pool improves mean agreement. Reducing affects both the estimated correctness probability and the teacher preference gap. Since the full response pools are available, we use all responses for the reported teacher choices. These intervals describe variation within the existing response pools. We will add the benchmark-level response counts, clarify response-pool reuse, and report these sampling-sensitivity results in the updated manuscript.
Response to reviewer qPyg (Part 1/5)
Thank you for recognizing our findings across sampling budgets and our explanation of why teacher accuracy alone is insufficient. We show that a more accurate teacher does not necessarily produce a better distilled student, and investigate how to select teachers according to the student's target sampling budget.
We agree that comparisons with simple teacher-selection rules and an unweighted preference score are useful for assessing the additional value of TAS@K. Following your suggestion, we compared TAS@K with random selection, teacher pass@1, teacher pass@K, and unweighted TAS (UTAS@K). UTAS@K uses the same per-problem preferences and denominator as TAS@K, removing only the budget weight. Therefore, its teacher choice does not vary with K.
Entries show selection agreement (%) [95% CI] using 10,000 paired problem-bootstrap samples. We draw problems with replacement within each benchmark and use the same sampled problems at every K. We then recompute both OPD-trained models' pass@K and evaluate the original teacher choices.
As shown in the table, TAS@K achieves the highest overall agreement at 83.6%, compared with 81.4% for UTAS@K and 59.6% for teacher pass@K. TAS@K is particularly useful in DeepSeek-Math and Qwen-Fact, where it outperforms selecting by teacher pass@1 or pass@K. Budget weighting also improves agreement over UTAS@K in Qwen-Code (77.8% vs. 72.2%) and Qwen-Fact (72.7% vs. 61.4%). Teacher pass@K performs better in Qwen-Math and Qwen-Code, so the advantage depends on the teacher pair. We will add these per-pair and overall comparisons, including the budget-weight ablation, to the updated manuscript.
For each comparison, we define
L(K)=maxSA(K),SB(K)−Sselected(K),
where SA(K) and SB(K) are the two OPD-trained models' pass@K, and Sselected(K) is the pass@K of the model distilled from the selected teacher. The table reports the mean of L(K) over all comparisons in percentage points.
As shown in the table, TAS@K has the lowest overall mean selection loss at 0.498 percentage points, compared with 0.700 for UTAS@K, 1.632 for teacher pass@K, and 2.130 for teacher pass@1. In DeepSeek-Math, its mean loss is 0.303 pp, compared with 4.908 pp for teacher pass@K. Budget weighting also lowers mean loss relative to UTAS@K in Qwen-Code and Qwen-Fact. Thus, the overall advantage of TAS@K extends from selection agreement to the student performance retained after selection. We will add these per-pair and overall selection-loss comparisons to the updated manuscript.