acceptodds
Under review as a conference paper at ICLR 2027

Towards Understanding On-Policy Distillation through the Lens of Test-Time Scaling

41 views

Abstract

On-policy distillation (OPD) has emerged as a promising post-training technique for enhancing LLM reasoning. Under the reverse KL objective, the idealized optimum of OPD aligns the student distribution with that of the teacher. When the teacher consistently outperforms the student, this naturally suggests that OPD should yield broad improvements over the pre-OPD student. However, do such improvements extend across the entire range of test-time sampling budgets? In this work, we revisit this expectation through the lens of test-time scaling by varying the sampling budget and evaluating performance with pass@. Across multiple settings, we observe two distinct patterns: OPD can improve pass@ at both small and large sampling budgets, but it can also improve small-budget performance while reducing large-budget pass@. We show one condition that guarantees such a reversal and an idealized reverse KL counterexample where it occurs even when the teacher has higher accuracy on every problem. To choose between two candidate teachers at a target sampling budget, we propose the Teacher Advantage Score at (TAS@), which can be computed before OPD training to predict which teacher will lead to a larger improvement in pass@. Across three domains and thirteen benchmarks, the ordering predicted by TAS@ agrees with the observed pass@ improvements of the resulting OPD models in 83.6% of experiments, providing a useful signal for teacher selection at the target pass@.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Test-Time Scaling, Large Language Model Reasoning, Knowledge Distillation, Teacher Selection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qPygOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines whether improvements from on-policy distillation (OPD) persist as the test-time sampling budget increases. Across mathematics, code generation, and factual question answering, the authors observe both sustained pass@K improvements and cases where better pass@1 accompanies worse large-budget performance. They derive a sufficient condition for this reversal and construct a reverse-KL example in which it occurs despite the teacher being more accurate on every problem. The paper also proposes TAS@K, a pre-training teacher-selection score based on teachers’ relative likelihood preferences for correct versus incorrect student responses, weighted by the target sampling budget. TAS@K agrees with the observed distilled-student ordering in 148 of 177 benchmark-budget comparisons across four teacher pairs.

Critical strengths

1.The empirical study establishes that OPD can have materially different effects at different sampling budgets. The paper covers eight primary student–teacher configurations across three domains and examines several alternative OPD formulations. Some reversals are substantial: in the Qwen3-1.7B-Base setting on AIME2024, pass@1 increases by approximately 7.8 percentage points while pass@1024 decreases by 16.7 points. The inclusion of Code and Fact settings with persistent gains makes the conclusion appropriately qualified. This is consequential because it changes how improvements from OPD should be evaluated. 2.The reverse-KL counterexample gives a precise explanation of why teacher accuracy is insufficient. Appendix A.4 shows that a teacher with higher correctness probability can nevertheless induce a negative student accuracy derivative. Replacing the teacher’s distribution over incorrect responses, while preserving its accuracy, reverses that derivative. This is a useful distinction between teacher performance and the supervision actually delivered to the student. The authors also carefully separate finite-training behavior from exact teacher matching.

Critical weaknesses

1.The teacher-selection experiments do not establish an advantage over simple alternatives. The headline 83.6% agreement pools correlated budget points from only four teacher pairs. No comparison is provided against selecting by teacher pass@1, teacher pass@K, or the same preference score without budget weighting. Table 6 makes this omission consequential: selecting the teacher with higher teacher pass@K would identify the better distilled student in 33 of the 36 Code comparisons, whereas TAS@K succeeds in 28. The RL-Code teacher produces the better student in all 36 comparisons. These results leave open whether TAS@K provides useful additional predictive information, particularly about changes in teacher preference as K increases. 2.Reliability in the rare-success regime remains insufficiently tested. TAS@K has no directional signal when the sampled base responses contain no correct answer. Appendix A.6 acknowledges this limitation and reports usable correct/incorrect comparisons for only 213 of 800 Fact problems and 35 of 175 LiveCodeBench v6 problems. Such missing information matters at large K, where gains on initially difficult problems can determine the final ordering. Meanwhile, evaluation uses seed 0, with no reported repeated-run variability or uncertainty intervals. At K=n, the pass@K estimator becomes an indicator of whether any sampled response succeeds; on AIME, a 3.3-point endpoint difference corresponds to one problem. The larger reversals are convincing observations, but the stability of the finer teacher rankings and the pooled selection accuracy is harder to assess.

Questions for authors

The following questions concern evidence needed to assess the current claims. 1.Selection baselines: How does TAS@K compare across all four teacher pairs with selecting by teacher pass@1 or teacher pass@K, and with removing the budget weight from TAS? Please report results by teacher pair and the performance lost through incorrect selections, in addition to pooled agreement. The Code comparison above suggests these baselines could materially change the interpretation. 2.Sampling protocol and missing information: What response count N is used to compute TAS@K on each benchmark, and are the same base responses reused in the pass@K evaluation? How sensitive are the selections to resampling or reducing N? What fraction of the final student performance difference comes from problems that supply no TAS signal? 3.Uncertainty: Which reversals and changes in teacher ordering remain stable across independent sampling pools or training runs? How would the reported agreement change if statistically unresolved student orderings were treated as ties rather than counted as definite selection outcomes?

9 comments
AuthorsOpenReview ↗

Response to reviewer qPyg (Part 1/5)

Thank you for recognizing our findings across sampling budgets and our explanation of why teacher accuracy alone is insufficient. We show that a more accurate teacher does not necessarily produce a better distilled student, and investigate how to select teachers according to the student's target sampling budget.

W1: The teacher-selection experiments do not establish an advantage over simple alternatives. The headline 83.6% agreement pools correlated budget points from only four teacher pairs. No comparison is provided against selecting by teacher pass@1, teacher pass@K, or the same preference score without budget weighting. Table 6 makes this omission consequential: selecting the teacher with higher teacher pass@K would identify the better distilled student in 33 of the 36 Code comparisons, whereas TAS@K succeeds in 28. The RL-Code teacher produces the better student in all 36 comparisons.

Q1: Selection baselines: How does TAS@K compare across all four teacher pairs with selecting by teacher pass@1 or teacher pass@K, and with removing the budget weight from TAS?

We agree that comparisons with simple teacher-selection rules and an unweighted preference score are useful for assessing the additional value of TAS@K. Following your suggestion, we compared TAS@K with random selection, teacher pass@1, teacher pass@K, and unweighted TAS (UTAS@K). UTAS@K uses the same per-problem preferences and denominator as TAS@K, removing only the budget weight. Therefore, its teacher choice does not vary with KK.

Entries show selection agreement (%) [95% CI] using 10,000 paired problem-bootstrap samples. We draw problems with replacement within each benchmark and use the same sampled problems at every KK. We then recompute both OPD-trained models' pass@K and evaluate the original teacher choices.

Teacher pairRandomTeacher pass@1Teacher pass@KUTAS@KTAS@K
Qwen-Math50.090.9 [81.8, 97.7]95.5 [84.1, 100.0]90.9 [81.8, 97.7]90.9 [81.8, 97.7]
Qwen-Code50.072.2 [69.4, 75.0]91.7 [88.9, 94.4]72.2 [69.4, 72.2]77.8 [77.8, 80.6]
DeepSeek-Math50.03.8 [0.0, 20.8]4.7 [0.0, 21.7]96.2 [75.5, 98.1]90.6 [73.6, 94.3]
Qwen-Fact50.043.2 [25.0, 63.6]63.6 [38.6, 75.0]61.4 [43.2, 84.1]72.7 [54.5, 86.4]
Overall50.049.2 [44.1, 57.1]59.6 [52.3, 64.7]81.4 [73.4, 85.9]83.6 [75.7, 86.4]

As shown in the table, TAS@K achieves the highest overall agreement at 83.6%, compared with 81.4% for UTAS@K and 59.6% for teacher pass@K. TAS@K is particularly useful in DeepSeek-Math and Qwen-Fact, where it outperforms selecting by teacher pass@1 or pass@K. Budget weighting also improves agreement over UTAS@K in Qwen-Code (77.8% vs. 72.2%) and Qwen-Fact (72.7% vs. 61.4%). Teacher pass@K performs better in Qwen-Math and Qwen-Code, so the advantage depends on the teacher pair. We will add these per-pair and overall comparisons, including the budget-weight ablation, to the updated manuscript.

Q1: Please report results by teacher pair and the performance lost through incorrect selections, in addition to pooled agreement. The Code comparison above suggests these baselines could materially change the interpretation.

For each comparison, we define

L(K)=max⁡SA(K),SB(K)−Sselected(K),L(K)=\max\\{S_A(K),S_B(K)\\}-S_{\mathrm{selected}}(K),

where SA(K)S_A(K) and SB(K)S_B(K) are the two OPD-trained models' pass@K, and Sselected(K)S_{\mathrm{selected}}(K) is the pass@K of the model distilled from the selected teacher. The table reports the mean of L(K)L(K) over all comparisons in percentage points.

Teacher pairRandomTeacher pass@1Teacher pass@KUTAS@KTAS@K
Qwen-Math8.6540.2080.0790.2080.208
Qwen-Code4.3792.0810.2422.5791.468
DeepSeek-Math2.5114.9324.9080.0900.303
Qwen-Fact0.6520.7150.3770.3890.230
Overall3.9562.1301.6320.7000.498

As shown in the table, TAS@K has the lowest overall mean selection loss at 0.498 percentage points, compared with 0.700 for UTAS@K, 1.632 for teacher pass@K, and 2.130 for teacher pass@1. In DeepSeek-Math, its mean loss is 0.303 pp, compared with 4.908 pp for teacher pass@K. Budget weighting also lowers mean loss relative to UTAS@K in Qwen-Code and Qwen-Fact. Thus, the overall advantage of TAS@K extends from selection agreement to the student performance retained after selection. We will add these per-pair and overall selection-loss comparisons to the updated manuscript.

AuthorsOpenReview ↗

Response to reviewer qPyg (Part 2/5)

Thank you for your insightful questions.

W1: These results leave open whether TAS@K provides useful additional predictive information, particularly about changes in teacher preference as K increases.

Following your suggestion, we examined the five teacher-benchmark combinations where the preferred teacher changes with KK. The preferred teacher is the one whose OPD-trained model has higher pass@K. The table compares this ordering with the choices made by teacher pass@K, UTAS@K, and TAS@K across budgets. Mean loss is averaged over all 11 evaluated budgets for each benchmark.

Teacher pairBenchmarkKKPreferred teacherTeacher pass@KUTAS@KTAS@K
Qwen-MathAMC20231-64BBBB
128BABB
256-1024AABB
Agreement10/118/118/11
Mean loss (pp)0.0120.5270.527
Qwen-MathAIME20261-512BBBB
1024ABBB
Agreement10/1110/1110/11
Mean loss (pp)0.3030.3030.303
DeepSeek-MathAIME20241-256BABB
512-1024AABB
Agreement2/119/119/11
Mean loss (pp)4.9650.4360.436
Qwen-FactHotpotQA1-8BBAA
16-128ABAA
256-512BBAA
1024BBAB
Agreement7/114/115/11
Mean loss (pp)0.1300.3080.217
Qwen-FactTriviaQA1BBBB
2ABBB
4ABBA
8-128AABA
256-1024AABB
Agreement9/111/117/11
Mean loss (pp)0.1221.2470.406
Overall agreement38/5532/5539/55
Overall mean loss (pp)1.1060.5640.378

A/B denote Qwen3-8B / RL-Math for Qwen-Math, Skywork / JustRL for DeepSeek-Math, and Binary-RAR / SESA for Qwen-Fact.

As shown in the table, TAS@K matches 39/55 student rankings, compared with 32/55 for UTAS@K and 38/55 for teacher pass@K, with the lowest mean selection loss of 0.378 pp. For example, on HotpotQA, the preferred teacher changes from A at K=128K=128 to B at K=1024K=1024. TAS@K selects A and B at these respective budgets, whereas UTAS@K selects A at both. These results show that TAS@K provides useful additional predictive information about budget-dependent teacher preferences in this subset. We will add these budget-wise teacher-choice and selection-loss comparisons to the updated manuscript.

Q3: Uncertainty: Which reversals and changes in teacher ordering remain stable across independent sampling pools or training runs?

To test teacher selection under another training condition, we trained two additional DeepSeek-Math students on Hendrycks MATH and evaluated the same TAS@K teacher choices against their pass@K ordering. The pre-OPD base model and the two teachers, Skywork and JustRL, remain unchanged. The table compares TAS@K agreement (%) [95% CI] under the two training datasets, with 11 budgets per benchmark.

BenchmarkDAPO-Math-17KHendrycks MATH
AMC2023100.0 [63.6, 100.0]100.0 [63.6, 100.0]
AIME202481.8 [36.4, 90.9]90.9 [54.5, 100.0]
AIME202590.9 [81.8, 100.0]90.9 [90.9, 100.0]
AIME202681.8 [54.5, 100.0]81.8 [54.5, 100.0]
Overall88.6 [70.5, 93.2]90.9 [77.3, 97.7]

TAS@K matches 40/44 student rankings after changing the training dataset, compared with 39/44 under DAPO-Math-17K. All six teacher-benchmark combinations showing higher pass@1 but lower pass@1024 under DAPO-Math-17K retain this reversal under Hendrycks MATH, with one additional combination exhibiting a reversal. We will add this training-data comparison to the updated manuscript.

AuthorsOpenReview ↗

Response to reviewer qPyg (Part 3/5)

Thank you for your constructive comments.

Q2: Sampling protocol and missing information: What response count N is used to compute TAS@K on each benchmark, and are the same base responses reused in the pass@K evaluation?

We agree that the response count, response reuse, and sensitivity to sampling should be made explicit. We would like to clarify that NN is the number of responses sampled per problem from the pre-OPD base model, while KK is the target sampling budget. In our evaluation, N=max⁡KN=\max K for each benchmark. We use all NN responses to estimate the correctness probability and teacher preference gap at every target KK, changing only the budget weight.

DomainBenchmarkNNmax⁡K\max K
MathAMC2023, AIME2024, AIME2025, AIME202610241024
MathMATH500256256
CodeHumanEval+, MBPP+512512
CodeLiveCodeBench v5, LiveCodeBench v6-only128128
FactHotpotQA, TriviaQA, PopQA, 2WikiMultiHopQA10241024

The same Base response pool is used to compute TAS@K and estimate the pre-OPD base model's pass@K. Each OPD-trained model is evaluated using its own response pool.

Q2: How sensitive are the selections to resampling or reducing N?

To assess this sensitivity, we recomputed TAS@K using smaller subsets of Base responses and full-NN resampling, with 1,000 repetitions per design. Smaller subsets are sampled without replacement, while full-NN samples are drawn with replacement. Both teachers use the same sampled responses. The target budgets and student evaluation results remain fixed. Entries show mean selection agreement (%) [95% resampling interval] across all 177 comparisons. The last column reports changes from the original full-NN result.

Sampling designAgreement [95% interval]Change
Original full NN83.60.0 pp
N/2N/280.8 [75.1, 85.9]-2.8 pp
N/4N/478.8 [71.2, 85.9]-4.8 pp
N/8N/876.9 [68.1, 84.7]-6.7 pp
Full-NN bootstrap81.1 [75.7, 85.9]-2.5 pp

As shown in the table, TAS@K retains 80.8% mean agreement using half the Base responses, 2.8 pp below the full-NN result, and 76.9% using one eighth. Full-NN resampling yields 81.1% mean agreement. These results indicate that TAS@K remains informative with smaller response pools, while using the full pool improves mean agreement. Reducing NN affects both the estimated correctness probability and the teacher preference gap. Since the full response pools are available, we use all NN responses for the reported teacher choices. These intervals describe variation within the existing response pools. We will add the benchmark-level response counts, clarify response-pool reuse, and report these sampling-sensitivity results in the updated manuscript.

Reviewer_SoMAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies on-policy distillation under different test-time sampling budgets. Empirically, across Math, Code, and Fact settings, the authors observed both persistent gains and cases where small-budget gains get reversed into large-budget degradation. With this observation, the authors showed that from a dataset perspective, higher pass@1 does not necessarily lead to higher pass@K, due to the fact that budget-dependent sensitivity upweights low-accuracy problems. Based on this, this paper further proposes a metric TAS@K, which compares two teachers and decides which one is more suitable as the distillation target at a given target budget K. Under controlled experiments, the authors demonstrated that the prediction from TAS@K agrees with the observed pass@K trends in 83.6% of the experiments, indicating that TAS@K serves as a useful metric for budget-aware teacher selection.

Critical strengths

The raised question deserves investigation. Today's OPD approaches typically evaluate distilled models with pass@1 only, while pass@K performances are largely neglected. However, as practitioners increasingly rely on repeated sampling at test time (especially in math), conclusions drawn from pass@1 alone may be misleading if the gains do not persist at larger budgets.

The question is well-motivated by both theoretical characterizations and empirical observations. The authors provide a construction in which pass@1 and pass@K show reversed trends, showing that a stronger teacher does not guarantee improvements across all budgets. The theory is also backed with empirical observations on Math datasets, where several settings exhibit precisely the predicted pattern.

Critical weaknesses
  1. Table 2 suggests that the degradation at higher sampling budgets only occurs in the math domain, but not in the others (code/fact). It is unclear when such degradation may occur, whether this phenomenon is tied to certain choices of datasets, and whether this is common among other domains. This significantly limits the method's applicability.

  2. Although the proposed TAS@K is indicative, it should be compared with simple baseline metrics, such as using the expert with higher pass@K on the target benchmark. The current correlation (83.6%) is hard to contextualize without proper comparison.

Questions for authors
  1. I am wondering how common the observed degradation really is. For example, could the authors explain why the degradation only happens on math tasks, but not the code and fact tasks? Besides, since all math tasks are using the same dataset for training, does the observed degradation still persist if we use another training dataset?

  2. General RLVR also suffers from similar phenomon (i.e. increased pass@1 but decreased pass@K), and the common explanation is that RL sacrifices response diversity. How is the degradation in OPD related to the degradation in RL? Are they revealing the same phenomenon or they are caused by difference mechanisms?

9 comments
AuthorsOpenReview ↗

Response to reviewer SoMA (Part 1/2)

Thank you for recognizing the importance of looking beyond pass@1 and the support provided by our theory and experiments. We examine both cases where OPD gains persist and cases where they reverse at larger budgets, using experiments in Math, Code, and Fact.

W1: Table 2 suggests that the degradation at higher sampling budgets only occurs in the math domain, but not in the others (code/fact). It is unclear when such degradation may occur, whether this phenomenon is tied to certain choices of datasets, and whether this is common among other domains. This significantly limits the method's applicability.

Q1: I am wondering how common the observed degradation really is. For example, could the authors explain why the degradation only happens on math tasks, but not the code and fact tasks? Besides, since all math tasks are using the same dataset for training, does the observed degradation still persist if we use another training dataset?

We agree that it is important to examine whether the observed degradation persists under different training datasets. In our experiments, budget-dependent reversals occur mainly in Math, while Code and Fact generally retain gains as the sampling budget increases. Our analysis explains how changes in problem-level correctness affect pass@K, but does not establish why these patterns differ across domains.

Following your suggestion, we trained DeepSeek-R1-Distill-Qwen-1.5B on Hendrycks MATH, another widely used math training dataset, using Skywork-OR1-Math-7B and JustRL-DS-1.5B as teachers. Entries report Hendrycks-trained pass@K (%) at K=1,4,16,64,256,1024K=1,4,16,64,256,1024. The last two columns indicate whether each training setting yields higher pass@1 but lower pass@1024 than the pre-OPD base model.

TeacherBenchmarkHendrycks pass@1 (%)Hendrycks pass@4 (%)Hendrycks pass@16 (%)Hendrycks pass@64 (%)Hendrycks pass@256 (%)Hendrycks pass@1024 (%)Reversal: DAPOReversal: Hendrycks
SkyworkAMC202373.7990.5194.9895.3096.0997.50YesYes
SkyworkAIME202435.0056.0770.5075.8679.4883.33NoYes
SkyworkAIME202527.2937.1246.1355.3965.0773.33YesYes
SkyworkAIME202624.0740.0955.5167.8876.6083.33YesYes
JustRLAMC202386.8494.8896.2198.4099.97100.00NoNo
JustRLAIME202447.7867.0277.0080.6482.2883.33YesYes
JustRLAIME202534.9046.8157.7164.0769.2073.33YesYes
JustRLAIME202634.2051.1363.6973.7080.0783.33YesYes

Reversals occur in six of eight teacher-benchmark combinations with DAPO-Math-17K and seven with Hendrycks MATH. These results show that the observed degradation still persists when we use another training dataset. We will add this training-dataset comparison and clarify the scope of the observed domain differences in the updated manuscript.

W2: Although the proposed TAS@K is indicative, it should be compared with simple baseline metrics, such as using the expert with higher pass@K on the target benchmark. The current correlation (83.6%) is hard to contextualize without proper comparison.

We agree that comparison with simple teacher-selection baselines is important for interpreting the reported agreement. Following your suggestion, we compared TAS@K with random selection and selecting the teacher with higher pass@1 or pass@K on the target benchmark. Agreement is measured against the pass@K ordering of the two OPD-trained models.

Teacher pairComparisonsRandomTeacher pass@1Teacher pass@KTAS@K
Qwen-Math4450.0%90.9%95.5%90.9%
Qwen-Code3650.0%72.2%91.7%77.8%
DeepSeek-Math5350.0%3.8%4.7%90.6%
Qwen-Fact4450.0%43.2%63.6%72.7%
Overall17750.0%49.2%59.6%83.6%

Teacher pass@K yields higher agreement than TAS@K in Qwen-Math and Qwen-Code, whereas TAS@K performs better in DeepSeek-Math and Qwen-Fact. Overall, TAS@K achieves 83.6% agreement, compared with 49.2% for teacher pass@1 and 59.6% for teacher pass@K. We will add the comparison of random selection, teacher pass@1, teacher pass@K, and TAS@K, with agreement reported for each teacher pair and overall, to the updated manuscript.

AuthorsOpenReview ↗

Response to reviewer SoMA (Part 2/2)

Thank you for your insightful comments.

Q2: General RLVR also suffers from similar phenomon (i.e. increased pass@1 but decreased pass@K), and the common explanation is that RL sacrifices response diversity. How is the degradation in OPD related to the degradation in RL? Are they revealing the same phenomenon or they are caused by difference mechanisms?

As you point out, reduced response diversity has been proposed as an explanation for the budget-dependent reversal observed in RLVR. Following your suggestion, we compared response diversity before and after OPD across three domains, six benchmarks, and 12 teacher-benchmark combinations using Self-BLEU-4.

Self-BLEU-4 measures how much each response overlaps with the other responses to the same problem in sequences of one to four words, averaged across responses. For each problem, we average this score over three paired draws of 16 responses per model. Changes are OPD minus the pre-OPD base model. Lower Self-BLEU indicates higher lexical diversity: a negative Δ\DeltaSelf-BLEU means diversity increased, while a positive value means it decreased. The table reports pass@1 and pass@K changes (pp) and mean Self-BLEU changes [95% CI]. Here K=1024K=1024 for Math and Fact, and K=512K=512 for Code.

DomainTeacherBenchmarkΔ\Deltapass@1Δ\Deltapass@KΔ\DeltaSelf-BLEU [95% CI]
MathSkyworkAMC2023+3.73-2.50+0.0494 [+0.0416, +0.0570]
MathSkyworkAIME2025+5.06-6.67+0.0650 [+0.0517, +0.0763]
MathJustRLAMC2023+15.360.00+0.0509 [+0.0401, +0.0625]
MathJustRLAIME2025+11.30-3.33+0.0777 [+0.0653, +0.0895]
CodeQwen3-8BHumanEval++2.70+7.93-0.0056 [-0.0101, -0.0012]
CodeQwen3-8BMBPP++1.04+6.61-0.0357 [-0.0404, -0.0313]
CodeRL-CodeHumanEval++5.80+18.29-0.1090 [-0.1170, -0.1009]
CodeRL-CodeMBPP++2.26+18.78-0.1035 [-0.1091, -0.0977]
FactBinary-RARHotpotQA-0.34+7.00-0.1093 [-0.1215, -0.0972]
FactBinary-RARTriviaQA+1.22+7.50-0.1215 [-0.1334, -0.1096]
FactSESAHotpotQA-0.003+8.00-0.0905 [-0.1052, -0.0762]
FactSESATriviaQA+1.27+5.50-0.0757 [-0.0883, -0.0638]

These results are consistent with the diversity-based explanation that concentrating probability on a narrower set of responses may improve pass@1 while reducing the chance of sampling some correct responses at larger KK. In our experiments, the three Math reversal cases show reduced lexical diversity, whereas all eight Code and Fact cases show increased mean lexical diversity and large-KK gains. JustRL on AMC2023 also shows reduced diversity but unchanged pass@1024, so reduced lexical diversity does not uniquely identify a reversal.

These results are consistent with a diversity-based account of large-budget degradation in OPD, as proposed for RLVR. Our analysis complements this account by explaining why gains in pass@1 can coexist with losses in pass@K at larger budgets. Whether OPD and RLVR share the same underlying causal mechanism remains open. We will add this diversity analysis and its connection to our theoretical analysis to the updated manuscript.

AuthorsOpenReview ↗

Kind reminder: updated manuscript and additional training-data experiments

Thank you again for your time and constructive suggestions.

We have updated the manuscript. All new and revised text is marked in purple, and the captions of revised tables and figures are also marked in purple.

Following up on our earlier Hendrycks MATH results, we have extended the training-data comparison across all three domains, including APPS for Code and TriviaQA for Fact. Math exhibits large-budget reversals in 10/16 teacher-benchmark combinations with DAPO-Math-17K and 12/16 with Hendrycks MATH. Code and Fact retain large-budget gains in all eight combinations per domain with both training datasets. These results are included in Section 4.2, Figures 2–4, and Table 3, with training details in Appendix C.3 and numerical results in Appendix D.1.

We have also added the teacher-selection baseline comparisons in Section 5.2 and Table 4, and the response-diversity analysis in Section 6 and Appendix I.

We would be happy to discuss any further questions or suggestions.

Authors of Submission 29401

Reviewer_nmesOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper offers an explanation for a phenomenon observed in OPD practice: training can sometimes improve pass@1 while decreasing pass@K at larger sampling budgets.

The authors further propose TAS@K, a metric designed to predict which teacher is likely to produce a student with higher pass@K after OPD.

Critical strengths
  1. The explanation of why OPD can improve pass@1 but hurt pass@K is intuitive and makes sense. It also gives a clear motivation for TAS@K.
  2. The experiments are fairly thorough, covering different domains, OPD methods, and sampling budgets.
Critical weaknesses
  1. I'm not fully convinced of the practical value of TAS@K. Based on Table 6, simply choosing the teacher with higher pass@K seems to work better in the Code setting (33/36 vs. 28/36).
  2. TAS@K also ignores problems where the base student gets no correct responses, even though a teacher might help the student learn to solve them. I think this could matter quite a lot for large-budget pass@K.
Questions for authors
  1. Have you tried comparing TAS@K with simply choosing the teacher with higher pass@1 or pass@K?
  2. For problems where the base student gets zero correct responses, how much do they contribute to the final pass@K difference between the two distilled students?
5 comments
AuthorsOpenReview ↗

Response to reviewer nmes (Part 1/1)

Thank you for recognizing our explanation of why OPD can improve pass@1 while reducing pass@K at larger budgets, as well as the breadth of our experiments. We study this behavior across different domains, OPD methods, and sampling budgets, and use these findings to guide teacher selection.

W1: I'm not fully convinced of the practical value of TAS@K. Based on Table 6, simply choosing the teacher with higher pass@K seems to work better in the Code setting (33/36 vs. 28/36).

Q1: Have you tried comparing TAS@K with simply choosing the teacher with higher pass@1 or pass@K?

We agree that teacher pass@1 and pass@K are natural baselines. Following your suggestions, we compared selecting the teacher with higher pass@1 or pass@K against random selection and TAS@K. Selection agreement measures how often each method selects the teacher whose OPD-trained model has higher pass@K.

Teacher pairComparisonsRandomTeacher pass@1Teacher pass@KTAS@K
Qwen-Math4450.0%90.9%95.5%90.9%
Qwen-Code3650.0%72.2%91.7%77.8%
DeepSeek-Math5350.0%3.8%4.7%90.6%
Qwen-Fact4450.0%43.2%63.6%72.7%
Overall17750.0%49.2%59.6%83.6%

Overall, TAS@K achieves higher agreement across the four teacher pairs, although teacher pass@K performs better in Qwen-Code, as you point out, and in Qwen-Math. In DeepSeek-Math, the teacher-performance baselines almost always select Skywork because it has higher pass@1 and pass@K. However, the student distilled from JustRL achieves higher pass@K in 51/53 comparisons. TAS@K still matches the student ordering in 48/53 comparisons (90.6%), compared with 3.8% and 4.7% for teacher pass@1 and pass@K. This result shows that TAS@K can guide teacher selection when the teacher-performance baselines favor the teacher that produces the weaker student.

Across the four teacher pairs, TAS@K achieves the highest overall agreement at 83.6%, compared with 49.2% for teacher pass@1 and 59.6% for teacher pass@K. We will add the comparison of random selection, teacher pass@1, teacher pass@K, and TAS@K, with agreement reported for each teacher pair and overall, to the updated manuscript.

W2: TAS@K also ignores problems where the base student gets no correct responses, even though a teacher might help the student learn to solve them. I think this could matter quite a lot for large-budget pass@K.

Q2: For problems where the base student gets zero correct responses, how much do they contribute to the final pass@K difference between the two distilled students?

We agree that problems with no correct responses from the pre-OPD base model provide no direct TAS@K preference signal, even though they may be solved after OPD. The same applies when all sampled responses are correct. As described in Appendix A.6, our estimator sets the preference gap for these problems to zero while retaining them in the full benchmark denominator. Following your suggestion, we report the number and proportion of problems with both correct and incorrect responses, which we call valid problems.

Teacher pairValid problems / totalPercentage
Qwen-Math103/13079.2%
Qwen-Code461/159728.9%
DeepSeek-Math447/63071.0%
Qwen-Fact213/80026.6%

Valid problems account for 79.2% and 71.0% of problems in Qwen-Math and DeepSeek-Math, respectively, with lower proportions in Qwen-Code and Qwen-Fact. We recomputed the two OPD-trained models' pass@K using only valid problems and report selection agreement with these rankings below. Parentheses show changes from the results using all problems, in percentage points.

Teacher pairTeacher pass@1Teacher pass@KTAS@K
Qwen-Math86.4% (-4.5)90.9% (-4.5)86.4% (-4.5)
Qwen-Code69.4% (-2.8)88.9% (-2.8)86.1% (+8.3)
DeepSeek-Math3.8% (0.0)4.7% (0.0)90.6% (0.0)
Qwen-Fact45.5% (+2.3)65.9% (+2.3)65.9% (-6.8)
Overall48.0% (-1.1)58.5% (-1.1)82.5% (-1.1)

TAS@K achieves 82.5% overall agreement on valid problems, compared with 48.0% for teacher pass@1 and 58.5% for teacher pass@K. Its overall agreement is 1.1 percentage points lower than when all problems are included. Thus, TAS@K remains a useful guide for teacher selection despite the lower signal coverage in some pairs, although the effect varies across pairs. Thank you for highlighting this point. In the updated manuscript, we will clarify the treatment of problems without a preference signal and add the coverage and subset-agreement results.

AuthorsOpenReview ↗

Kind reminder: updated manuscript

Thank you again for your time and valuable feedback.

We have updated the manuscript. All new and revised text is marked in purple, and the captions of revised tables and figures are also marked in purple.

The teacher-selection comparisons are now included in Section 5.2 and Table 4. They cover random selection, teacher pass@1, teacher pass@K, UTAS@K, and TAS@K, with selection agreement and mean selection loss reported for each teacher pair and overall. Appendix A.6 clarifies how problems without a preference signal are handled and reports signal coverage and selection agreement on valid problems.

We would be happy to discuss any further questions or suggestions.

Authors of Submission 29401

AuthorsOpenReview ↗

Looking forward to your reply

Dear Reviewer nmes,

We sincerely thank you for your time and effort in reviewing our work and for your constructive feedback.

Please let us know if you need any further information or if there are additional points you would like to discuss with us. We would be very happy to continue the discussion and answer any questions you may have.

Thank you very much!

Best regards,

Authors of Submission 29401

Comments on the paper

5 comments
AuthorsOpenReview ↗

Response to reviewer EWdU (Part 1/2)

Thank you for recognizing the importance of evaluating OPD at different sampling budgets and the explanation offered by our analysis. We examine why gains at small budgets may not persist at larger budgets, and develop TAS@K to help select a teacher for the intended sampling budget.

W1: The teacher-selection evaluation contains correlated comparisons. The 177 reported outcomes are drawn from four teacher pairs, with multiple sampling budgets evaluated on the same benchmarks and trained models. The 83.6% agreement is encouraging, but should not be interpreted as the result of 177 independent selection problems. It is difficult to judge reliability across genuinely new teacher-pair and benchmark combinations.

Q2: Can the authors report teacher-selection accuracy with uncertainty estimates that account for correlations among the different K values from the same trained model pair and benchmark?

We agree that results at different sampling budgets can be correlated. We would like to clarify that TAS@K is recomputed for each target budget: each problem's teacher preference gap is weighted by Kp^x(1−p^x)KK\hat p_x(1-\hat p_x)^K, where p^x\hat p_x is the estimated correctness probability of the pre-OPD base model on problem xx. The correctness estimates and preference gaps are computed from the same base responses and remain fixed across budgets, while the weights change with KK.

Following your suggestion, we computed 95% confidence intervals using 10,000 paired problem-bootstrap samples. For each sample, we draw problems with replacement within each benchmark and use the same sampled problems at every KK. We then recompute both students' pass@K and measure agreement with the original teacher choices. The table reports selection agreement (%) [95% CI] for each teacher pair and overall.

Teacher pairComparisonsRandomTeacher pass@1Teacher pass@KTAS@K
Qwen-Math4450.090.9 [81.8, 97.7]95.5 [84.1, 100.0]90.9 [81.8, 97.7]
Qwen-Code3650.072.2 [69.4, 75.0]91.7 [88.9, 94.4]77.8 [77.8, 80.6]
DeepSeek-Math5350.03.8 [0.0, 20.8]4.7 [0.0, 21.7]90.6 [73.6, 94.3]
Qwen-Fact4450.043.2 [25.0, 63.6]63.6 [38.6, 75.0]72.7 [54.5, 86.4]
Overall17750.049.2 [44.1, 57.1]59.6 [52.3, 64.7]83.6 [75.7, 86.4]

TAS@K achieves the highest overall agreement at 83.6% [75.7, 86.4], compared with 49.2% [44.1, 57.1] for teacher pass@1 and 59.6% [52.3, 64.7] for teacher pass@K. We will add the paired problem-bootstrap procedure and the per-pair and overall confidence intervals to the updated manuscript.

AuthorsOpenReview ↗

Response to reviewer EWdU (Part 2/2)

Thank you for your constructive suggestions.

W2: The practical generalization of TAS@K is not established. The score uses correctness labels from base-model responses on the target dataset. It is unclear whether selecting a teacher using a separate calibration set would produce the same benefits on held-out problems. This is relevant to the proposed use of TAS@K as a pre-training decision tool.

Q1: How well does TAS@K predict the preferred teacher when it is computed using a labeled calibration set and the resulting OPD models are evaluated on disjoint test problems?

We agree that teacher selection should also be evaluated on test problems separate from those used to compute the selection scores. Following your suggestion, we evaluated TAS@K using separate calibration and test benchmarks in two protocols.

First, we performed leave-one-benchmark-out calibration across all four teacher pairs. Within each pair, we held out each of its four benchmarks in turn and selected a teacher by averaging the TAS@K scores of the other three benchmarks with equal weight. We used the budgets supported by all four benchmarks in that pair. For each pair, the table reports correct selections across its held-out benchmarks and evaluated budgets.

Teacher pairCorrect / TotalTAS@K
Qwen-Math40/4490.9%
Qwen-Code30/3293.8%
DeepSeek-Math39/4488.6%
Qwen-Fact25/4456.8%
Overall134/16481.7%

Second, for DeepSeek-Math (Skywork and JustRL), we used AMC2023 and AIME2024-2026 as the calibration set (130 problems), averaging their TAS@K scores equally at each KK. We evaluated both DAPO-trained students on three additional test benchmarks: BRUMO2025, CMIMC2025, and HMMT-Feb2025 (100 problems disjoint from the calibration set). We then checked whether the teacher selected on the calibration set produced the student with higher pass@K on these test benchmarks. The table reports agreement (%) [95% CI] across 11 budgets per test benchmark, with Overall pooling 33 comparisons. The intervals use paired test-problem bootstrap samples with the calibration choices fixed.

Test benchmarkTAS@K
BRUMO202581.8 [54.5, 81.8]
CMIMC202581.8 [81.8, 90.9]
HMMT-Feb202581.8 [45.5, 90.9]
Overall81.8 [69.7, 84.8]

TAS@K achieves 81.7% overall agreement in leave-one-benchmark-out calibration and 81.8% on the additional DeepSeek-Math test benchmarks. We will add these calibration-test protocols and results to the updated manuscript.

AuthorsOpenReview ↗

Kind reminder: updated manuscript

Thank you again for your time and thoughtful feedback.

We have updated the manuscript. All new and revised text is marked in purple, and the captions of revised tables and figures are also marked in purple.

Section 5.3 and Table 5 now report teacher selection using leave-one-benchmark-out calibration and calibration on separate benchmarks, followed by evaluation on disjoint test problems. TAS@K achieves 81.7% agreement in the former protocol and 81.8% on the additional DeepSeek-Math test benchmarks. Appendices H.1 and H.3 describe the paired problem-bootstrap confidence intervals and the calibration-test protocols.

We would be happy to discuss any further questions or suggestions.

Authors of Submission 29401

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.