- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents an extensive empirical study of how prompt quantity, data sources, and selection strategies affect knowledge transfer in on-policy distillation (OPD). The authors investigate multiple teacher–student pairs and report several findings, including strong transfer with very few prompts, teacher-dependent prompt preferences, functional alignment despite different parameter updates, recovery from unfavorable transfer, and the competitiveness of random prompt selection.
Critical strengths
1.Important and timely research question. Understanding how training data affects knowledge transfer in OPD is a practically relevant problem, especially given the growing interest in data-efficient post-training.
2.Comprehensive empirical evaluation. The paper covers multiple teacher–student configurations, model sizes, data sources, and evaluation benchmarks, providing a broad empirical perspective on OPD.
3.Interesting observations on teacher-dependent prompt preferences. The controlled teacher-swap experiments demonstrate that the effectiveness of a prompt source depends on the teacher–student relationship, rather than solely on the properties of the prompts.
4.Useful analysis of transfer behavior and data selection. The functional alignment and recovery experiments provide additional insights into OPD behavior, while comparisons with random sampling offer practical guidance for prompt selection.
Critical weaknesses
1.Insufficient robustness of the headline prompt-efficiency result. The striking 960× comparison uses different data sources and relies on a particular four-prompt support. Since DAPO is also used to train the JustRL teacher, the result may reflect data compatibility rather than prompt efficiency alone. Additional random seeds and contamination checks are needed to establish its generality and rule out potential post-hoc selection effects.
2.Despite presenting five empirical takeaways, the paper reads more like a comprehensive technical report than a study with a clear central contribution. The observations remain largely descriptive, without establishing a generalizable principle or mechanism explaining which prompts lead to successful transfer.
3.The functional alignment analysis is interesting but primarily characterizes training outcomes rather than explaining their causes. In particular, the relationship between functional alignment, prompt effectiveness, and downstream performance remains unclear, limiting the explanatory and predictive value of these findings.
4.Several prompt-selection methods are evaluated on a single support, while random sampling uses multiple draws. Additionally, some recovery experiments involve unequal training budgets, making it difficult to isolate the effects of prompt selection from additional optimization.
Questions for authors
-
How consistent is the four-prompt result across different random seeds? To what extent does the 960× efficiency gap reflect prompt count rather than data-source compatibility?
-
Given that several findings align with existing intuitions about OPD, what fundamentally new and generalizable insights does this study provide beyond prior work?
-
Can the authors establish a clearer connection between functional alignment and downstream transfer performance to explain why certain prompts are more effective?
7 comments
Response to Reviewer bdZF (2/3)
W2 & Q2: The central contribution.
Thank you for the suggestion to state the central contribution more explicitly.
Our claim is that in on-policy distillation the value of a prompt is relational. A prompt determines where the student starts generating trajectories, and the states the student then visits determine which differences between the teacher's and the student's predictions training can reach at all. What matters for transfer is whether the student's predictions, after learning from those differences, move in a direction that helps the target capability — not properties of the prompt itself such as difficulty, length or diversity.
Three sets of results support this.
- Properties of the prompt or of the student do not identify effective data. None of eight selectors is significantly better than uniform sampling, the only significant effect being Shortest at −1.16 (95% CI −2.02 to −0.33); MoPPS shows no consistent advantage over a matched uniform online control (−0.11 and +0.61 across two seeds); and on two fresh candidate pools the Hard and Semantic advantages reverse (W4).
- The same prompts are effective or not depending on the teacher. With initialization and supports held fixed, swapping the teacher reverses the preferred source: the difference-in-differences is 27.3 points on Math and 13.4 on the four-task average, positive in all five seed configurations.
- Once the data exposes the right differences, the count can be very small and a failure can be corrected. Four prompts match sampling from a 17,170-prompt pool at a matched source and token budget (W1), and a support that degrades the student recovers fully when the prompts are switched at a matched budget (W4).
We also thank the reviewer for asking W3/Q3, which led to a valuable and interesting result: this direction can be measured. α, the projection of the student's functional change onto the direction of a teacher competent on the target task, explains 93% of the variance in Math accuracy across the 20 teacher-swap runs, and measured after only 20% of training it already predicts the final ordering (Spearman 0.97). Prompt effectiveness is therefore not only relational but diagnosable early, which is what makes the claim usable. Our next comment reports this analysis.
To our knowledge, the effect of training data on on-policy distillation was not studied systematically before the concurrent work that appeared within two months of our submission: the literature addressed objectives, teacher choice and sampling cost. Relative to the three concurrent studies:
| What the concurrent work found | What our study establishes beyond it |
|---|---|
| [1] Transfer improves when prompt content and templates are aligned with the teacher's post-training data. | The alignment does not act through the teacher's memory of specific items: instances the teacher saw during its own RL and instances it never saw transfer equally (45.26 against 45.28), and a teacher never trained on DAPO still prefers DAPO (26.25 against 25.57). How many prompts a source needs is also teacher-dependent: under JustRL, DeepMath goes from 29.17 at 8 prompts to 32.55 at 3,840, while under DeepScaleR it does not improve at all from 8 to 57,046. |
| [2] A single training query can transfer substantially, with effectiveness related to training-state coverage. | Sufficient is not interchangeable. Single code prompts span 50.03 to 61.37 Code Macro, straddling the untrained student at 53.30, so the same count gives opposite outcomes. A support that degrades the student is recovered by switching prompts rather than by further optimization, and at a matched budget the recovered path is indistinguishable from one that never degraded (W4). |
| [3] On-policy distillation propagates strengths and weaknesses across domains, with generalization depending on model origin. | We isolate the teacher's role: with the student initialization and the supports held fixed, swapping the teacher alone reverses which source is preferred. We also measure interactions inside a fixed 48-prompt mixture, where instruction-following prompts cost 10.1 BFCL points while a mixture holding both instruction-following and agent prompts scores 17.3 above it. |
[1] Y. Li et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe. arXiv:2604.13016, 2026.
[2] Z. Fu et al. Rethinking on-policy distillation of large language models II: One training example. arXiv:2609.04172, 2026.
[3] Z. Li et al. Every coin has two sides: On the dual nature of generalization in on-policy distillation of large language models. arXiv:2608.16647, 2026.
One practical insight holds across every setting we tested: in on-policy distillation, no selector beats uniform online sampling over a mid-sized candidate pool, and each costs extra screening compute on top.
Response to Reviewer bdZF (3/3)
W3 & Q3: Connecting functional alignment to downstream transfer.
Thank you for this question. We tested whether functional alignment also explains and predicts downstream transfer, and it does.
We measured how far the student's predictions moved from initialization along a task-specific reference direction (α), using frozen probes and the same reference teacher for every run: Math-RL for math and Code-RL for code. The table reports means over five seed configurations.
| Prompt set | Supervising teacher | Cosine to supervising teacher | α along Math-RL direction | Math |
|---|---|---|---|---|
| DeepMath-48 | Math-RL | 0.58 | 1.18 | 47.6 |
| DeepMath-48 | Code-RL | 0.94 | 0.10 | 25.1 |
| Code-48 | Math-RL | 0.60 | 0.28 | 29.5 |
| Code-48 | Code-RL | 0.87 | 0.29 | 34.2 |
With DeepMath-48 fixed, switching to the Code-RL teacher raised cosine from 0.58 to 0.94, yet α fell from 1.18 to 0.10 and Math accuracy fell from 47.6 to 25.1. Under each teacher, the prompt set with higher α also scored higher on Math. Across all 20 runs, α tracked downstream scores more closely than cosine did (in-sample R² of 0.93 vs. 0.34 for Math and 0.90 vs. 0.29 for Code).
On math, α is close to the product of two measured quantities: how much of the teacher's own functional change the prompt set reproduces in the student, and how much of that teacher's change points toward math competence. For DeepMath-48 under the Code-RL teacher the first is about 1.0 and the second is 0.09, which predicts α = 0.09 against 0.10 measured. A prompt set can only pass on what the teacher's own update contains, and that is what caps this cell.
This signal appeared early. At update 5 of 25, α already tracked the final Math ranking across the 20 runs (Spearman ρ = 0.97). In a separate code experiment across 13 supports, alignment one quarter of the way through training also tracked final Code Macro (ρ = 0.89).
W4: Robustness of prompt selection and recovery at matched updates.
Thanks for raising these points. We repeated the Table 26 comparison on two independently drawn pools of 2,304 candidates, selecting 48 prompts per method. The table reports Avg4 differences relative to uniform sampling (95% CIs in brackets).
| Selector − Uniform | Original pool | New pool 1 | New pool 2 |
|---|---|---|---|
| Hard | +0.46 | −1.58 [−2.63, −0.52] | −1.06 [−2.16, +0.05] |
| Semantic | +0.47 | −0.95 [−2.08, +0.17] | −0.06 [−1.17, +1.06] |
Both selectors scored slightly above uniform sampling on the original pool and below it on both new pools. Only the Hard–Uniform difference on the first new pool had a 95% CI excluding zero.
For code recovery, we compared three paths with 200 updates each and an optimizer reset at update 100. The first two started their second stage from the same degraded checkpoint.
| Training path | Code Macro at update 200 |
|---|---|
| A′ → A′ (continue on the original prompt) | 44.24 |
| A′ → M48 (switch to 48 prompts) | 63.16 |
| M48 → M48 (use the 48 prompts throughout) | 62.48 |
With the same starting checkpoint and 100 further updates, switching prompts improved Code Macro by 18.92 points over continuing on A′ (95% CI, +16.56 to +21.31). The switched run finished 0.68 points above the M48-only run (95% CI, −0.05 to +1.42).
Changes to the manuscript.
- Prompt count. §§3.2–4.1, Table 15 and Appendix D.3 add the same-source, seed and second-teacher controls; Appendix B.3 documents pre-specification and overlap checks.
- Central claim. §1 and Appendix A state it explicitly, and §5.1 shows how early task-specific alignment tracks final accuracy.
- Selection and recovery. Table 4 compares MoPPS with uniform online sampling, Appendix F.4 tests selectors on two new pools, and Figure 6(c) compares all three recovery paths at 200 updates.
We thank the reviewer again for these comments, and we are happy to answer any further questions during the discussion period.
Response to Reviewer bdZF (1/3)
We thank the reviewer for the detailed assessment.
Thank you for these two questions.
Additional draws and seeds. We varied the support draw and the training seed separately. First, we held the training seed fixed and drew four additional four-prompt DAPO supports. We then held the original DAPO and DeepMath supports fixed and trained each with seeds 42, 43, and 44.
The difference between the two means is −0.92 (95% CI, −2.40 to +0.45).
Count with the source held fixed. Within DAPO, at update 50 and with supervision budgets within 5% of each other (91.4M against 96.1M), four prompts give 32.24 Math and sampling from the full 17,170-prompt pool gives 32.34, a difference of −0.10 (95% CI, −2.24 to +1.93). The count claim does not rest on the choice of source.
A second teacher with different RL data. We repeated the comparison with DeepScaleR-1.5B-Preview (teacher 26.82, initial student 19.22). Its RL data is AIME, AMC, Omni-MATH and Still; DAPO is not among its reported RL sources.
With DeepScaleR, eight DAPO prompts scored 26.25, against 25.57 for eight DeepMath prompts and 25.33 for the full DeepMath pool. Compatibility with the teacher's RL data is therefore not the explanation.
Seen versus unseen instances. In the 1.7B direct-code control of Appendix D.3, prompts from instances the teacher saw during its own RL and prompts it never saw transfer equally (45.26 against 45.28), so the effect is not recall of specific training items.
Contamination. We audited both pools against all evaluation sets using shared 13-grams and word-level 5-gram Jaccard overlap: no evaluation problem is a near-duplicate of a training prompt in either pool.
Pre-specification. Supports were written before the final evaluations (the seed-42 support on 31 August at 21:09, evaluated on 1 September at 17:26); each endpoint is fixed by the training configuration, not chosen by validation score; and each single-prompt selection carries an audit file with its criterion and correctness_used_for_selection = False.