acceptodds
Under review as a conference paper at ICLR 2027

Beyond Prompt Count: How Data Shapes Transfer in On-Policy Distillation

Abstract

On-policy distillation (OPD) trains students using teacher feedback on their own sampled responses, yet how prompt choice shapes transfer across teacher–student pairs remains poorly understood. We systematically study prompt quantity, source, and selection across RL- and SFT-continuation pairs and cross-model settings. We find that OPD can be highly prompt-efficient: a few prompts can approach large-pool performance, with four DAPO prompts matching the observed mathematics score of 3,840 DeepMath prompts. However, prompt utility is relational rather than intrinsic: changing only the teacher can reverse the relative effectiveness of mathematics and code prompts. To characterize these transfer differences, we analyze parameter and functional changes across prompt supports and model pairs. Functional alignment with the teacher varies across supports and target tasks; in continuation pairs, teacher-aligned prediction changes can coexist with weak parameter alignment. Continued OPD on effective supports can restore performance after unfavorable transfer. Finally, targeted selection does not consistently outperform uniform random sampling, and filtering out a source that performs poorly alone yields no consistent gain across three paired support draws. Overall, our results distinguish prompt efficiency from prompt interchangeability and show that effective data choice depends on the teacher–student pair and target capability, with random sampling providing a competitive baseline in the studied settings. Our code is available at https://anonymous.4open.science/r/dissecting-opd-13D4.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, data selection, data efficiency, on-policy distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bdZFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an extensive empirical study of how prompt quantity, data sources, and selection strategies affect knowledge transfer in on-policy distillation (OPD). The authors investigate multiple teacher–student pairs and report several findings, including strong transfer with very few prompts, teacher-dependent prompt preferences, functional alignment despite different parameter updates, recovery from unfavorable transfer, and the competitiveness of random prompt selection.

Critical strengths

1.Important and timely research question. Understanding how training data affects knowledge transfer in OPD is a practically relevant problem, especially given the growing interest in data-efficient post-training.

2.Comprehensive empirical evaluation. The paper covers multiple teacher–student configurations, model sizes, data sources, and evaluation benchmarks, providing a broad empirical perspective on OPD.

3.Interesting observations on teacher-dependent prompt preferences. The controlled teacher-swap experiments demonstrate that the effectiveness of a prompt source depends on the teacher–student relationship, rather than solely on the properties of the prompts.

4.Useful analysis of transfer behavior and data selection. The functional alignment and recovery experiments provide additional insights into OPD behavior, while comparisons with random sampling offer practical guidance for prompt selection.

Critical weaknesses

1.Insufficient robustness of the headline prompt-efficiency result. The striking 960× comparison uses different data sources and relies on a particular four-prompt support. Since DAPO is also used to train the JustRL teacher, the result may reflect data compatibility rather than prompt efficiency alone. Additional random seeds and contamination checks are needed to establish its generality and rule out potential post-hoc selection effects.

2.Despite presenting five empirical takeaways, the paper reads more like a comprehensive technical report than a study with a clear central contribution. The observations remain largely descriptive, without establishing a generalizable principle or mechanism explaining which prompts lead to successful transfer.

3.The functional alignment analysis is interesting but primarily characterizes training outcomes rather than explaining their causes. In particular, the relationship between functional alignment, prompt effectiveness, and downstream performance remains unclear, limiting the explanatory and predictive value of these findings.

4.Several prompt-selection methods are evaluated on a single support, while random sampling uses multiple draws. Additionally, some recovery experiments involve unequal training budgets, making it difficult to isolate the effects of prompt selection from additional optimization.

Questions for authors
  1. How consistent is the four-prompt result across different random seeds? To what extent does the 960× efficiency gap reflect prompt count rather than data-source compatibility?

  2. Given that several findings align with existing intuitions about OPD, what fundamentally new and generalizable insights does this study provide beyond prior work?

  3. Can the authors establish a clearer connection between functional alignment and downstream transfer performance to explain why certain prompts are more effective?

7 comments
AuthorsOpenReview ↗

Response to Reviewer bdZF (1/3)

We thank the reviewer for the detailed assessment.

W1 & Q1: Robustness of the prompt-count result, and count versus data-source compatibility.

Thank you for these two questions.

Additional draws and seeds. We varied the support draw and the training seed separately. First, we held the training seed fixed and drew four additional four-prompt DAPO supports. We then held the original DAPO and DeepMath supports fixed and trained each with seeds 42, 43, and 44.

Support drawMathAvg4
seed 42 (paper)32.5538.76
seed 4330.8338.26
seed 4432.2437.78
seed 4531.8238.48
seed 4632.7137.70
Mean ± SD32.03 ± 0.7538.20 ± 0.45
Training seedDAPO, 4 promptsDeepMath, 3,840 prompts
42 (paper)32.5532.55
4331.2032.03
4430.7332.66
Mean ± SD31.49 ± 0.9432.41 ± 0.34

The difference between the two means is −0.92 (95% CI, −2.40 to +0.45).

Count with the source held fixed. Within DAPO, at update 50 and with supervision budgets within 5% of each other (91.4M against 96.1M), four prompts give 32.24 Math and sampling from the full 17,170-prompt pool gives 32.34, a difference of −0.10 (95% CI, −2.24 to +1.93). The count claim does not rest on the choice of source.

A second teacher with different RL data. We repeated the comparison with DeepScaleR-1.5B-Preview (teacher 26.82, initial student 19.22). Its RL data is AIME, AMC, Omni-MATH and Still; DAPO is not among its reported RL sources.

Prompt set (DeepScaleR teacher, update 65)Math
DAPO, 8 prompts26.25
DeepMath, 8 prompts25.57
DeepMath, 3,840 prompts25.62
DeepMath, 57,046 prompts (full pool)25.33

With DeepScaleR, eight DAPO prompts scored 26.25, against 25.57 for eight DeepMath prompts and 25.33 for the full DeepMath pool. Compatibility with the teacher's RL data is therefore not the explanation.

Seen versus unseen instances. In the 1.7B direct-code control of Appendix D.3, prompts from instances the teacher saw during its own RL and prompts it never saw transfer equally (45.26 against 45.28), so the effect is not recall of specific training items.

Contamination. We audited both pools against all evaluation sets using shared 13-grams and word-level 5-gram Jaccard overlap: no evaluation problem is a near-duplicate of a training prompt in either pool.

Pre-specification. Supports were written before the final evaluations (the seed-42 support on 31 August at 21:09, evaluated on 1 September at 17:26); each endpoint is fixed by the training configuration, not chosen by validation score; and each single-prompt selection carries an audit file with its criterion and correctness_used_for_selection = False.

AuthorsOpenReview ↗

Response to Reviewer bdZF (2/3)

W2 & Q2: The central contribution.

Thank you for the suggestion to state the central contribution more explicitly.

Our claim is that in on-policy distillation the value of a prompt is relational. A prompt determines where the student starts generating trajectories, and the states the student then visits determine which differences between the teacher's and the student's predictions training can reach at all. What matters for transfer is whether the student's predictions, after learning from those differences, move in a direction that helps the target capability — not properties of the prompt itself such as difficulty, length or diversity.

Three sets of results support this.

  1. Properties of the prompt or of the student do not identify effective data. None of eight selectors is significantly better than uniform sampling, the only significant effect being Shortest at −1.16 (95% CI −2.02 to −0.33); MoPPS shows no consistent advantage over a matched uniform online control (−0.11 and +0.61 across two seeds); and on two fresh candidate pools the Hard and Semantic advantages reverse (W4).
  2. The same prompts are effective or not depending on the teacher. With initialization and supports held fixed, swapping the teacher reverses the preferred source: the difference-in-differences is 27.3 points on Math and 13.4 on the four-task average, positive in all five seed configurations.
  3. Once the data exposes the right differences, the count can be very small and a failure can be corrected. Four prompts match sampling from a 17,170-prompt pool at a matched source and token budget (W1), and a support that degrades the student recovers fully when the prompts are switched at a matched budget (W4).

We also thank the reviewer for asking W3/Q3, which led to a valuable and interesting result: this direction can be measured. α, the projection of the student's functional change onto the direction of a teacher competent on the target task, explains 93% of the variance in Math accuracy across the 20 teacher-swap runs, and measured after only 20% of training it already predicts the final ordering (Spearman 0.97). Prompt effectiveness is therefore not only relational but diagnosable early, which is what makes the claim usable. Our next comment reports this analysis.

To our knowledge, the effect of training data on on-policy distillation was not studied systematically before the concurrent work that appeared within two months of our submission: the literature addressed objectives, teacher choice and sampling cost. Relative to the three concurrent studies:

What the concurrent work foundWhat our study establishes beyond it
[1] Transfer improves when prompt content and templates are aligned with the teacher's post-training data.The alignment does not act through the teacher's memory of specific items: instances the teacher saw during its own RL and instances it never saw transfer equally (45.26 against 45.28), and a teacher never trained on DAPO still prefers DAPO (26.25 against 25.57). How many prompts a source needs is also teacher-dependent: under JustRL, DeepMath goes from 29.17 at 8 prompts to 32.55 at 3,840, while under DeepScaleR it does not improve at all from 8 to 57,046.
[2] A single training query can transfer substantially, with effectiveness related to training-state coverage.Sufficient is not interchangeable. Single code prompts span 50.03 to 61.37 Code Macro, straddling the untrained student at 53.30, so the same count gives opposite outcomes. A support that degrades the student is recovered by switching prompts rather than by further optimization, and at a matched budget the recovered path is indistinguishable from one that never degraded (W4).
[3] On-policy distillation propagates strengths and weaknesses across domains, with generalization depending on model origin.We isolate the teacher's role: with the student initialization and the supports held fixed, swapping the teacher alone reverses which source is preferred. We also measure interactions inside a fixed 48-prompt mixture, where instruction-following prompts cost 10.1 BFCL points while a mixture holding both instruction-following and agent prompts scores 17.3 above it.

[1] Y. Li et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe. arXiv:2604.13016, 2026.

[2] Z. Fu et al. Rethinking on-policy distillation of large language models II: One training example. arXiv:2609.04172, 2026.

[3] Z. Li et al. Every coin has two sides: On the dual nature of generalization in on-policy distillation of large language models. arXiv:2608.16647, 2026.

One practical insight holds across every setting we tested: in on-policy distillation, no selector beats uniform online sampling over a mid-sized candidate pool, and each costs extra screening compute on top.

AuthorsOpenReview ↗

Response to Reviewer bdZF (3/3)

W3 & Q3: Connecting functional alignment to downstream transfer.

Thank you for this question. We tested whether functional alignment also explains and predicts downstream transfer, and it does.

We measured how far the student's predictions moved from initialization along a task-specific reference direction (α), using frozen probes and the same reference teacher for every run: Math-RL for math and Code-RL for code. The table reports means over five seed configurations.

Prompt setSupervising teacherCosine to supervising teacherα along Math-RL directionMath
DeepMath-48Math-RL0.581.1847.6
DeepMath-48Code-RL0.940.1025.1
Code-48Math-RL0.600.2829.5
Code-48Code-RL0.870.2934.2

With DeepMath-48 fixed, switching to the Code-RL teacher raised cosine from 0.58 to 0.94, yet α fell from 1.18 to 0.10 and Math accuracy fell from 47.6 to 25.1. Under each teacher, the prompt set with higher α also scored higher on Math. Across all 20 runs, α tracked downstream scores more closely than cosine did (in-sample R² of 0.93 vs. 0.34 for Math and 0.90 vs. 0.29 for Code).

On math, α is close to the product of two measured quantities: how much of the teacher's own functional change the prompt set reproduces in the student, and how much of that teacher's change points toward math competence. For DeepMath-48 under the Code-RL teacher the first is about 1.0 and the second is 0.09, which predicts α = 0.09 against 0.10 measured. A prompt set can only pass on what the teacher's own update contains, and that is what caps this cell.

This signal appeared early. At update 5 of 25, α already tracked the final Math ranking across the 20 runs (Spearman ρ = 0.97). In a separate code experiment across 13 supports, alignment one quarter of the way through training also tracked final Code Macro (ρ = 0.89).

W4: Robustness of prompt selection and recovery at matched updates.

Thanks for raising these points. We repeated the Table 26 comparison on two independently drawn pools of 2,304 candidates, selecting 48 prompts per method. The table reports Avg4 differences relative to uniform sampling (95% CIs in brackets).

Selector − UniformOriginal poolNew pool 1New pool 2
Hard+0.46−1.58 [−2.63, −0.52]−1.06 [−2.16, +0.05]
Semantic+0.47−0.95 [−2.08, +0.17]−0.06 [−1.17, +1.06]

Both selectors scored slightly above uniform sampling on the original pool and below it on both new pools. Only the Hard–Uniform difference on the first new pool had a 95% CI excluding zero.

For code recovery, we compared three paths with 200 updates each and an optimizer reset at update 100. The first two started their second stage from the same degraded checkpoint.

Training pathCode Macro at update 200
A′ → A′ (continue on the original prompt)44.24
A′ → M48 (switch to 48 prompts)63.16
M48 → M48 (use the 48 prompts throughout)62.48

With the same starting checkpoint and 100 further updates, switching prompts improved Code Macro by 18.92 points over continuing on A′ (95% CI, +16.56 to +21.31). The switched run finished 0.68 points above the M48-only run (95% CI, −0.05 to +1.42).

Changes to the manuscript.

  • Prompt count. §§3.2–4.1, Table 15 and Appendix D.3 add the same-source, seed and second-teacher controls; Appendix B.3 documents pre-specification and overlap checks.
  • Central claim. §1 and Appendix A state it explicitly, and §5.1 shows how early task-specific alignment tracks final accuracy.
  • Selection and recovery. Table 4 compares MoPPS with uniform online sampling, Appendix F.4 tests selectors on two new pools, and Figure 6(c) compares all three recovery paths at 200 updates.

We thank the reviewer again for these comments, and we are happy to answer any further questions during the discussion period.

Reviewer_sZqDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how prompt count, source, and selection affect on-policy distillation across several teacher-student settings. A few prompts can approach the performance of a much larger pool, but the preferred source changes with the model pair and target capability. Teacher swaps, token-matched comparisons, and support-switch experiments help explain these differences. The selection experiments also show that uniform sampling is a competitive baseline.

Critical strengths
  1. The teacher-swap experiment is particularly useful. With the student initialization and prompt supports held fixed, changing the teacher reverses the source preference on all four evaluations (Figure 3b). Table 19 repeats this comparison across five support/seed configurations. Together with the approximately token-matched results in Tables 2 and 20, this supports the conclusion that source quality depends on more than supervision volume.

  2. The paper tests several practical explanations rather than relying only on the few-prompt examples. It reports selection costs and variation across random supports (Table 4), uses paired draws for source filtering (Table 5), and includes matched-update continuation and alternative answer scoring for the Math recovery result (Appendix E.5). These controls make the findings useful for choosing distillation data.

Critical weaknesses

The main limitation is how reliably a newly sampled small support will work. The four-DAPO versus 3,840-DeepMath comparison reaches the same reported Math score, but also changes the source (Table 15). Some count sweeps use one nested sequence of supports (Appendix B.3), and the single-code-prompt examples vary considerably (Table 13). The evidence supports the stated claim that a few prompts can suffice; it gives less guidance about the chances of success with a new draw. This does not outweigh the controlled source comparisons, and I lean toward acceptance.

Questions for authors
  1. Table 4 already reports three support draws for the selection study. Are there analogous independent draws for the four-DAPO versus 3,840-DeepMath comparison in Table 15? If available, please report their variability and the sampling rule.

  2. For the prompt-count and single-prompt results, please clarify whether the illustrated supports and checkpoints were fixed before inspecting the final benchmarks, and how the training-time AIME evaluations in Appendix C.3 affected selection.

4 comments
AuthorsOpenReview ↗

Response to Reviewer sZqD

We thank the reviewer for the positive assessment.

W1: The chance of success with a newly sampled support.

Thank you for raising this.

Random single code prompts. The single-prompt supports in Table 13 were chosen by pre-specified criteria, so they cannot speak to a random draw. We drew four single prompts uniformly at random from the 22,618-prompt Eurus Code pool and trained with the Code-RL teacher under the Table 13 protocol. We evaluated all four at update 50.

DrawHE+MBPP+LCBCode Macro
seed 10186.4372.7227.8662.34
seed 10287.2771.7628.1462.39
seed 10387.1270.9029.4362.48
seed 10485.2971.8326.3661.16

At update 50, the four random draws scored 61.16–62.48 on Code Macro (mean 62.09), compared with 50.92 for A′ at the same update. We also continued seed 104 to update 100, where it scored 61.71, close to the 48-prompt support’s 61.79. All four draws avoided the degradation seen with A′, though four draws give only a limited picture of how often it occurs.

Removing the source change. Holding the source fixed, four DAPO prompts and sampling from the full DAPO pool reach 32.24 and 32.34 Math at update 50, with token budgets within 5% (91.4M against 96.1M); the difference is −0.10 (95% CI, −2.24 to +1.93).

What remains nested. The DAPO count sweep still uses one nested sequence of supports. Below, we report five independent draws at M=4 to show how much the result varies at that support size.

Q1: Independent draws and the sampling rule for Table 15.

Thanks for asking. Yes. We ran four additional independent draws of four DAPO prompts, holding the teacher, protocol and training seed at the Table 15 settings. Each draw ranks the cleaned 17,170-prompt DAPO pool by SHA-256(selection seed, "dapo-support-order", prompt index) and takes the first four prompts.

Support drawMathAvg4
seed 42 (paper)32.5538.76
seed 4330.8338.26
seed 4432.2437.78
seed 4531.8238.48
seed 4632.7137.70
Mean ± SD32.03 ± 0.7538.20 ± 0.45

Math ranges from 30.83 to 32.71 across the five draws. The comparison condition in Table 15, 3,840 DeepMath prompts, scores 32.55, so the draws average 0.52 points below it.

We also varied the training seed, and ran the same two extra seeds for the 3,840-prompt condition so that both sides are comparable.

Training seedDAPO, 4 promptsDeepMath, 3,840 prompts
42 (paper)32.5532.55
4331.2032.03
4430.7332.66
Mean ± SD31.49 ± 0.9432.41 ± 0.34

The difference between the two means is −0.92 (95% paired-bootstrap CI, −2.40 to +0.45).

Q2: Whether the reported supports and checkpoints were fixed in advance.

Thank you for checking this. Yes. We fixed the prompt sets before final evaluation, and the training configuration specified in advance which checkpoint would serve as the endpoint for each run. For example, the seed-42 DAPO-4 support was saved on 31 August at 21:09 and evaluated on 1 September at 17:26. For the single-prompt selections in Table 13, the audit files record the selection criterion, the hash of the candidate ranking, and correctness_used_for_selection = False. We ran five single-prompt supports with the Code-RL teacher and report all five, including the two weakest: A′ at 50.03 and the sweep’s M=1 at 57.95.

We used the training-time AIME evaluations to monitor training and plot learning curves. They played no role in choosing which supports or checkpoints to report.

Changes to the manuscript.

  • Sampling rule and pre-specification. Appendix B.3 now gives the corrected DAPO sampling rule, explains how supports and checkpoints were fixed, and clarifies that the original count sweep used nested supports.
  • Independent draws. Table 15 reports the five independent DAPO draws at M=4 separately from the training-seed repeats.
  • Random single prompts. Section 5.3 summarizes the four random single-prompt code runs, with per-draw results in Table 13.
  • Supervision budget. Section 4.1 reports the within-DAPO comparison at similar supervision-token budgets.

We thank the reviewer again for these questions. We are happy to answer any further questions during the discussion period.

AuthorsOpenReview ↗

A Kind Reminder for Reading the Response

Dear Reviewer sZqD,

Thank you for your insightful suggestions. We have revised the paper and added additional results to address your concerns.

Since the rebuttal period is closing very soon, could you please check the response to see whether it mitigates your concerns? We would greatly appreciate that!

Thank you, The authors of Submission 1533

Thank you for following up. The additional runs give me a clearer sense of how much the results vary. That addresses most of my concerns, and I’ll keep my original rating of 3.

AuthorsOpenReview ↗

Thank you for the feedback!

Dear Reviewer sZqD,

Thank you for your time and for such an insightful review. We are glad that the additional runs address your concerns, and your suggestions have made the paper better.

Best regards, The Authors

Reviewer_7o4DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors investigate how prompt quantity and data source affect transfer in on-policy distillation. Through experiments across multiple teacher-student pairs, they show that small, fixed prompt sets can recover much of the improvement achieved with larger datasets when fresh student rollouts are generated throughout training. They also find that the effectiveness of a data source depends on the teacher, while sophisticated prompt selection methods do not consistently outperform uniform sampling. Further analyses show that students can approach the teacher’s behavior through different parameter changes and that switching to more effective prompts can help recover performance after poor initial training.

Critical strengths

The authors provide comprehensive results regarding how prompt count and selection affect on-policy distillation.

The authors study an important problem of prompt importance, showing useful insights of training with less but better prompts.

Critical weaknesses

In general, I have problem with the strong conclusions from the experiments. Please see some points below.

Comparison between the number of rollouts is not exactly fair, if in one case the model is supervised with less tokens. I would like to see the (closely) matched token budget training supervision comparison.

The stronger baselines should be implemented for better comparison in Section 5.4 that study prompt selection in RL. (Qu, Yun, et al. "Can prompt difficulty be online predicted for accelerating rl finetuning of reasoning models?." Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1. 2026.)

The conclusions in the experiment of Section 4.3 is unclear, because not only the composition ratio is changed, but also 2 more domains are added to the mix, which is really hard to get a conclusion here.

Questions for authors

State in weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 7o4D

W1: About the supervision budget.

Thanks for raising this. We counted supervision tokens from the stored rollouts and evaluated intermediate checkpoints.

SourceTraining promptsUpdatesSupervision tokensMath
DAPO4 fixed prompts100181.8M32.55
DeepMath3,840 fixed prompts100155.8M32.55
DAPO4 fixed prompts5091.4M32.24
DAPOSampled from the pool5096.1M32.34

At 100 updates, the four-prompt run used 16.7% more supervision tokens, so the 960× reduction refers to distinct prompts only. Table 15 reports one run per condition; across three training seeds the means are 31.49 and 32.41.

The 50-update rows address the budget question directly: both runs used DAPO with token budgets within 5%, and the pool run saw 12,800 distinct prompts. The four-prompt run scored 0.10 Math points lower (95% paired-bootstrap CI, −2.24 to +1.93), an interval that allows about two points either way.

The other sweeps show the same pattern: with the Math-RL teacher, 1 prompt consumes 41.0M tokens against 26.5M for 3,840 and scores lower (44.74 vs. 46.77); for 30B→4B, 48 prompts consume 16.6M against 21.1M at an equal score (42.81/42.60); for Code-RL, 48 prompts and the full pool differ by 0.10 Code Macro at 109M vs. 118M.

W2: About MoPPS as an online selection baseline.

Thanks for suggesting MoPPS [1]. Its MATH configuration uses a candidate batch of 16B; with B=32 this exceeds our 384-prompt pool, so the candidate set was the full pool at each update. Following Algorithm 1, we sampled a success probability from each prompt's Beta posterior, trained on the 32 closest to the 0.5 target, and updated their posteriors from their eight rewards.

To isolate difficulty-based selection, we ran a uniform online control drawing 32 prompts uniformly from the same pool at each update, with eight responses per prompt and 65 updates.

Training seedMoPPS Avg4Uniform online Avg4DifferenceSupervision tokens (MoPPS / uniform)
4238.2238.33−0.11122.8M / 99.5M
4338.8638.25+0.61121.7M / 100.5M

MoPPS scored 0.25 Avg4 points higher on average and used about 22% more supervision tokens. The difference changed sign across the two seeds, so we found no consistent advantage for difficulty-based selection here.

The selector behaved as intended. MoPPS held the batch success rate near its 0.5 target (0.60 and 0.59 across seeds, against 0.74 and 0.76 for the control), and it cut the fraction of prompts whose eight rollouts share the same outcome — those with zero advantage under an outcome reward — from about 56% to 21%. The quantity MoPPS optimizes therefore changed substantially, while Avg4 did not.

This is expected here: MoPPS is an online curriculum whose benefit in outcome-reward RL comes from avoiding prompts that produce no gradient, and in on-policy distillation the teacher supplies a signal on every response, so that inefficiency does not arise. The result is consistent with Takeaway 5.

[1] Y. Qu et al. Can prompt difficulty be online predicted for accelerating RL finetuning of reasoning models? KDD, 2026.

W3: About the Mix3–Mix5 comparison.

We agree. In Section 4.3, Mix5 scored higher than Mix3 on IFBench and BFCL and lower on GPQA at the same 48-prompt budget. Their six-task averages were nearly identical (57.17 vs. 57.16). The comparison could not isolate domain count because the allocation and the prompts also changed.

Both controls used the Mix-RL teacher and 48 prompts; within each domain, smaller allocations are subsets of larger ones. Differences are first minus second.

ComparisonWhat we held fixedΔ IFBenchΔ BFCL
12 instruction-following vs. 12 agent promptsThe same 36 math, science, and code prompts+12.67−6.78
14/14/14/3/3 vs. 10/10/10/9/9 prompts (math/science/code/IF/agent)The same five domains and 48 total prompts−1.23+2.97

With the same 36 base prompts, instruction-following prompts favored IFBench and agent prompts favored BFCL, and with all five domains fixed, reallocating prompts still shifted BFCL by 2.97 points. Both the choice of domain and its share of the budget affect which tasks benefit.

Changes to the manuscript.

  • Supervision budget. 960× now refers to distinct prompt count (§3.2), and §4.1 adds a within-DAPO comparison at similar supervision-token budgets, with token counts in Appendix G.2.
  • Online selection. Table 4 compares MoPPS with uniform online sampling; §5.4 describes our adaptation.
  • Mixture controls. §4.3 reports both controls and narrows Takeaway 2 to the mixtures tested; Appendix D.4 gives the paired intervals.
  • Captions. Tables 13, 15 and 26 state what each comparison fixes.

We thank the reviewer again for these suggestions, which made the paper more accurate, and we are happy to answer further questions during the discussion.

AuthorsOpenReview ↗

A Kind Reminder for Reading the Response

Dear Reviewer 7o4D,

Thank you again for your detailed review. Following your comments, we have added a within-DAPO comparison at supervision-token budgets within 5% of each other and reported the token counts for every sweep, implemented MoPPS (Qu et al., KDD 2026) and compared it with a matched uniform online control, and added two mixture controls that separate the choice of domain from its share of the budget. The revised PDF is uploaded, with all changes marked in dark blue.

Since the discussion period is closing very soon, could you please take a look at our response and let us know whether it addresses your concerns? We would appreciate it if you could take the new results into account in your final evaluation.

Thank you, The authors of Submission 1533

Reviewer_7o4DOpenReview ↗

Thank you for your reply!

I would like to thank the authors for their reply. I acknowledge reading the author's response and will take that into consideration.

AuthorsOpenReview ↗

Thank you for your time!

Dear Reviewer 7o4D,

Thank you for taking the time to read our response during such a busy period. We really appreciate it.

If anything remains unclear, we would be glad to answer it in the hours left in the discussion. If the added experiments have addressed your concerns, we hope they are helpful as you finalise your assessment.

Best regards, The Authors

Comments on the paper

2 comments

Dear Reviewers,

The authors have posted their responses to the reviews. Please go through their rebuttal at your earliest convenience, and engage in a discussion or leave any follow-up comments.

The discussion period ends on November 18.

Thanks,
Area Chair

AuthorsOpenReview ↗

General Response

We thank all three reviewers for their careful reading and constructive feedback. We are glad that the research question was seen as important and timely (bdZF), and the empirical study as comprehensive across teacher–student pairs, model sizes, sources and benchmarks (bdZF, 7o4D). The reviewers also found the controlled teacher-swap experiment particularly useful (sZqD, bdZF), the functional-alignment and recovery analyses informative about how on-policy distillation behaves (bdZF), and the controls and the comparison with random sampling useful as practical guidance for choosing distillation data (sZqD, bdZF, 7o4D).

The main concerns were the supervision-token budget behind the 960× comparison, the robustness of the four-prompt result, a stronger online selection baseline, what the mixture comparison isolates, and the central claim of the paper. We have added controlled experiments and quantitative analyses accordingly, and uploaded a revised PDF in which all changes are marked in dark blue; the table below summarizes them and where they appear.

ConcernReviewerWhere in the revisionOur actions
1. Supervision-token budget7o4D§3.2, §4.1, App. G.2Restated 960× as a count of distinct prompts, added a within-DAPO comparison at budgets within 5% (32.24 vs 32.34), and reported supervision tokens for every sweep
2. Variation across draws and seedsbdZF, sZqD§4.1, §5.3, Tables 13 and 15Ran five independent four-prompt DAPO draws (32.03 ± 0.75) and three training seeds per side (31.49 vs 32.41), plus four uniformly random single code prompts (61.16–62.48)
3. Prompt count vs. source compatibilitybdZFApp. D.3Repeated the comparison with a teacher whose RL data excludes DAPO: eight DAPO prompts scored 26.25 and eight DeepMath prompts 25.57
4. Pre-specification and contaminationbdZF, sZqDApp. B.3Corrected the sampling rule, dated every support and endpoint, explained how the training-time AIME evaluations were used, and audited both pools against the evaluation sets
5. Selection baselines and robustness7o4D, bdZFTable 4, §5.4, App. F.4Implemented MoPPS (Qu et al., KDD 2026), which scores −0.11 and +0.61 against a matched uniform online control, and retested the selectors on two fresh pools, where the Hard and Semantic advantages reverse
6. Unequal recovery budgetsbdZFFig. 6(c)Compared three recovery paths at 200 updates each: 44.24, 63.16 and 62.48
7. Mixture composition vs. domain count7o4D§4.3, App. D.4Added two controls that separate domain choice from allocation, and narrowed Takeaway 2 to the mixtures tested
8. Central contributionbdZF§1, App. AStated the claim explicitly — the value of a prompt is relational, not a property of the prompt — and set it against the concurrent studies
9. Functional alignment and transferbdZF§5.1Showed that α explains 93% of the variance in Math accuracy and predicts the final ordering of runs after 20% of training

Our individual responses to each reviewer give the point-by-point replies, with the full tables behind the numbers above. We have done our best to address every concern raised, and we would be glad to run further experiments or provide any clarification during the discussion period.

We are sincerely grateful to the three reviewers for the care and expertise they brought to these reviews, and to the Area Chair for taking part in the discussion. The reviewers' comments have made this a stronger and more accurate paper, and we very much look forward to continuing the conversation with all of you.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.