- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how many training queries are needed for on-policy distillation. It reports that repeatedly training on one query recovers a substantial fraction of the improvement obtained with the full dataset, while 16 semantically diverse queries can match full-data performance under the tested budget. The experiments cover mathematical reasoning, code generation, instruction following, and tool use, with additional results for multiple teachers and content-light inputs. The paper offers two explanations. First, rollouts from a small query set visit many of the teacher-hidden-state clusters reached by full-data training. Second, teacher–student alignment slows at a similar rate across query-set sizes. The authors interpret these observations as evidence that OPD receives abundant supervision but absorbs it inefficiently.
Critical strengths
The central empirical observation is interesting. Substantial improvement from a single query, including one the student never solves, is worth documenting. The extension to content-light templates makes the result more informative than a demonstration based only on a carefully chosen problem.
The experiments cover several domains and model families. The multi-teacher results and out-of-domain evaluations provide useful evidence that the phenomenon is not confined to one mathematical benchmark.
The appendix includes meaningful controls. These examine cluster count, reference construction, state-extraction positions, query diversity at a fixed query count, frozen trajectories, and learning rate. The paper also explicitly distinguishes rollout-subsampling uncertainty from variation across trained models.
Critical weaknesses
-
The coverage metric does not establish how much useful supervision is shared. Touching 71.5% of the K-means clusters leaves open whether states within a cluster have similar teacher targets or training gradients. Appendix B.1 checks robustness to clustering choices, but not this connection. Coverage and accuracy could simply rise together as rollouts become more varied.
-
The absorption-rate analysis mixes learning with changes in the sampling distribution. Equations (5)–(6) evaluate each checkpoint on its own rollouts, so a smaller distance may partly reflect visiting states where teacher and student already agree. Similar normalized curves do not establish that OPD is "algorithm-starved." A common, fixed probe set is needed to separate these effects; prolonged learning on frozen trajectories does not resolve this issue.
-
The main comparisons lack independent repetitions across training seeds and query sets. Mathematics uses three selected queries, and the other domains use one query each. The 30 repetitions in Appendix B.1 resample existing rollouts. They do not show whether independently chosen 16-query sets consistently match full-data training, particularly when the reported scores differ by only 0.1 points.
-
The data-sufficiency claim depends on the training budget and model pairing. One-shot recovery of full-data gains falls from 87% at step 300 to 72% at step 1000. The teacher–student pairs are also within-family and same-size; the instruction-following teacher is obtained by post-training the student itself. It remains unclear whether so few queries suffice with a larger teacher or a less closely related pair.
-
The training loss needs clearer specification. Equations (1)–(2) define advantages, but leave the optimized surrogate and treatment of stored trajectories unclear. There is also a gap between the headline experiment and its explanation: Section 3 uses top-16 supervision, whereas Sections 4–5 use sampled-token supervision. The proposed mechanism should be checked under the setting used for the main result.
Questions for authors
Do states assigned to the same coverage cluster have similar teacher distributions or training gradients? Can coverage predict held-out improvement across independently selected query sets?
Does the decline in absorption rate persist when all checkpoints and query settings are evaluated on the same fixed states and token distribution? How much of the measured change comes from parameter updates versus changes in visited states?
What is the distribution of one-shot performance across randomly selected queries and independent training seeds? Do independently sampled 16-query sets consistently match full-data training?
Does the 16-shot result persist beyond 300 steps and with independently tuned full-data baselines? What happens with a larger teacher or a less closely related teacher–student pair?
Please provide the exact training loss and off-policy update rule, repeat the main mechanistic comparisons under top-16 supervision, and resolve the evaluation-protocol inconsistencies noted above.
7 comments
Response to W3 and Q3
W3: The main comparisons lack independent repetitions across training seeds and query sets. Mathematics uses three selected queries, and the other domains use one query each. The 30 repetitions in Appendix B.1 resample existing rollouts. They do not show whether independently chosen 16-query sets consistently match full-data training, particularly when the reported scores differ by only 0.1 points.
Q3: What is the distribution of one-shot performance across randomly selected queries and independent training seeds? Do independently sampled 16-query sets consistently match full-data training?
Building upon Our Original Analysis
-
Training query difficulty. Figure 4 shows that one-shot OPD improves held-out mathematics accuracy with each of the easy, medium, and hard training queries, including the hard query that the student never solves during training.
-
Comparison with full-data OPD. At step 300, the selected 16-query set reaches 70.9% average accuracy versus 70.8% for full-data OPD in mathematics (§4.2). In three-domain MOPD, the selected sets with 16 queries per domain reach 52.9% versus 52.8% for full-data MOPD (Appendix D). Whether independently sampled 16-query sets consistently achieve comparable results remains open.
-
Domains, model families, and sizes. Across our experiments, one-shot OPD improves performance in four task domains and three model families, with student sizes ranging from 1.5B to 7B parameters.
One-shot performance across queries (Appendix A.4, Table 6)
We report step-300 results for three distinct training queries in each non-mathematical domain.
| Domain and metric | Initial student | Teacher | Query 1 | Query 2 | Query 3 |
|---|---|---|---|---|---|
| Code generation (LCB v6, avg@3) | 19.43 | 30.10 | 22.86 | 23.24 | 27.24 |
| Instruction following (Multi-IF third-turn, 8-language mean) | 20.84 | 28.58 | 25.07 | 25.92 | 25.39 |
| Agentic tool use (BFCL v3, avg@8) | 70.775 | 78.948 | 76.021 | 73.608 | 75.091 |
One-shot OPD improves on the initial student in all nine runs. Across the three queries tested per domain, teacher–student gap recovery ranges from 32.1% to 73.2% in code generation, 54.7% to 65.6% in instruction following, and 34.7% to 64.2% in agentic tool use. Estimating the one-shot performance distribution across randomly sampled queries and independent training seeds would require substantial additional compute and training time.
Response to W4 and Q4
W4: The data-sufficiency claim depends on the training budget and model pairing. One-shot recovery of full-data gains falls from 87% at step 300 to 72% at step 1000. The teacher–student pairs are also within-family and same-size; the instruction-following teacher is obtained by post-training the student itself. It remains unclear whether so few queries suffice with a larger teacher or a less closely related pair.
Q4: Does the 16-shot result persist beyond 300 steps and with independently tuned full-data baselines? What happens with a larger teacher or a less closely related teacher–student pair?
Building upon Our Original Analysis
-
Comparison under a shared training recipe. At step 300, the selected 16-query set reaches 70.9% average accuracy versus 70.8% for full-data OPD in mathematics (§4.2). In three-domain MOPD, 16 queries per domain reach 52.9% versus 52.8% for full-data MOPD (Appendix D). We hold the training recipe fixed across query budgets so that the comparison isolates the choice of training queries.
-
One-shot performance through 1,000 steps. In mathematics, one-shot accuracy stays near its step-300 level, moving from 68.5% to 68.4%, while full-data accuracy rises from 69.8% to 72.1%. One-shot OPD thus recovers 87% of the full-data gain at step 300 and 72% at step 1,000 (§3.2).
-
Choice of teacher–student pairs. We match the teacher and student in size and model family to study query budgets in a setting where OPD already works. For code generation and instruction following, the teachers are obtained by post-training the same 1.5B model used to initialize the student. Changing the teacher's size or family can affect distillation stability independently of the query budget. For a larger or less closely related teacher, we would first establish a working full-data OPD baseline for that pair and then vary the number of training queries.
New Experiment: Extending 16-Shot OPD to 1,000 Steps
We extend 16-shot OPD to 1,000 steps in code generation and instruction following and report the scores every 100 steps.
| Training step | LiveCodeBench (avg@3) | Multi-IF English (turn 3 overall) |
|---|---|---|
| 0 | 19.05 | 23.22 |
| 100 | 25.90 | 35.51 |
| 200 | 26.48 | 34.55 |
| 300 | 26.48 | 35.61 |
| 400 | 26.67 | 34.21 |
| 500 | 27.81 | 34.92 |
| 600 | 28.19 | 34.71 |
| 700 | 29.33 | 32.49 |
| 800 | 28.19 | 34.12 |
| 900 | 29.71 | 34.31 |
| 1,000 | 29.90 | 32.99 |
At step 1,000, 16-shot OPD retains substantial gains over the initial student in both domains. Code improves from 26.48 at step 300 to 29.90 at step 1,000. Instruction following declines from 35.61 to 32.99 over the same interval and finishes 9.77 points above its initial score.
Response to W1、Q1
[Global Response 1] tests both the semantic interpretation and the alternative rollout-diversity explanation. Same-cluster states have closer teacher targets, with mean JS divergence 0.4683 versus 0.6869; gradient directions are more aligned on average, but the intervals include zero. Coverage and accuracy also rise together across response diversity, semantic query diversity, and the 1/4/16 ladder, while changing query order has little effect. This joint pattern supports coverage as an empirical correlate of data efficiency beyond simple rollout variation. We will continue causal validation on independently selected query sets.
Response to W2、Q2
[Global Response 2] isolates state visitation by evaluating every checkpoint on the same 1,024 prefixes with full-vocabulary JS divergence and reverse KL. Both divergences decrease, and the per-update absorption rate is lower over steps 250–300 than over 50–100 in every tested run. The slowdown therefore remains when all checkpoints see the same evaluation states, which separates parameter-driven slowdown from changes in visited states. This fixed-prefix control also complements the separate fixed-training-trajectory experiment: one holds evaluation states fixed, while the other holds training trajectories fixed.