acceptodds
Under review as a conference paper at ICLR 2027

What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection

16 views

Abstract

On-Policy Distillation (OPD) has emerged as a widely adopted post-training paradigm for enhancing large language models in reasoning domains. However, the data-centric mechanisms in OPD remain relatively underexplored. This paper presents an empirical study of data efficiency and data selection in OPD. We begin by investigating an extreme setting: training OPD on only one example, namely **1-shot OPD**. Surprisingly, we find that 1-shot OPD is consistently effective across all sampled training examples and harder examples often yield superior performance gain. We next investigate what actually drives the student model’s improvement in the training data. Our analysis reveals that the improvement is not driven by high token entropy, but the longer CoT paths which hard problem naturally generate. Training on longer CoT can help maintain closer alignment with the teacher over a long reasoning horizon, and learn critical thinking patterns usually missing in short CoTs, such as reflection (e.g., *"Alternatively"*). Based on these insights, we propose a simple data selection method that selects only hard examples for training, where even "unsolvable" examples that completely exceed the teacher's capability can be successfully used. Our experiments conducted on five models ranging from 1.5B to 8B show that training the student model on only 8 selected hard examples matches the performance of the 17K dataset baseline. Code and models will be publicly available.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Large Language Model, Data Selection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_C5mwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an empirical study of data efficiency and data selection in On-Policy Distillation (OPD). It finds that OPD training only on 1 data is consistently effective across benchmarks when scaling training steps. In addition, this paper finds that OPD training on hard problems and longer CoT help better. Experimental results across model sizes, model families, and task domains further demonstrate the empirical findings.

Critical strengths
  1. A good perspective in OPD area and easy to follow.
  2. All findings are interpretable with experiments.
  3. Findings can generalize across model sizes, model families, and task domains.
Critical weaknesses
  1. My biggest concern is whether these findings can truly guide model training. The experiments in this paper are limited to no more than a few dozen training examples, but it is an undeniable fact that the larger the training dataset is, the stronger the model is. It is worth investigating whether using such a small amount of training data might affect other capabilities of the model itself. Could you provide me with OOD experiment results using N-shot OPD and Full OPD?
  2. This paper reports that "Distillation Benefits from Longer CoTs," but it is unclear whether this observation is specific to the N-shot OPD settings or represents a more general scaling trend. In particular, the effect of response length may change when training is performed with the full dataset. It would strengthen this claim to include Full OPD experiments across different training response length settings and examine whether the same trend persists.
Questions for authors

Please refer to the Weaknesses.

1 comment

We sincerely thank you for the constructive feedback. Below, we address your specific questions.

W1: By investigating the distillation performance from a 1-shot OPD perspective, our study aims to better understand the underlying role of data selection. We hope these findings can serve as a helpful reference for practical data curation pipelines. Our observations indicate that in the OPD phase, paying closer attention to data quality, particularly problem difficulty, has the potential to bring meaningful efficiency improvements. Rather than simply scaling up data quantities, which often incurs considerable overhead for data synthesis, curating a smaller subset of challenging problems may offer a more economical pathway to attain competitive reasoning capabilities, thereby alleviating the resource burden associated with large-scale data collection. We have incorporated these insights into the Section 7 of our revised manuscript (highlighted in blue).

Regarding the concern about OOD performance and other potential side effects of small data volumes, we demonstrated in Appendix B.1 that the reasoning capabilities acquired through 1/N-shot OPD purely in the mathematical domain can successfully generalize to coding and scientific domains. Specifically, we evaluated the models on LiveCodeBench (coding reasoning) and GPQA (scientific reasoning). As shown in Table 6 (for 1-shot OPD) and Table 7 (for N-shot OPD), 1/N shot OPD consistently achieves strong OOD performance, with OPD training on 8 hard problems setup nearly matching the generalization capabilities of the Full-Set OPD baseline.

Furthermore, we have added a new experiment in non-reasoning domains, with the detailed setup and results incorporated into Appendix B.2 of the revised manuscript (highlighted in blue). We evaluated general linguistic and creative abilities using the Creative Writing v3 benchmark [1]. We utilized DeepSeek-V4-Pro as the judge model to evaluate the Rubric Score (comprehensive overall writing quality) and Fine-grained Ability Scores (covering distinct creative writing skills such as coherence, descriptive imagery, and instruction following). The evaluation results demonstrate that applying our few-shot OPD method does not compromise the semantic richness or natural language fluency of the outputs.

ModelRubric Score
Student11.45
Teacher11.53
π178\pi_{178} (Easy)11.49
π794\pi_{794} (Medium)11.47
π954\pi_{954} (Hard)11.51

W2: To investigate whether the advantage of longer CoTs generalizes beyond the N-shot setting, we conducted additional Full-Set OPD experiments across different training response lengths, which have now been added to Appendix C.5 of the revised manuscript (highlighted in blue). Specifically, we trained the model on the full dataset with maximum rollout lengths truncated at 1K, 2K, 4K, and 7K, respectively. As the results show, the model's performance consistently improves as the rollout length increases from 1K to 7K. This trend also aligns with similar OPD observations reported in [2].

[1] Samuel J Paech. Eq-bench creative writing benchmark v3. https://github.com/EQ-bench/creative-writing-bench, 2025.

[2] Yaxuan Li et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe. arXiv preprint arXiv:2604.13016.

Reviewer_FiwEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies data efficiency and data selection in on-policy distillation. The authors find that training on a few hard problems can approach full-dataset performance across several teacher–student pairs. The analysis reveals that the improvement is not driven by high token entropy, but by the longer CoT paths that hard problems naturally generate. They also propose a simple data selection method that selects only hard examples for training, where even "unsolvable" examples that completely exceed the teacher's capability can be successfully used.

Critical strengths
  • This paper shows an interesting phenomenon: OPD can work with very few unique training problems.
  • This paper performs comprehensive experiments, including five student models, mathematical reasoning, code generation, and cross-domain evaluation.
  • This paper is written clearly and is easy to follow.
Critical weaknesses
  • Longer rollouts also provide more training tokens. Is the advantage of long rollouts simply due to the model having seen more training tokens? Please compare them with more short rollouts using the same total number of training tokens.
  • Since the authors believe that the benefits primarily stem from a longer CoT, why not simply select questions based on the length of the response?
  • Lower KL on words such as “Alternatively” does not necessarily mean the model has learned to reflect better. Please show whether it actually becomes better at detecting and correcting mistakes.
Questions for authors

None

3 comments
AuthorsOpenReview ↗

Response to Reviewer FiwE – Part 1

We sincerely thank you for the constructive feedback. Below, we address your specific questions.

W1: To investigate whether the advantage of long rollouts simply stems from a larger training token volume, we conducted a newly controlled experiment to isolate the effect of token volume. We have detailed this experimental setup in Appendix C.6 and summarized the results in Section 4.2. Specifically, we kept the optimization steps constant (279 steps) and proportionally scaled up the mini_batch_size for shorter lengths to match the total training tokens of the longest rollout. The baseline 7K-length setting uses a mini_batch_size of 64. For the 1K-length setting, we expanded the mini_batch_size to 448 (64×764 \times 7). Similarly, for the 2K and 4K settings, we adjusted the mini_batch_size to 224 (64×3.564 \times 3.5) and 112 (64×1.7564 \times 1.75), respectively. This configuration ensures that models trained on shorter CoTs process an equivalent or even slightly higher total token volume (accounting for truncation effects).

We then evaluated the final checkpoints trained on π954\pi_{954}, reporting the average accuracy across the AIME24, AIME25, and AMC23.

Maximum Rollout LengthToken-Matched Setting
1K46.6%
2K47.6%
4K48.9%
7K50.1%

As shown in the table, even when the total token volume are controlled, the 7K model still consistently outperforms the token-matched shorter variants. This indicates that the performance gain is not merely a byproduct of seeing more tokens. Instead, continuous, long-horizon reasoning chains contain critical reasoning patterns (such as self-reflection and backtracking) that cannot be replaced by simply accumulating a larger volume of short-horizon tokens.

W2: As noted in Appendix F, we considered the length-based selection and viewed it as a simpler proxy approach, especially in scenarios where rule-based verifiers are unavailable. However, in general, difficulty-based selection is superior. Specifically, we compare the extreme cases among the 8 sampled training examples for each difficulty level. We selected the two longest rollouts from the easy (π21\pi_{21}, π84\pi_{84}) and medium (π763\pi_{763}, π820\pi_{820}) categories, and compare them against the two shortest rollouts from the hard category (π954\pi_{954}, π961\pi_{961}). We have added this detailed comparison to Appendix C.7 (see Table 11) and summarized it in Section 5.1 (highlighted in blue text). The comparison of their 1-shot OPD performance is presented in the table below.

DifficultyPrompt IDToken LengthValidation Accuracy
Easyπ21\pi_{21}5,52348.9%
Easyπ84\pi_{84}5,61849.3%
Mediumπ763\pi_{763}7,07350.0%
Mediumπ820\pi_{820}6,68149.3%
Hardπ954\pi_{954}5,76651.1%
Hardπ961\pi_{961}5,06250.6%

As shown in the table, the token lengths of easy problems are comparable to those of hard problems, while medium problems generate even longer responses. Despite having similar or even longer token lengths, the rollouts from easy and medium problems consistently fail to match the performance gains achieved by the hard problems.

This discrepancy primarily stems from the issue of overthinking [1], which refers to unnecessary repetition, excessive hesitation on reasoning path. This phenomenon frequently occurs in model responses to easy and medium problems. To quantify this, we use the reasoning efficiency ratio [2] to evaluate the proportion of tokens that actively contribute to the reasoning process; the remainder are redundant tokens that can be safely removed without altering the final accuracy. Specifically, the reasoning efficiency ratio for the long easy prompts (π21\pi_{21} and π84\pi_{84}) are 10.3% and 17.1%, whereas those for the long medium prompts (π661\pi_{661} and π794\pi_{794}) are 38.4% and 42.8%. This demonstrates that only a small fraction of the extended CoTs in easy and medium problems actually contributes to the reasoning process and provides valuable supervision signals, which explains why lengthy rollouts on these easier problems do not work well in OPD.

We further added a 8-shot experiment using length-based selection across three independent random seeds. We compare these new results against the random and difficulty-based (ours) selection in Table 12. To ensure a fair comparison with our selection, we restricted the sampling pool for the length-based baseline to the top 17.5% longest rollouts, aligning exactly with the proportion of hard problems in the dataset. We then randomly sampled 8 examples from this pool for each seed. The average accuracy over six mathematical benchmarks are reported below. The results indicates that our difficulty-based selection is superior.

Selection StrategyAverage Accuracy
Random51.6% ± 0.5%
Length-based52.5% ± 0.3%
Difficulty-based (Ours)53.6% ± 0.3%
AuthorsOpenReview ↗

Response to Reviewer FiwE – Part 2

W3: To determine whether the model improves at detecting and correcting mistakes, we evaluate it on CriticBench [3]. We have incorporated this new evaluation and the corresponding analysis into Section 4.2 and Appendix B.3 of the revised manuscript (Page 8, highlighted in blue). This benchmark comprises 3,825 responses across five domains and is explicitly designed to assess LLMs' reflection capabilities. It focuses on two primary dimensions: (a) Critique Score, which evaluates whether the target model can accurately judge the correctness of given responses; and (b) Correction Score, which measures the model's accuracy when correcting those flawed responses.

We present the evaluation results of the base Student, the Teacher, and the models distilled on one examples of varying difficulties (π178\pi_{178} for easy, π794\pi_{794} for medium, and π954\pi_{954} for hard):

ModelCritique ScoreCorrection Score
Student63.041.5
Teacher69.851.2
π178\pi_{178} (Easy)65.043.8
π794\pi_{794} (Medium)66.946.3
π954\pi_{954} (Hard)68.750.1

As shown in the table, the model achieves higher Critique and Correction scores when distilled on harder problems, indicating that conducting OPD on harder problems indeed enables the model to better learn reflective reasoning patterns.

[1] Yuyang Wu, et al. When more is less: Understanding chain-of-thought length in llms. ICLR 2026.

[2] Haoran Zhao, et al. Let LRMs Break Free from Overthinking via Self-Braking Tuning. NeurIPS 2026.

[3] Zicheng Lin, et al. CriticBench: Benchmarking LLMs for Critique-Correct Reasoning. ACL 2024.

Thanks for the rebuttal, I have raised my score to 3.

Reviewer_nED6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This is an empirical study of the data side of on-policy distillation, with four findings. First, 1-shot OPD works: distilling DeepSeek-R1-Distill-Qwen-1.5B on a single example lifts the six-benchmark average from 42.4 to as high as 51.7, against 53.7 for the full 17K DAPO-Math set, and training stays stable to 2,000 steps without collapse. Second, harder examples consistently beat easy and medium ones, including "unsolvable" examples whose training accuracy stays at 0% throughout, which is the opposite of GRPO where zero advantage yields no gradient. Third, the driver is CoT length rather than token entropy: under a length-controlled design where the rollout cap forces over 99.5% of generations from both problems to the same length, the easy/hard gap collapses to 0.5% and −0.9%, and artificially raising entropy by temperature on an easy problem does nothing. Longer CoTs also keep token-level KL lower over long horizons and surface reflection tokens such as "Alternatively" that are absent from short-CoT models. Fourth, eight selected hard examples reach 53.6 against 53.7 for the full 17K set, replicated across five teacher-student pairs spanning 1.5B to 8B and two backbones.

Critical strengths
  1. The length-controlled experiment is the best-designed causal test in this submission and, to my mind, the paper's real contribution. Capping the maximum rollout length so easy and hard problems emit essentially the same token count, then showing the gap collapses to within one point, separates difficulty from length cleanly and rules out the token-entropy explanation that the field would otherwise reach for. The temperature-based entropy manipulation is a sensible second control on the same question.

  2. "Eight examples match seventeen thousand" is a consequential claim, and it is checked across five teacher-student pairs rather than one lucky configuration, with a code-generation extension as well. If it holds it changes how people budget data collection for OPD in specialised domains, which is the motivation the paper opens with.

Critical weaknesses
  1. The N=1 and N=4 training sets are selected on the evaluation benchmarks. Section 5.1 reuses examples already scored individually in Table 1, and the choices are exactly the argmax: π973 is the single best-performing example of all 24 tested (51.7), and {π874, π890, π954, π973} are precisely the top four hard examples by measured benchmark average. The 1-shot and 4-shot headline numbers are therefore a maximum over 24 measured candidates, presented as the output of a difficulty-based selection rule. Only N=8, which uses all eight hard examples, avoids this.

  2. The framing overstates what selection buys, and one counterexample goes unflagged. A random 1K subset also scores 53.6, identical to the eight hard examples, and random-8 reaches 52.1 against 53.7 for the full set. The honest reading of Table 2 is that OPD is remarkably insensitive to data quantity, with hard-selection worth roughly 1.5 points over random at N=8. That is a smaller claim than the one made, and arguably a more interesting one. The main tables carry no error bars, and the "8 ≈ full set" claim visibly fails for the two base-model pairs in Table 3 (21.3 vs 22.5, and 29.2 vs 30.8), which is not discussed.

  3. The control implied by the paper's own mechanism is missing. If length rather than difficulty is what matters, the direct recipe is to select by CoT length, which needs no teacher rollouts and no difficulty estimate at all. Without that comparison the practical recommendation ("select hard") is not shown to beat the simpler and much cheaper alternative ("select long"), and the selection cost of 16 rollouts each from student and teacher over the candidate pool is not justified.

Questions for authors
  • Does selecting the eight longest-CoT examples, regardless of difficulty, match the eight hard ones?
  • For the "unsolvable" examples stuck at 0% training accuracy, is the teacher's own accuracy also 0 on those problems? If so, what distinguishes its KL target from an easy problem's?
  • The primary teacher (JustRL-DeepSeek-1.5B) is the same size as the student, and two of the five pairs are same-size GRPO'd siblings. Do the conclusions hold at a genuinely wide teacher-student capability gap?
3 comments

We sincerely thank you for the constructive feedback. Below, we address your specific questions.

W1: To address the concern about potential selection bias and provide a more comprehensive evaluation, we supplement our study with expected performance metrics based on random sampling from the Hard example pool. Specifically, for N=1N=1, we calculate the expected performance (mean and standard deviation) utilizing the 8 individual hard examples already evaluated in Table 1. For both N=4N=4 and N=8N=8, we calculate the expected performance based on the experiments conducted across 3 independent random seeds, where a distinct subset of 4 or 8 hard problems was randomly sampled for each trial. The detailed evaluation results for each seed have been added to Appendix D (highlighted in blue text).

Training Size (Randomly Selected Hard)Benchmark Performance (Mean ± Std)
1-Shot OPD50.9% ± 0.4%
4-Shot OPD52.7% ± 0.2%
8-Shot OPD53.6% ± 0.3%

As the above results show, the consistently small standard deviations across all settings demonstrate that our difficulty-based selection method is stable and robust to sampling randomness.

W2: We agree with the reviewer's perspective that the underlying phenomenon revealed by our experiments is OPD's remarkable insensitivity to data quantity. This high data efficiency is a main observation we highlight in the paper, such as the 1-shot experiments in Section 3 demonstrating that OPD requires extremely minimal data. Regarding the statistical significance of the performance gap, we conducted the 8-shot experiments across 3 independent random seeds as detailed in Table 12 of Appendix D. The results are summarized below:

Selection StrategyMath Benchmark Avg (Mean ± Std)
Random-851.6% ± 0.5%
Hard-8 (Ours)53.6% ± 0.3%
Full-Set (17K)53.7%

The small standard deviations demonstrate that the 1.5 to 2.0 point improvement over random selection is stable and statistically significant. While OPD is naturally data-efficient, difficulty-based selection reliably bridges the final performance gap to match the 17K full-set OPD.

We also thank the reviewer for pointing out the performance gap for the Qwen3-Base models in Table 3, specifically for Qwen3-1.7B-Base (21.3 vs. 22.5) and Qwen3-4B-Base (29.2 vs. 30.8). This discrepancy stems from the difference between Base (Qwen3-1.7/4B-Base) and Instruct/Chat models (DeepSeek-R1-Distill-Qwen-1.5/7B). Because Instruct models already possess formatting and instruction-following skills from Supervised Fine-Tuning (SFT), a small amount of data data is needed to effectively activate their reasoning during OPD. Conversely, Base models need simultaneously learn complex reasoning patterns and basic output formatting. This explains why 8 examples fall slightly short of full-set performance for Base models. We explicitly explain these mentioned counterexamples in Appendix F (highlighted in blue text).

W3&Q1: To address this concern, we have included a length-based selection baseline. The detailed experimental results and analysis have been incorporated into Section 5.1 and Appendix D of the revised manuscript (highlighted in blue). Specifically, we conducted an 8-shot experiment comparing length-based selection against our difficulty-based method across three independent random seeds. As calculated from the multi-seed results in Appendix D, we found that while length-based selection (averaging 52.5%) provides a noticeable improvement over random sampling (51.6%), it consistently falls short of our difficulty-based selection (53.6%).

To understand this performance gap, we further analyzed the long rollouts from easy and medium problems. These lengthy responses primarily suffer from "overthinking", generating redundant tokens and self-hesitation rather than meaningful reasoning steps. Such redundant patterns fail to provide effective supervisory signals for the student model. Our difficulty-based selection successfully bypasses this token bloat to capture deeper and high-quality reasoning processes.

Q2: As indicated in Table 10, the Teacher model itself struggles on these specific "unsolvable" examples. For π874\pi_{874}, the Teacher achieves a marginal accuracy of 6.3%, while for the other examples, its accuracy remains at 0%.

The distinction between the KL target of these hard problems and that of easy problems lies entirely in the underlying reasoning patterns. When tackling a hard problem, even if the Teacher ultimately fails to reach the correct answer, its probability distribution remains superior. The Teacher's distribution can better capture advanced cognitive behaviors: exploring multiple pathways, generating hypotheses, and exhibiting structural reflections (such as braking or attempting to self-correct). In contrast, an easy problem typically elicits a straightforward deduction, yielding a KL target with shallow supervisory signals. Fitting the KL target of a hard problem forces the student to align with the Teacher's superior problem-solving patterns, then teaching the student how to think, explore, and reflect when faced with difficulty. As demonstrated in Figure 2, the student successfully internalizes these advanced thinking patterns. This explains why OPD remains effective even on these 'unsolvable' problems.

Q3: We have added a new experiment in Section 5.2 to investigate whether the conclusions hold at a genuinely wide teacher-student capability gap. Specifically, we introduce a new teacher–student pair, Qwen3-14B → Qwen3-1.7B-Base, which features a ~8× parameter gap and a correspondingly large capability gap. As shown in the new results below, 8-shot OPD on selected hard examples outperforms the Random baseline and closely approaches Full-Set OPD, even though the teacher is far larger and stronger than the student.

Model / MethodSizeAIME24AIME25AMC23MATH500OlympiadMinervaAvg
Qwen3-1.7B-Base (student)--1.42.512.313.34.83.76.3
Qwen3-14B (teacher)--31.022.871.390.551.231.649.7
Full-Set17K9.85.629.755.124.614.823.3
Random85.94.625.147.222.012.319.5
{π874,…,π997}\{\pi_{874}, \dots, \pi_{997}\}89.65.328.953.324.114.422.6

Thank you for the authors' response. I will maintain my positive rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.