- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents an empirical study of data efficiency and data selection in On-Policy Distillation (OPD). It finds that OPD training only on 1 data is consistently effective across benchmarks when scaling training steps. In addition, this paper finds that OPD training on hard problems and longer CoT help better. Experimental results across model sizes, model families, and task domains further demonstrate the empirical findings.
Critical strengths
- A good perspective in OPD area and easy to follow.
- All findings are interpretable with experiments.
- Findings can generalize across model sizes, model families, and task domains.
Critical weaknesses
- My biggest concern is whether these findings can truly guide model training. The experiments in this paper are limited to no more than a few dozen training examples, but it is an undeniable fact that the larger the training dataset is, the stronger the model is. It is worth investigating whether using such a small amount of training data might affect other capabilities of the model itself. Could you provide me with OOD experiment results using N-shot OPD and Full OPD?
- This paper reports that "Distillation Benefits from Longer CoTs," but it is unclear whether this observation is specific to the N-shot OPD settings or represents a more general scaling trend. In particular, the effect of response length may change when training is performed with the full dataset. It would strengthen this claim to include Full OPD experiments across different training response length settings and examine whether the same trend persists.
Questions for authors
Please refer to the Weaknesses.
We sincerely thank you for the constructive feedback. Below, we address your specific questions.
W1: By investigating the distillation performance from a 1-shot OPD perspective, our study aims to better understand the underlying role of data selection. We hope these findings can serve as a helpful reference for practical data curation pipelines. Our observations indicate that in the OPD phase, paying closer attention to data quality, particularly problem difficulty, has the potential to bring meaningful efficiency improvements. Rather than simply scaling up data quantities, which often incurs considerable overhead for data synthesis, curating a smaller subset of challenging problems may offer a more economical pathway to attain competitive reasoning capabilities, thereby alleviating the resource burden associated with large-scale data collection. We have incorporated these insights into the Section 7 of our revised manuscript (highlighted in blue).
Regarding the concern about OOD performance and other potential side effects of small data volumes, we demonstrated in Appendix B.1 that the reasoning capabilities acquired through 1/N-shot OPD purely in the mathematical domain can successfully generalize to coding and scientific domains. Specifically, we evaluated the models on LiveCodeBench (coding reasoning) and GPQA (scientific reasoning). As shown in Table 6 (for 1-shot OPD) and Table 7 (for N-shot OPD), 1/N shot OPD consistently achieves strong OOD performance, with OPD training on 8 hard problems setup nearly matching the generalization capabilities of the Full-Set OPD baseline.
Furthermore, we have added a new experiment in non-reasoning domains, with the detailed setup and results incorporated into Appendix B.2 of the revised manuscript (highlighted in blue). We evaluated general linguistic and creative abilities using the Creative Writing v3 benchmark [1]. We utilized DeepSeek-V4-Pro as the judge model to evaluate the Rubric Score (comprehensive overall writing quality) and Fine-grained Ability Scores (covering distinct creative writing skills such as coherence, descriptive imagery, and instruction following). The evaluation results demonstrate that applying our few-shot OPD method does not compromise the semantic richness or natural language fluency of the outputs.
W2: To investigate whether the advantage of longer CoTs generalizes beyond the N-shot setting, we conducted additional Full-Set OPD experiments across different training response lengths, which have now been added to Appendix C.5 of the revised manuscript (highlighted in blue). Specifically, we trained the model on the full dataset with maximum rollout lengths truncated at 1K, 2K, 4K, and 7K, respectively. As the results show, the model's performance consistently improves as the rollout length increases from 1K to 7K. This trend also aligns with similar OPD observations reported in [2].
[1] Samuel J Paech. Eq-bench creative writing benchmark v3. https://github.com/EQ-bench/creative-writing-bench, 2025.
[2] Yaxuan Li et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe. arXiv preprint arXiv:2604.13016.