acceptodds
Under review as a conference paper at ICLR 2027

Data-Free On-Policy Distillation: How Far Can We Go Without External Data?

61 views

Abstract

On-policy distillation (OPD) is increasingly applied to frontier foundation model post-training. Prior work in this area has largely focused on algorithmic advances, yet it remains unclear how much OPD depends on its training questions and, in particular, how far this dependence can be reduced. Across two representative single-teacher OPD settings, we find that training on eight real prompts yields performance comparable to training on 17k problems, while datasets differing substantially in measured difficulty and initial distillation gap yield similar outcomes. Our analyses suggest two complementary explanations: repeated sampling could allow even a few prompts to expose substantial teacher supervision, while OPD transfers generalizable reasoning patterns beyond question-specific knowledge. Building on these observations, we next propose a data-free on-policy distillation (DF-OPD) setting to investigate whether the system can supply the training questions itself, eliminating the need for external data. With 64 self-generated questions obtained without seed examples, DF-OPD yields performance comparable to full-data OPD in both single-teacher settings. This finding also holds in multi-teacher OPD: across mathematics, code, and instruction following, 1k generated questions achieve performance comparable to training on approximately 7k real post-training examples. We further explore whether OPD can operate even without explicit training questions. The experiments show that this is effective only in limited cases, where the student unexpectedly generates and answers its own questions, thereby reducing the process to an implicit form of DF-OPD. Together, these findings invite a reassessment of the role of training data in on-policy distillation. Code is available at https://anonymous.4open.science/r/XE8EE.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Data Efficiency, Synthetic Data Generation, Multi-Teacher On-Policy Distillation, Large Language Models, Post-Training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7AdLOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This work dives deep into how training data, specifically the training questions, affects the final performance of OPD. They conduct comprehensive and interesting experiments with the number of training questions and teacher generated questions. Their findings, including OPD performance can be mostly achieved using 8 questions, teacher generated questions can obtain on-par performance with annotated questions, are very interesting and provide novel perspectives into how OPD data should be curated.

Critical strengths

1, The findings from this work are novel and interesting. I believe they are great contributions to the community.

2, Besides empirically answering research questions, this work also conducts reasonable investigation into the observations, which provides preliminary insights and encourages further studies.

3, The manuscript is well-written and easy to follow.

Critical weaknesses

1, The critical weakness, as acknowledged in the conclusion section, is the scale of the experiments. It is not guaranteed that the conclusions could straightforwardly generalize to larger models. However, considering the complexity of such experiments, these limitations are acceptable; and the work still provides meaningful insights.

Questions for authors

From my understanding, the 8 samples in the THE NUMBER OF TRAINING QUESTIONS section and 64 samples in DF-OPD are randomly sampled. It's actually surprising to me that without any advanced sampling heuristics, those few samples could already recover the OPD performance on the whole set.

That said, I am wondering how the random seed would affect the results? Could you try some other random seeds, and see if the conclusions will change?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 7AdL (1/2)

Dear Reviewer 7AdL,

Thank you very much for your positive assessment and strong support of our work, and for recognizing the novelty of our findings, our mechanistic investigations, and our contributions to the community! Below, we address your question on random-seed sensitivity by evaluating independently sampled, mutually disjoint question pools as well as independent training seeds, further corroborated by evaluating the impact of different checkpoint-selection rules on the results (Q1), and briefly discuss our ongoing efforts toward larger model scales (W1). The corresponding results and analyses are highlighted in blue in Sections 3.1 and 4.2 and Appendices G.1 and G.2 of the revised manuscript.


Q1. How does the random seed affect the results? Could you try some other random seeds, and see if the conclusions will change?

Thank you for raising this important question! As you noted, our small question pools are drawn purely at random without heuristic selection. Following your suggestion, to test whether our conclusions depend on a fortunate random draw—or on the random seed used during training—we repeated the two key small-pool and DF-OPD conditions (Real, |D|=8 and Self-generated, |D|=64 in the two single-teacher settings), as well as Self-generated, |D|=1k in the multi-teacher setting (MOPD), along two independent random-seed axes:

  • Varying question-sampling seed: three runs on mutually disjoint question pools randomly sampled without replacement using seeds 42, 43, and 44 (with 42 as the paper default) from the same source (DAPO-17k for real questions, and the teacher-generated corpus for self-generated ones, with domain proportions held fixed in MOPD), while keeping the training seed fixed at the default.

  • Varying training seed: three runs on the original question pool using different random seeds (0, 1, and 2, with 0 as the paper default) for student trajectory sampling, which affects the sampled on-policy rollouts and in turn alters the teacher supervision computed along them at every step. (We vary this seed rather than the data-loader seed because in the single-teacher settings, each update of 64 questions (batch size = 64) already covers the entire 8- or 64-question pool, so data-loader shuffling would not change which questions enter a batch.)

Sharing the original paper run, each condition comprises five distinct runs. Table R1 below reports the paper score, the mean ± std (with all three runs in parentheses) along each axis, and the full real-data baseline:

SettingQuestion poolPaper scoreVarying poolVarying seedFull real data
RL-tuned teacherReal, |D|=856.3356.61 ± 0.57 (56.33, 56.23, 57.27)57.12 ± 0.69 (56.33, 57.53, 57.51)56.41
(init 47.08, teacher 61.44)Self-generated, |D|=6456.6156.95 ± 0.41 (56.61, 57.40, 56.85)56.58 ± 0.33 (56.61, 56.23, 56.89)56.41
Larger teacherReal, |D|=847.7348.02 ± 0.43 (47.73, 48.52, 47.82)47.93 ± 0.19 (47.73, 47.95, 48.11)47.36
(init 34.49, teacher 75.46)Self-generated, |D|=6449.1649.76 ± 0.52 (49.16, 50.02, 50.10)48.70 ± 0.48 (49.16, 48.21, 48.74)47.36
Multi-teacher (init 24.21, teachers 31.42)Self-generated, |D|=1k31.3131.21 ± 0.21 (31.31, 30.97, 31.36)31.28 ± 0.13 (31.31, 31.39, 31.14)31.18

Two clear takeaways emerge from Table R1:

  • Varying the question-sampling seed does not change the conclusions: Across three mutually disjoint question pools drawn purely at random without heuristics, both Real, |D|=8 (56.61 ± 0.57 vs. 56.41 for the RL-tuned teacher, and 48.02 ± 0.43 vs. 47.36 for the larger teacher) and Self-generated, |D|=64 (56.95 ± 0.41 vs. 56.41, and 49.76 ± 0.52 vs. 47.36) consistently match the 17,917-question baseline, while Self-generated, |D|=1k in MOPD (31.21 ± 0.21) matches the 7,458-question baseline (31.18), with relatively small standard deviations throughout (0.21–0.57 points).

  • Varying the training seed likewise yields stable results: Across independent rollout-sampling seeds on the same question pool, both Real, |D|=8 (57.12 ± 0.69 vs. 56.41, and 47.93 ± 0.19 vs. 47.36) and Self-generated, |D|=64 (56.58 ± 0.33 vs. 56.41, and 48.70 ± 0.48 vs. 47.36) again stay on par with full-data OPD, as does Self-generated, |D|=1k in MOPD (31.28 ± 0.13 vs. 31.18), with similarly low standard deviations (0.13–0.69 points).

Together, these results confirm that small-pool OPD and DF-OPD do not rely on a fortunate question draw or training seed.

AuthorsOpenReview ↗

Response to Reviewer 7AdL (2/2)

Additional robustness check across checkpoint-selection rules.

To further verify that our small-pool OPD and DF-OPD results are also insensitive to the choice of checkpoint-selection rule, we evaluate two additional rules alongside the original one used in our paper:

  • Original: selects the checkpoint achieving the highest overall score on the evaluation benchmarks across training, as used in the paper (Appendix B.3).
  • Final checkpoint: directly reports the last checkpoint at the end of training, with no checkpoint selection.
  • Held-out val: selects the checkpoint scoring highest on an independent validation set formed by holding out a random 20% of the real training corpus (3,583 questions in the single-teacher settings and 1,492 stratified by domain in MOPD), sampled to avoid any overlap with the small pools and disjoint from all evaluation benchmarks. To prevent any overlap with this validation set, the full-data baseline is retrained on the remaining 80% questions.

Table R2 below reports performance under all three rules for our key configurations:

SettingQuestion poolOriginalFinal checkpointHeld-out val
RL-tuned teacherReal, |D|=17,917 (full data)56.4156.05—
(init 47.08, teacher 61.44)Real, |D|=14,334 (full data minus held-out val)56.3955.7256.39
Real, |D|=856.3355.7756.33
Self-generated, |D|=6456.6156.5356.61
Larger teacherReal, |D|=17,917 (full data)47.3646.61—
(init 34.49, teacher 75.46)Real, |D|=14,334 (full data minus held-out val)47.5747.0747.25
Real, |D|=847.7347.6147.56
Self-generated, |D|=6449.1649.0449.04
Multi-teacherReal, |D|=7,458 (full data)31.1830.53—
(init 24.21, teachers 31.42)Real, |D|=5,966 (full data minus held-out val)31.2430.3931.24
Self-generated, |D|=1k31.3131.3131.31

As shown in Table R2, both small-pool OPD and DF-OPD remain comparable to the full-data baselines whether evaluated directly at the final checkpoint (55.77 / 56.53 vs. 56.05, 47.61 / 49.04 vs. 46.61, and 31.31 vs. 30.53) or selected via held-out validation (56.33 / 56.61 vs. 56.39, 47.56 / 49.04 vs. 47.25, and 31.31 vs. 31.24), further confirming the stability of our findings.


W1. Generalization to larger model scales.

Thank you very much for your kind understanding of the complexity of such experiments! We fully agree that testing larger student models to expand the scale of our experiments is an important direction. We are actively running experiments with larger student models (e.g., 7B) during the discussion period. However, because multi-step on-policy rollout generation and multi-benchmark evaluation across pool sizes are computationally intensive, they may not finish before the discussion period ends. Should we unfortunately be unable to share them in time during the discussion phase, we commit to including results on at least one to two larger student models in the final version of the paper!


Thank you once again for your generous and encouraging review! Your feedback has not only strengthened our work, but also motivated us to keep moving forward on our research. Please let us know if you have any further questions, and we would be more than happy to address them! :-D

Best regards,

Authors of Paper 36946

I thank the authors for their detailed responses, and decide to keep my positive rating.

Reviewer_1pUEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to answer this question: as on-policy distillation (OPD) is widely used in post-training, how much its performance depends on its training questions is uncertain. In this paper, authors study this in both single-teacher and multi-teacher settings, across various domains including math, code and instruction following. The main findings is that a handful of real questions performs nearly equally well to the full 17k-question DAPO set. Besides, the data sources with different difficulty and teacher-student gap produce similar results. Authors attribute this into two factors: 1. the sampling process during training already exposes most of the signal from the teacher even from only few questions. and 2. gains transfer across domains. Based on these observation, they propose DF-OPD, where the questions are generated from the teacher. Experimental results show that even with 64 generated questions, DF-OPD matches full-data OPD in the single-teacher settings, and 1k generated questions match 7.5k real ones in the multi-teacher setting.

Critical strengths
  1. The research question is worth-exploring and practical, as OPD is increasingly used in post-training. Know how it work from the perspective of data helps the community better understand the important field.
  2. The design is sound. The experiments are conducted on multiple teacher-student pairs and various data resources, which makes the observation genelizable and solid.
  3. The proposed method is simple and potentially useful, especially when the data is limited.
Critical weaknesses
  1. The novelty is somewhat limited. The paper notes concurrent work reporting OPD’s limited sensitivity to training data, including Fu et al. (2026b), which uses a single training example and from which the empty-prompt setup is adopted. Similar few-example findings exist for SFT (LIMA, s1, LIMO) and RL (1-shot RLVR). Section 3 therefore largely confirms known findings, and DF-OPD applies Self-Instruct-style question generation to OPD. A direct comparison with Fu et al. (2026b) would clarify the contribution.
  2. The source of the gain might be confounded. Given these findings: 1. code-to-math transfer retains 92% to 96% of the gains; 2. the OPD performance is insensitive to data difficulty; and 3. empty prompts suffice for one student. The gain might come from the generic behavioral changes, e.g. response length, answer format, rather than the transfer of reasoning patterns. R1-Distill-Qwen-1.5B in particular is known for overlong outputs.
  3. In the RL-tuned setting, the teacher is an RL-tuned version of the student. Prior work on RL (e.g., 1-shot RLVR) suggests such a small RL delta can be recovered from very few examples.
Questions for authors

How do response length, truncation rate, and answer format change during OPD, and how much of the gain remains with prompts unrelated to reasoning?

9 comments
AuthorsOpenReview ↗

Response to Reviewer 1pUE (1/6)

Dear Reviewer 1pUE,

Thank you very much for your careful and constructive review, and for recognizing the practical value of our research question, the soundness of our experimental design, and the simplicity and usefulness of our proposed method! Below, we address each of your concerns and questions with direct comparisons with Fu et al. (2026b) and prior few-shot SFT/RLVR and Self-Instruct studies (W1), diagnostics on length, truncation, and answer format plus robustness checks of our core findings (W2 & Q1, Part 1), controlled comparisons using training questions unrelated to reasoning (W2 & Q1, Part 2), and analyses of the capability gap in the non-RL-tuned Larger-teacher setting, comparisons with few-shot RLVR on identical examples, and parameter-update directions (W3). The corresponding discussions and new results are highlighted in blue in Section 6 and Appendices G.3, H.2–H.3, and J.2–J.3 of the revised manuscript.


W1. A direct comparison with Fu et al. (2026b), as well as prior few-shot SFT/RLVR and Self-Instruct-style studies, would clarify the contribution.

Thank you for this constructive suggestion, which gives us a great opportunity to clarify the differences and highlight our unique contributions! Fu et al. (2026b) is a concurrent work posted on arXiv in September, just a few weeks before the submission deadline. Prior to its release, we had already independently conducted this research and completed all the core analyses, methodology design, and experiments of our paper (those in Sections 3–4). Upon seeing their preprint, we not only added a discussion in Section 6, but also further investigated the empty-prompt setup in Section 5—clarifying its applicability boundary and underlying mechanism by showing that it does not hold universally, and that its effectiveness essentially stems from the student spontaneously exhibiting Self-QA, which constitutes an "implicit DF-OPD." We fully agree that a more direct comparison is valuable. Following your suggestion, Table R1 systematically compares the two works across four dimensions: core research question, breadth of study, mechanistic analysis, and depth of study:

DimensionSpecific aspectFu et al. (2026b) (concurrent work)Ours
1. Core questionResearch goalWhy OPD can train effectively on a few questions while maintaining comparable performanceHow far OPD's dependence on external training data can be reduced, whether the system can be self-sustaining (Data-Free), and where this boundary lies
2. Breadth of studyTeacher–student pairingsWhile covering both single-teacher and MOPD, all teachers are same-scale post-trained variants from the same base model, lacking a cross-scale Larger-teacher settingCovers not only single-teacher (RL-tuned) and MOPD, but also a 17× larger cross-scale Larger-teacher (30B → 1.7B) not fine-tuned from the student
Data properties examinedExamines question quantity, as well as single-question difficulty, length cap, and temperatureBeyond question quantity, systematically examines dataset-level difficulty and the OPD-specific initial distillation gap (varying by nearly 3× across 3 sources)
3. Mechanistic analysisCoverage metricState coverage: clusters only the teacher hidden states hT(s)h_T(s) at visited states, capturing which states are visited without accounting for the teacher–student correction signal at those statesSupervision signal coverage: jointly encodes state representation hsh_s and local correction direction gsg_s, directly measuring effective supervision signals that drive parameter updates and correlating strongly with downstream gains throughout training (r = 0.94, see our response to Reviewer iTKJ W2)
Cross-source analysisTests WildChat on R1-Distill-1.5B, arguing that task content can be decoupled from state coverageDemonstrates across both RL-tuned and Larger-teacher settings via code → math transfer that OPD can transfer reasoning capabilities beyond dataset-specific knowledge
4. Depth of studyEliminating external dataDoes not explore a broadly effective approach to eliminating external training dataProposes DF-OPD: zero-shot self-generated questions (64 in single-teacher / 1k in MOPD) match full real data across settings without reference solutions or correctness filtering
Removing explicit questionsTested only on a single student (R1-Distill-1.5B) without identifying its applicability boundaryTested across 3 distinct student-teacher pairs and verified via within-model controlled interventions (see our response to Reviewer iTKJ Q3), showing it does not hold generally and relies on the student spontaneously producing Self-QA (implicit DF-OPD)
AuthorsOpenReview ↗

Response to Reviewer 1pUE (2/6)

As shown in Table R1, whereas concurrent work focuses on explaining the few-shot phenomenon, our work centers on exploring the limits of reducing OPD's reliance on external training data—progressing step by step from How Sensitive Is OPD to Its Training Data? →\to Can Self-Generated Questions Replace External Data? →\to Can OPD Work Without Explicit Training Questions?, making a distinct and systematic contribution that offers a fresh perspective on the role of data in OPD.

Furthermore, regarding the prior few-shot SFT/RLVR and Self-Instruct studies you noted, our core distinctions are as follows:

  • Substantive contributions of few-shot OPD (Section 3) beyond few-shot SFT (LIMA, s1, LIMO) and RL (1-shot RLVR):

    • Compared with few-shot SFT (LIMA, s1, LIMO): These offline SFT studies rely on pairing prompts with high-quality reference responses/solutions and emphasize carefully curating ~1k examples from large pools (e.g., s1 and LIMO explicitly show that random sampling or lower-difficulty questions substantially degrade SFT performance). In stark contrast, our work reveals that few-shot OPD requires neither reference solutions nor heuristic curation, matching 17.9k full data with just 8 purely random real questions (or even 64 self-generated questions).
    • Compared with 1-shot RLVR: RLVR relies on verifiable ground-truth answers, requires the initial student to already be able to sample correct solutions (otherwise outcome rewards stay zero), and—without an external teacher—mainly sharpens capabilities within the student's own sampling distribution. In contrast, by leveraging dense token-level distribution supervision from a teacher, OPD requires no ground-truth answers and is not constrained by whether the student can solve the training problems on its own: it transfers capabilities well beyond the initial student's pass@16 limit in the Larger-teacher setting (75.7% vs. 58.3%, see our response to W3 below), and remains effective where few-shot RLVR struggles—as shown in Table R4 below, on R1-Distill-1.5B using the exact same 1 and 8 random questions, the official 1/8-shot RLVR recipe improves accuracy by only +0.67 / +2.33 points over 500 steps, whereas 1/8-shot OPD achieves +5.76 / +9.25 points in just 180 steps.
  • Fundamental distinction between DF-OPD and Self-Instruct: Offline synthetic-data pipelines (such as Self-Instruct and Evol-Instruct) start from an initial seed pool for few-shot bootstrapping or multi-round instruction evolution (coupled with similarity or heuristic filtering) and must offline-generate complete reference responses to construct (instruction, response) pairs for SFT. In DF-OPD, however, the system requires no external seed examples, no generated reference solutions, no correctness/difficulty verification, and no elaborate prompt engineering—just 64 (single-teacher) or 1k (MOPD) unverified questions generated from simple prompts suffice to replace 17.9k or 7.5k real questions, and even completely removing topic guidance from the generation prompts changes performance by only -0.60 to +1.50 points (Appendix E.6).

AuthorsOpenReview ↗

Response to Reviewer 1pUE (3/6)

W2 & Q1 (Part 1). How do response length, truncation rate, and answer format change during OPD? Do changes in these behaviors affect the key findings?

Thank you for raising this insightful question, which helped us further verify the reliability of our conclusions! Following your suggestion, we conducted sample-level diagnostics across our evaluation generations and supplemented them with a 32k de-truncated re-evaluation and budget forcing. Our results show that all of our core findings—the effectiveness of few-shot real pools (Real, |D|=8) and self-generated questions (Self-generated, |D|=64), the relative insensitivity to dataset difficulty and initial distillation gap (DeepMath and ORZ, |D|=1k), and the transfer of reasoning capability beyond dataset-specific knowledge (DeepCoder, |D|=1k)— still hold as these behaviors change. Specifically, our evaluation setup consists of two parts:

  • Answer-format, response-length, and truncation diagnostics: First, we inspect all evaluation trajectories to measure the fraction of naturally finished (non-truncated) responses that lack an extractable answer (No-Ans | Fin), the mean response length (in tokens), and the truncation rate at the default 16k generation cap (Trunc (16k)).
  • 32k de-truncated re-evaluation and budget forcing (BF): Second, to test whether our core findings rely on the response length and truncation rate, we double the default evaluation cap from 16k to 32k tokens. Furthermore, to ensure truncation effects are completely removed, we also evaluate 32k + BF following the budget forcing (BF) protocol of s1 (Muennighoff et al., 2025): for any sample still unfinished at 32k tokens, we append an answer-prompting suffix (</think>\n\n**Final Answer**\n\boxed{) and greedily generate up to 64 additional tokens to force an answer.

Table R2 summarizes the behavioral metrics alongside the macro-average accuracy at 16k, 32k, and 32k + BF (with the relative gain ratio over Full DAPO under each evaluation setting, Acc−InitAccFull−Init\frac{\text{Acc} - \text{Init}}{\text{Acc}_{\text{Full}} - \text{Init}}, in parentheses):

SettingQuestion poolNo-Ans | Fin (16k)Mean len (16k)Trunc (16k)Trunc (32k)Trunc (32k+BF)Acc (16k) (vs. Full)Acc (32k) (vs. Full)Acc (32k+BF) (vs. Full)
RL-tuned teacherInitial student0.56%7,42517.9%3.6%0.22%47.08 (—)48.76 (—)48.79 (—)
(R1-Distill-1.5BFull DAPO (|D|=17,917)0.02%5,4822.5%1.1%0.16%56.41 (100.0%)56.44 (100.0%)56.45 (100.0%)
←\leftarrow JustRL-1.5B)Real, |D|=80.03%5,4713.6%1.2%0.16%56.33 (99.1%)56.37 (99.1%)56.38 (99.1%)
Self-generated, |D|=640.00%5,5613.2%1.8%0.16%56.61 (102.1%)56.67 (103.0%)56.69 (103.1%)
DeepMath, |D|=1k0.07%6,1945.3%1.1%0.25%57.52 (111.9%)57.58 (114.8%)57.59 (114.9%)
ORZ, |D|=1k0.03%6,4035.2%0.8%0.08%58.28 (120.0%)58.29 (124.1%)58.29 (124.0%)
DeepCoder, |D|=1k0.05%5,2223.4%0.4%0.03%56.99 (106.2%)57.06 (108.1%)57.06 (108.0%)
Larger teacherInitial student0.00%1,3990.7%0.6%0.09%34.49 (—)34.63 (—)34.64 (—)
(Qwen3-1.7BFull DAPO (|D|=17,917)0.15%7,12128.1%16.4%0.13%47.36 (100.0%)51.16 (100.0%)52.67 (100.0%)
←\leftarrow Qwen3-30B-A3B)Real, |D|=80.11%6,89825.8%13.6%0.03%47.73 (102.9%)52.20 (106.3%)53.17 (102.8%)
Self-generated, |D|=640.17%6,94325.7%8.5%0.06%49.16 (114.0%)52.95 (110.8%)53.69 (105.7%)
DeepMath, |D|=1k0.12%6,26820.6%4.8%0.08%49.65 (117.8%)51.32 (101.0%)51.46 (93.3%)
ORZ, |D|=1k0.19%6,76424.9%13.2%0.17%48.93 (112.2%)52.78 (109.8%)53.86 (106.6%)
DeepCoder, |D|=1k0.07%6,22621.8%7.6%0.06%46.92 (96.6%)49.81 (91.8%)50.73 (89.2%)

(Note: For DeepCoder, |D|=1k, Section 3.3 of the paper reports 96% and 92% retention relative to the size-matched DAPO, |D|=1k baseline, whereas Table R2 normalizes all conditions against Full DAPO (|D|=17,917) for consistency in this table, yielding 106.2% and 96.6%.)

How do response length, truncation rate, and answer format change during OPD?

  • Under the default 16k evaluation, missing-answer rates among finished responses (No-Ans | Fin) remain near zero (≤ 0.56% before training and ≤ 0.19% after), while mean response length and truncation rate decrease in the RL-tuned setting (7,425 → 5,222–6,403 tokens; 17.9% → 2.5%–5.3%) but increase in the Larger-teacher setting (1,399 → 6,226–7,121 tokens; 0.7% → 20.6%–28.1%).
Reviewer_iTKJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the extent to which On-Policy Distillation (OPD) relies on external training problems. In two single-teacher settings, using as few as 8 real-world problems yields performance comparable to that achieved with approximately 17,000 problems. Furthermore, final results remain consistent across different datasets, even when they vary significantly in terms of measured difficulty and initial distillation gap—defined as the KL divergence between the teacher’s and student’s predicted distributions over trajectories generated by the student. The authors propose Data-Free On-Policy Distillation (DF-OPD): a frozen teacher model generates a fixed set of problems based on simple prompts—without requiring any seed examples—and subsequently provides token-level supervision on trajectories generated by the student, while maintaining the original distillation objective. Across six mathematical benchmarks, using 64 generated problems achieves a peak macro-average accuracy that matches or exceeds the full-data baseline. In multi-teacher experiments covering mathematics, programming, and instruction following, 1,000 generated problems yield a peak macro-average score of 31.31 across the three domains, comparable to the score of 31.18 obtained using 7,458 real-world problems. Analyses of signal coverage and transfer experiments—from programming to mathematics—suggest that repeated sampling of a small number of problems can expose a diverse range of teacher supervision signals, and that OPD may possess the capacity to generalize beyond the specific knowledge contained in the training problems.

Critical strengths

The study deepens the understanding of data requirements for OPD. Experiments involving both teacher and student models demonstrate that as few as 8 authentic examples can achieve distillation results comparable to those obtained using a dataset of approximately 17,000 examples; this finding is further corroborated by cross-data-source comparisons and experiments on transferring knowledge from programming to mathematics. This body of evidence extends beyond isolated few-shot results and constitutes the core contribution of this work.

The paper proposes an effective and well-defined DF-OPD workflow. Teacher-generated examples—produced without the need for seed prompts—achieve performance levels approaching full-dataset baselines, both in single-teacher mathematical tasks and in multi-teacher distillation scenarios spanning three distinct domains. Furthermore, experiments using "empty" prompts reveal that eliminating explicit questions is not a universally effective strategy. These analyses of practical utility and operational boundaries significantly strengthen the paper's contributions.

Critical weaknesses

The core few-shot results lack sufficient robustness verification. The paper reports the checkpoint that achieved the highest macro-average score on the evaluation benchmark (Appendix B.3), yet it fails to provide results from independent training runs, repeated sampling from different small-scale problem pools, or performance confidence intervals. Given that the problem pools contain only 8 or 64 questions, the specific selection of questions could influence the results—an effect that remains unquantified. In the multi-teacher experiments, the performance gap between models trained on synthetic data versus real data is a mere 0.13 points (31.31 vs. 31.18); the current evidence is insufficient to determine whether this proximity is stable across repeated experiments. While matching the training budget is a valid control, it does not eliminate these uncertainties; thus, the conclusion that "small problem pools can reliably serve as a substitute for the full dataset" requires further validation.

The mechanistic analysis fails to establish a causal link. "Signal coverage" is measured only on a frozen, initial student model—without tracking changes throughout the training process or verifying a correlation with downstream performance gains. The analysis of "Self-QA" compares three student models that differ in both their training history and chat templates, making it impossible to isolate the impact of these variables. Without a controlled experiment—such as suppressing or inducing Self-QA within the same model—it is difficult to confirm whether this behavior is indeed the driver behind the success of "null-prompt distillation." While these analyses offer plausible leads, they fall short of establishing the proposed mechanism, thereby limiting the strength of the paper's contributions regarding interpretability.

Questions for authors

Robustness. Were the eight-real-question and 64-generated-question results repeated across independent training seeds and independently sampled question pools? If so, please report the mean and variability across runs. If not, repeating these key conditions would be a targeted robustness check, rather than a broader extension: it would directly affect my confidence that the reported small-pool results are reproducible.

Checkpoint selection. Appendix B.3 reports the checkpoint with the highest macro-average score across the evaluation benchmarks. Do the small-pool and DF-OPD results remain comparable to the full-data baseline at the final checkpoint? Please provide the corresponding scores from the existing runs. Were any results also obtained using an independent validation set for checkpoint selection?

Self-QA interpretation. Is the connection between Self-QA and successful empty-prompt distillation intended as a causal explanation or an observed association? The current comparison involves different students and chat templates. Do you have within-model controls that isolate the effect of suppressing or inducing Self-QA? If not, would you narrow the claim that success “relies on” Self-QA? This clarification would affect my assessment of the mechanistic contribution; a full intervention study could remain future work.

6 comments
AuthorsOpenReview ↗

Response to Reviewer iTKJ (1/4)

Dear Reviewer iTKJ,

Thank you very much for your thoughtful and constructive review, and for recognizing our empirical study of OPD's data requirements, the effectiveness of DF-OPD across single- and multi-teacher settings, and our analysis of empty-prompt boundaries! Below, we address each of your questions and concerns with new multi-seed and disjoint-pool runs (Q1 & W1), final-checkpoint and held-out validation evaluations (Q2 & W1), within-model controlled experiments establishing the causal link between Self-QA and the effectiveness of empty-prompt OPD (Q3 & W2), and dynamic tracking of signal coverage throughout training, verifying its correlation with downstream performance gains (W2). The corresponding revisions and new results are highlighted in blue in Sections 3.1, 3.3, 4.2, and 5.2 and Appendices C.2, F.4, G.1–G.2, and H.1 of the revised manuscript.


Q1 & W1. Were the eight-real-question and 64-generated-question results repeated across independent training seeds and independently sampled question pools? Is the multi-teacher DF-OPD result also stable across repeated experiments?

Thank you for emphasizing the need to verify the stability of our small-pool and DF-OPD results! Following your suggestion, our experiments show that across both independent training seeds and mutually disjoint question pools, small-pool OPD and DF-OPD consistently match the full-data baselines with low variance in all three key settings (Real, |D|=8 and Self-generated, |D|=64 in single-teacher; Self-generated, |D|=1k in multi-teacher (MOPD)), evaluated along two independent axes:

  • Varying training seed: three runs on the original question pool using different random seeds (0, 1, and 2, with 0 as the paper default) for student trajectory sampling, which affects the sampled on-policy rollouts and in turn alters the teacher supervision computed along them at every step. (We vary this seed rather than the data-loader seed because in the single-teacher settings, each update of 64 questions (batch size = 64) already covers the entire 8- or 64-question pool, so data-loader shuffling would not change which questions enter a batch.)

  • Varying question pool: three runs on mutually disjoint question pools randomly sampled without replacement using seeds 42, 43, and 44 (with 42 as the paper default) from the same source (DAPO-17k for real questions, and the teacher-generated corpus for self-generated ones, with domain proportions held fixed in MOPD), while keeping the training seed fixed at the default.

Sharing the paper run, each condition comprises five distinct runs. Table R1 below reports the paper score, the mean ± std (with all three runs in parentheses) along each axis, and the full real-data baseline:

SettingQuestion poolPaper scoreVarying seedVarying poolFull real data
RL-tuned teacherReal, |D|=856.3357.12 ± 0.69 (56.33, 57.53, 57.51)56.61 ± 0.57 (56.33, 56.23, 57.27)56.41
(init 47.08, teacher 61.44)Self-generated, |D|=6456.6156.58 ± 0.33 (56.61, 56.23, 56.89)56.95 ± 0.41 (56.61, 57.40, 56.85)56.41
Larger teacherReal, |D|=847.7347.93 ± 0.19 (47.73, 47.95, 48.11)48.02 ± 0.43 (47.73, 48.52, 47.82)47.36
(init 34.49, teacher 75.46)Self-generated, |D|=6449.1648.70 ± 0.48 (49.16, 48.21, 48.74)49.76 ± 0.52 (49.16, 50.02, 50.10)47.36
Multi-teacher (init 24.21, teachers 31.42)Self-generated, |D|=1k31.3131.28 ± 0.13 (31.31, 31.39, 31.14)31.21 ± 0.21 (31.31, 30.97, 31.36)31.18

Two clear takeaways emerge from Table R1:

  • Small real pools consistently match full-data OPD: Across both single-teacher settings, Real, |D|=8 stays on par with the 17,917-question baseline across both training seeds and disjoint pools (57.12 ± 0.69 and 56.61 ± 0.57 vs. 56.41 for the RL-tuned teacher, and 47.93 ± 0.19 and 48.02 ± 0.43 vs. 47.36 for the larger teacher), with small standard deviations (0.19–0.69 points), confirming that small-pool OPD does not rely on a fortunate training seed or question draw.

  • DF-OPD likewise consistently matches real-data OPD: In the two single-teacher settings, Self-generated, |D|=64 stays on par with full-data OPD across seeds and pools (56.58 ± 0.33 and 56.95 ± 0.41 vs. 56.41 for the RL-tuned teacher, and 48.70 ± 0.48 and 49.76 ± 0.52 vs. 47.36 for the larger teacher), with similarly small standard deviations (0.33–0.52 points). In MOPD, Self-generated, |D|=1k also shows minimal variance across seeds (31.28 ± 0.13) and pools (31.21 ± 0.21), matching the 7,458-question baseline (31.18) and confirming that the close proximity in Table 1 (31.31 vs. 31.18) is stable across repeated experiments.

AuthorsOpenReview ↗

Response to Reviewer iTKJ (2/4)

Q2 & W1. Do the small-pool and DF-OPD results remain comparable to the full-data baseline at the final checkpoint, and when checkpoints are selected using an independent validation set?

Thank you for raising this important point! Yes—across all evaluated pool sizes and settings, evaluating at the final checkpoint or selecting checkpoints via an independent validation set closely preserves the scores and scaling trends reported in the paper, with our key small-pool OPD and DF-OPD configurations remaining comparable to the full-data baselines. Following your suggestion, we evaluate two additional checkpoint-selection rules alongside the one used in our paper across all existing runs:

  • Original: selects the checkpoint achieving the highest overall score on the evaluation benchmarks across training, as used in the paper (Appendix B.3).
  • Final checkpoint: directly reports the last checkpoint at the end of training, with no checkpoint selection.
  • Held-out val: selects the checkpoint scoring highest on an independent validation set formed by holding out a random 20% of the real training corpus (3,583 questions in the single-teacher settings and 1,492 stratified by domain in MOPD), sampled to avoid any overlap with the small pools and disjoint from all evaluation benchmarks. To prevent any overlap with this validation set, the full-data baseline is retrained on the remaining 80% (14,334 questions in the single-teacher settings and 5,966 in MOPD).

Table R2 below reports performance under all three rules across the two single-teacher settings (six-benchmark math macro-average accuracy) and the multi-teacher setting (three-domain overall score):

SettingQuestion poolOriginalFinal checkpointHeld-out val
RL-tuned teacherReal, |D|=17,917 (full data)56.4156.05—
(init 47.08, teacher 61.44)Real, |D|=14,334 (full data minus held-out val)56.3955.7256.39
Real, |D|=152.8452.8452.59
Real, |D|=856.3355.7756.33
Real, |D|=6457.7356.4256.42
Real, |D|=25657.6556.5156.72
Real, |D|=1k57.4356.9057.43
Self-generated, |D|=153.7753.6153.33
Self-generated, |D|=854.8454.8454.79
Self-generated, |D|=6456.6156.5356.61
Self-generated, |D|=1k56.3855.6856.35
Larger teacherReal, |D|=17,917 (full data)47.3646.61—
(init 34.49, teacher 75.46)Real, |D|=14,334 (full data minus held-out val)47.5747.0747.25
Real, |D|=145.3644.8245.36
Real, |D|=847.7347.6147.56
Real, |D|=6448.1246.5847.10
Real, |D|=25648.1248.1248.07
Real, |D|=1k48.0348.0347.78
Self-generated, |D|=148.6148.6148.03
Self-generated, |D|=849.4648.6649.46
Self-generated, |D|=6449.1649.0449.04
Self-generated, |D|=1k49.1248.7949.12
Multi-teacherReal, |D|=7,458 (full data)31.1830.53—
(init 24.21, teachers 31.42)Real, |D|=5,966 (full data minus held-out val)31.2430.3931.24
Self-generated, |D|=829.6629.2329.46
Self-generated, |D|=6430.3329.3730.33
Self-generated, |D|=25630.8030.5130.51
Self-generated, |D|=1k31.3131.3131.31

Two clear takeaways emerge from Table R2:

  • Results hold at the final checkpoint: Directly taking the final checkpoint with no selection yields scores and pool-size scaling trends that closely mirror Original across all evaluated pools. In particular, at our key pool sizes, both small-pool OPD (55.77 vs. 56.05 and 47.61 vs. 46.61 for Real, |D|=8 in the two single-teacher settings) and DF-OPD (56.53 vs. 56.05 and 49.04 vs. 46.61 for Self-generated, |D|=64 in the two single-teacher settings, and 31.31 vs. 30.53 for Self-generated, |D|=1k in MOPD) remain comparable to the full-data baseline across all settings.

  • Results hold under held-out validation: Selecting checkpoints on the independent 20% validation set likewise closely tracks Original across all evaluated pool sizes and settings. In particular, at our key pool sizes, both small-pool OPD (56.33 vs. 56.39 and 47.56 vs. 47.25 for Real, |D|=8 in the two single-teacher settings) and DF-OPD (56.61 vs. 56.39 and 49.04 vs. 47.25 for Self-generated, |D|=64 in the two single-teacher settings, and 31.31 vs. 31.24 for Self-generated, |D|=1k in MOPD) match the retrained 80% full-data baseline in every setting.

AuthorsOpenReview ↗

Response to Reviewer iTKJ (3/4)

Q3 & W2. The analysis of "Self-QA" compares three student models that differ in both their training history and chat templates. Do you have within-model controls that isolate the effect of suppressing or inducing Self-QA?

Thank you for this insightful question, which helped us strengthen the link between Self-QA and the performance of empty-prompt OPD! While our original cross-student comparison in Section 5.2 captured an observed association across different models and chat templates, following your suggestion we conducted a within-model controlled experiment using online trajectory filtering that directly isolates Self-QA and establishes that successful empty-prompt OPD causally relies on Self-QA rollouts. Specifically, we compare three arms that share the exact same student, empty-prompt template, teacher, and hyperparameters, differing only in which student rollouts are kept for training:

  • All rollouts: the original unfiltered empty-prompt OPD run, where all student-generated rollouts are used for training.
  • Self-QA only: only Self-QA rollouts (where the student poses a problem on its own and attempts to solve it within the same response, as in Appendix F.3) are kept for training.
  • Non-Self-QA only: the complementary counterpart to Self-QA only, keeping exclusively non-Self-QA rollouts (such as asking the user to provide a question or noting the missing input) while filtering out all Self-QA rollouts.

During training, each rollout is classified online before teacher scoring using the Self-QA rule in Appendix F.2, and discarded rollouts are replaced by resampling from the same empty prompt until the batch is full, ensuring an identical trajectory budget per update across all arms while keeping the empty input strictly unchanged, directly testing which subset drives the gain without introducing prompt-level confounders.

We run this experiment in the RL-tuned teacher setting and MOPD (initial Self-QA rates of 62.0% and 11.8%, respectively; Section 5.2). We omit the larger-teacher setting because its student produces Self-QA in only 0.1% of rollouts, making it impractical for Self-QA only to fill even a single training batch, while Non-Self-QA only would be virtually identical to All rollouts.

Table R3 reports results under peak- and final-checkpoint selection (Held-out val is omitted as empty prompts have no training questions or ground-truth answers to hold out). Recovery is the retained fraction of the full-data gain over the initial student under the corresponding checkpoint-selection rule, (score − initial) / (full-data − initial) × 100%:

SettingRollouts used for trainingPeak checkpointRecoveryFinal checkpointRecovery
RL-tuned teacherAll rollouts (empty-prompt OPD)56.0095.6%55.8097.2%
(init 47.08, full-data peak 56.41,Self-QA only56.3699.5%56.36103.5%
full-data final 56.05)Non-Self-QA only48.1711.7%48.0711.0%
Multi-teacher (MOPD)All rollouts (empty-prompt MOPD)26.0726.7%26.0729.4%
(init 24.21, full-data peak 31.18,Self-QA only27.8251.8%27.5552.8%
full-data final 30.53)Non-Self-QA only24.869.3%24.8610.3%

Note: In MOPD, even after filtering for Self-QA, the absolute score remains below explicit DF-OPD and full-data MOPD because the student's spontaneous Self-QA rollouts cannot be routed to the matching domain teacher without an explicit prompt.

Two clear takeaways emerge from Table R3:

  • Retaining only Self-QA rollouts fully accounts for—and even surpasses—empty-prompt OPD: Across both settings and under both checkpoint-selection rules, Self-QA only surpasses unfiltered All rollouts, lifting full-data gain recovery from 95.6% / 97.2% (56.00 / 55.80) to 99.5% / 103.5% (56.36 at both peak and final checkpoints, matching full-data OPD at 56.41 / 56.05) in the RL-tuned setting, and nearly doubling it from 26.7% / 29.4% (26.07) to 51.8% / 52.8% (27.82 / 27.55) in MOPD. This is expected, as filtering out relatively uninformative non-Self-QA responses concentrates the entire trajectory budget on substantive problem-solving attempts where teachers provide meaningful token-level supervision.

  • Suppressing Self-QA largely eliminates empty-prompt gains within the exact same model and template: Conversely, training on the complementary Non-Self-QA only subset causes gains to collapse—accuracy drops from 56.00 / 55.80 (95.6% / 97.2% recovery) to 48.17 / 48.07 in the RL-tuned setting (retaining only 11.7% / 11.0% of the full-data gain over the 47.08 initial student) and from 26.07 (26.7% / 29.4%) to 24.86 in MOPD (9.3% / 10.3% recovery, just +0.65 over 24.21), confirming that successful empty-prompt distillation relies primarily on Self-QA rollouts.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal & Discussion Summary

We sincerely thank the Program Chairs, Senior Area Chairs, Area Chair, and all reviewers for their time, thoughtful evaluation, and constructive feedback, which motivated comprehensive robustness checks, in-depth mechanistic analyses, and systematic clarifications of our key findings and contributions.

  • 🟧 Overall assessment and reviewer recognition.\textcolor{BurntOrange}{\textbf{Overall assessment and reviewer recognition.}}
    • Novelty and community contribution: Reviewer 7AdL highlighted our findings as “novel and interesting” and “great contributions to the community,” offering “novel perspectives into how OPD data should be curated” (Informativeness: 3/3). Reviewer iTKJ noted our study “deepens the understanding of data requirements for OPD,” extending “beyond isolated few-shot results,” and Reviewer 1pUE recognized the research question as “worth-exploring and practical.”
    • Simplicity and effectiveness of DF-OPD: Reviewer iTKJ commended our “effective and well-defined DF-OPD workflow,” Reviewer 1pUE described DF-OPD as “simple and potentially useful,” and Reviewer 7AdL noted self-generated questions obtain “on-par performance with annotated questions.”
    • Soundness, insights, and clarity: Reviewer 7AdL commended our “reasonable investigation into the observations,” which “provides preliminary insights and encourages further studies” (Soundness: 3/3). Reviewer 1pUE noted “The design is sound” across multiple model pairs and data sources, making the observations “solid.” All three reviewers rated presentation 3 ("Clear and easy to follow").
  • 🟩 Reviewer 7AdL (Rating: 4, Clear Acceptance). The reviewer asked about random-seed sensitivity (Q1) and scaling to larger students (W1). We evaluated 3 training seeds and 3 disjoint question pools alongside final-checkpoint and held-out validation checks, confirming low-variance stability of our key findings. In a subsequent comment, Reviewer 7AdL thanked us for our detailed responses and maintained their Clear Acceptance (4) rating. Following up on W1, we also completed and shared 7B runs on OLMo-3-7B, confirming clean generalization to a larger non-Qwen student.
  • 🟩 Reviewer iTKJ. The reviewer inquired about robustness across seeds and question pools (Q1 & W1), checkpoint selection (Q2 & W1), within-model controls for Self-QA (Q3 & W2), and dynamic signal coverage during training (W2). We completed every targeted experiment suggested in the review: multi-seed and disjoint-pool runs (plus OLMo-3-7B) confirming low-variance stability, final-checkpoint and 20% held-out validation evaluations preserving the reported scores, within-model rollout-filtering interventions proving the causal role of Self-QA, and dynamic signal-coverage tracking verifying persistent diminishing marginal returns and strong correlation with downstream gains (r = 0.94).
  • 🟩 Reviewer 1pUE. The reviewer raised questions on direct comparisons with concurrent (Fu et al., 2026b) and prior few-shot / synthetic studies (W1), diagnostics on length/truncation/format and non-reasoning prompts (W2, Q1), and whether few-shot OPD merely recovers a small RL delta (W3). We added structured comparisons with concurrent and prior work, eliminated length/truncation/format confounds via 32k + BF evaluations and OLMo-3-7B diagnostics, ran non-reasoning Dolly controls ruling out generic surface behaviors as the main driver, and provided three lines of evidence—our non-RL-tuned 17× Larger-teacher (30B → 1.7B), 1/8-shot RLVR comparisons, and weight-update geometry—confirming that OPD goes beyond recovering an RL delta.
  • 🟦 Core contributions.\textcolor{RoyalBlue}{\textbf{Core contributions.}} Through a systematic three-level exploration, we characterize the unique role of data in OPD— (1) showing strong insensitivity to external question count (8 random questions matching 17.9k), difficulty, and initial distillation gap, explained by supervision signal coverage and cross-domain reasoning transfer (Sec. 3); (2) proposing DF-OPD, a simple yet effective self-sustaining framework replacing external corpora across single- and multi-teacher settings without any seed examples or reference solutions (Sec. 4); and (3) revealing the question-free boundary where empty prompts work only via spontaneous student Self-QA (implicit DF-OPD, Sec. 5)—offering fresh perspectives on the role of data in OPD.
  • 🟦 Revisions and reproducibility.\textcolor{RoyalBlue}{\textbf{Revisions and reproducibility.}} All new experiments and clarifications are integrated into the revised manuscript (Secs. 3.1–3.3, 4.2, 5.2, 6; Apps. C.2, F.4, G.1–G.4, H.1–H.3, J.2–J.3), with code available in our anonymous repository since submission.

We sincerely thank the Program Chairs, Senior Area Chairs, Area Chair, and all reviewers once again for their time and valuable feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.