acceptodds
Under review as a conference paper at ICLR 2027

IOD: An Iterative On-Policy Distillation Framework for Self-Improving Language Models

Abstract

On-policy knowledge distillation reduces the mismatch between student training and inference by learning from student-generated trajectories. However, extending distillation across multiple student updates raises the question of how previously generated trajectories should be managed as the student evolves. We propose Iterative On-Policy Distillation (IOD), a multi-cycle framework that combines teacher-based trajectory filtering with cross-cycle accumulation. At each cycle, IOD jointly re-evaluates previously retained and newly generated trajectories using a fixed teacher, then updates the student through token-level distribution matching while preserving the original human-labeled data. Experiments on summarization and Machine translation, using encoder–decoder and decoder-only teacher–student architectures, show higher task-specific scores than reproduced distillation baselines under the evaluated settings. Ablation studies further examine the effects of trajectory filtering, accumulation, and the student-update objective. These findings highlight trajectory management as an important consideration when extending on-policy distillation across successive student updates.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Knowledge distillation, data filtering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8JsbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how to utilize cross-cycle data accumulation in OPD to improve performance. This paper introduces a data filtering process, selecting data with higher scores labeled by a fixed teacher, combining it with some human-labeled data, and then doing JSD-based distillation. This paper conducts experiments on the XSum and WMT14 datasets for evaluation.

Critical strengths

The paper presents a method for selecting and reusing historical trajectories during iterative distillation.

Experiments cover multiple model architectures and tasks, with several baselines and ablation studies.

Critical weaknesses

The soundness of this paper is very low. In the title, the paper covers “on-policy” and “self-improvement”. But this method is neither “on-policy” nor “self-improvement”. With “Data filter” and “JSD”, I don’t know how to classify this paper as an “On-Policy Distillation Framework”. With ”human-labeled data” and “distillation”, I don’t know how to classify it as “self-improvement”. The paper should adopt more precise terminology and reframe its claims accordingly.

The experimental justification is insufficient. The setup is too low. XSum and WMT14 are less representative in nowadays LLM research. ICLR submissions are supposed to adopt frontier tasks such as math reasoning, coding, or agentic tasks. In addition, the ablation studies do not isolate the contribution of cross-cycle re-selection, as removing accumulation also changes the training data volume and the optimization budget, which makes the actual source of the performance gains unclear.

Questions for authors

I don't have questions.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 8Jsb (1)

Dear Reviewer 8Jsb,

Thank you for your careful reading and for asking us to be more precise about the terms “on-policy” and “self-improvement.” In response, we revised the title and abstract and clarified how each source of training data enters IOD.

1. On-policy sampling and the role of the teacher

At the start of cycle cc, the student samples fresh candidates from the decoding policy of its current checkpoint, πS(c−1)\pi_S^{(c-1)}. This is the part of IOD that we describe as on-policy (§3.1, Eq. (3)). The fixed teacher then assigns sequence-level, length-normalized log-likelihood scores to student-generated trajectories. IOD uses these scores to select from the combined pool of fresh and previously retained trajectories (§§3.2–3.3, Eqs. (8)–(9)). The revised paragraph immediately after Eq. (9) explicitly identifies both sources: fresh samples from the checkpoint at the start of the current cycle and trajectories generated by earlier checkpoints.

Teacher scoring determines which student-generated trajectories are retained. During the update, token-level JSD matches the teacher’s and student’s next-token distributions on both the selected student trajectories and the human-written references (§3.4). Thus, the decoding policy defines how fresh candidates are collected, the teacher score guides trajectory selection, and JSD defines the distillation objective.

2. Human-written references and revised title

We retain the original human-labeled examples in every cycle to preserve a fixed source of human-written reference prefixes as the student-generated curriculum changes. The paragraph immediately after Eq. (10) in §3.3 now states this rationale and specifies that teacher-based filtering applies to student-generated trajectories.

Our original “self-improvement” wording referred to the student learning over successive cycles from its own generations under the guidance of the same fixed teacher. The teacher supplies both the selection scores and the next-token distillation targets, and the procedure requires no separate reward model. To describe the contribution more precisely, we replaced the original wording with “IOD: Iterative On-Policy Distillation with Cross-Cycle Trajectory Selection.” The revised abstract describes the method as teacher-guided distillation on selected student trajectories and original human-written references.

AuthorsOpenReview ↗

Response to Reviewer 8Jsb (2)

Your comments on the scope of the evaluation and the interpretation of the ablation prompted us to add a mathematical reasoning task and clarify the comparison conditions in the revised manuscript.

3. Mathematical reasoning evaluation

We added GSM8K using a Qwen2.5-3B teacher and a Qwen2.5-1.5B student (§§4.1–4.2 and Appendix A.1). On the 1,319-example test set, the common SFT student achieves 52.08% accuracy. At 2,893 post-SFT student optimizer updates for each method, IOD reaches 57.62%, compared with 56.03% for GKD and 53.83% for DistiLLM (Table 3). IOD’s accuracy also increases across the three cycles from 55.19% to 56.71% to 57.62% (Appendix B.2, Table 12). Across three independent IOD runs, the mean final-cycle accuracy is 57.59%, with a standard deviation of 0.19 percentage points (Appendix B.1, Table 11).

4. Scope of the ablation and optimizer-update comparison

As you point out, removing cross-cycle reuse changes both the trajectories eligible for selection and the number retained. We explicitly state this in §4.3, where we present Table 4 as a comparison of the resulting training configurations. On T5/XSum, full IOD achieves 35.50 ROUGE-L, compared with 32.79 when retaining all generated trajectories and 31.01 when selecting only from the current cycle. These scores capture the performance of the three configurations, including their associated differences in retained data volume.

We address the optimizer-update budget in Appendix A.4, Table 10. There, GKD and DistiLLM are evaluated at the same cumulative number of post-SFT student optimizer updates reached at each IOD cycle endpoint. IOD achieves higher ROUGE-L on T5/XSum and higher BLEU on T5/WMT14 than both baselines at every matched endpoint. Thus, Table 4 compares trajectory-retention configurations with different retained data volumes, and Table 10 shows that the baseline performance gap persists at equal optimizer update counts. The scope of each comparison is stated in §4.3 and Appendix A.4, respectively.

Reviewer_7HHvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

IOD is a multi-cycle on-policy distillation framework. Each cycle the student samples K outputs per training input, a fixed teacher scores every output by length-normalized log-likelihood, and the top-q fraction is kept — but crucially, previously retained trajectories are pooled with the new ones and re-filtered together, so old samples can be dropped instead of piling up forever.

Critical strengths
  1. The problem is real and nicely framed. Multi-round on-policy distillation genuinely forces you to decide what to do with stale trajectories, and nobody had spelled that out. "Keep everything" is wasteful and drags in junk; "keep only this round" throws away good stuff. IOD's answer — pool old and new, re-rank them under one fixed teacher — is simple, sensible, and clearly better than both extremes. It's the kind of idea that reads obvious in hindsight, which is usually a good sign.

  2. Experiments are unusually careful and honest. Baselines are re-run in-house from identical SFT checkpoints rather than copied from papers, they report absolute scores instead of teacher-relative gaps, they push GKD/DistiLLM to matched optimizer-step budgets, and they give seed variance, cycle-wise curves, and wall-clock cost. Very few KD papers bother with any of this.

  3. The paper is refreshingly upfront about its own limits. It repeatedly admits that retained trajectories aren't strictly on-policy, that teacher likelihood is a weak quality proxy, and that the accumulation ablation is confounded with dataset size. That honesty makes it easy to trust the parts that do hold up, which is rare.

Critical weaknesses
  1. Novelty feels thin, and the key baselines are missing. Strip away the wording and this is GKD run for several rounds with teacher-likelihood filtering — basically ReST/ReSTEM with a teacher score standing in for the reward. The authors admit self-training alone doesn't separate them, yet ReST, ReSTEM, and STaR are cited but never run as baselines. Without those, I can't tell what the teacher actually buys you. Also DistiLLM-2 is cited but not compared.

  2. The comparison isn't compute-matched, and the gains don't hold up everywhere. IOD does extra generation and teacher scoring per cycle, and its training set grows as trajectories accumulate, so "same optimizer steps" is a weak control — the paper says so itself. And the decoder-only XSum result is basically noise: ROUGE-L 34.119 vs GKD's 34.001, with teacher at 34.296. The headline gains are almost entirely the T5 setup.

Questions for authors

see weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer 7HHv (1)

Dear Reviewer 7HHv,

Thank you for your careful review and for highlighting the importance of deciding which student trajectories to carry forward across distillation cycles. In response to your comments, we clarified how IOD relates to prior self-training methods and how the fixed teacher is used in trajectory selection.

Weakness 1. Contribution and baseline coverage

We agree that repeated generation and filtering are shared with prior self-training methods. ReST uses reward-guided filtering and can reuse generated data across improvement stages. ReSTEM^{\mathrm{EM}} filters generated solutions using binary feedback, and STaR fine-tunes on rationales that lead to correct answers. We discuss these connections in §2.

We specifically study how to re-select previously retained trajectories alongside newly generated ones as the student evolves. At each cycle, IOD pools the trajectories retained from the previous cycle with candidates sampled from the current student. It ranks the combined pool using a fixed teacher’s length-normalized conditional log-likelihood and recomputes the selection threshold on that pool. This gives earlier trajectories a chance to remain in the training set and lets higher-scoring new candidates replace them (§3.3, Eqs. (8)–(9)).

The fixed teacher provides two forms of supervision. Its sequence-level scores give IOD a common ranking criterion for outputs generated by different student checkpoints. Its next-token distributions then serve as targets for the JSD update on the selected trajectories and the original human-written references (§3.4). On T5/XSum, full IOD achieves 35.50 ROUGE-L, compared with 32.79 when retaining all generated trajectories and 31.01 when selecting only from the current cycle (§4.3, Table 4). These results compare the resulting training configurations, including the differences in the number of trajectories each configuration retains.

The empirical comparisons in §4 use reproduced distillation baselines initialized from the same task-specific SFT checkpoints. The reported DistiLLM results refer to the original method (Ko et al., 2024). We discuss DistiLLM-2 (Ko et al., 2025) in §2 for its contrastive objective on teacher- and student-generated responses.

AuthorsOpenReview ↗

Response to Reviewer 7HHv (2)

Weakness 2. Training budget and performance across model families

Training budget. We agree that the number of student optimizer updates captures one part of the overall training budget. To control for this factor, we extended GKD and DistiLLM to the cumulative number of post-SFT student updates reached at the end of each IOD cycle. IOD scores higher than both baselines at every matched endpoint on T5/XSum and T5/WMT14 (Appendix A.4, Table 10). For example, at 106,437 updates on T5/XSum, IOD achieves 35.502 ROUGE-L, compared with 30.70 for GKD and 31.10 for DistiLLM. The performance difference thus persists at equal update counts.

Candidate generation, teacher scoring, and the number of processed tokens also contribute to end-to-end cost. Appendix B.3, Table 14 reports IOD’s measured total wall-clock time, including candidate generation, scoring, and student optimization. For the new GSM8K comparison, we report the number of newly generated student sequences alongside the update count. At 2,893 student optimizer updates per method, IOD, GKD, and DistiLLM (SRKL) generate 62,757, 92,576, and 5,952 student sequences, respectively (Table 3).

Qwen2.5 on XSum. Before distillation, the teacher scores 34.296 ROUGE-L and the SFT student scores 32.343, a gap of 1.953 points. IOD achieves 34.119, the highest ROUGE-L among the evaluated distillation methods, bringing the student within 0.177 points of the teacher. GKD scores 34.001, leaving a gap of 0.295 points to the teacher (Table 1). Thus, IOD closes the most of the initial teacher–student gap among the evaluated methods, with a modest observed margin of 0.118 points over GKD. Across three independent IOD runs, the result is 34.11 ± 0.12 ROUGE-L (Appendix B.1, Table 11).

Qwen2.5 on translation and mathematical reasoning. The same Qwen2.5-3B teacher and Qwen2.5-1.5B student model pair shows clearer gains on the other evaluated tasks. On WMT14 En–De, IOD reaches 23.908 BLEU, compared with 22.891 for the strongest reproduced baseline (Table 2). On the newly added GSM8K evaluation, IOD reaches 57.62% accuracy, compared with 56.03% for GKD and 53.83% for DistiLLM (SRKL), with all three methods using 2,893 student optimizer updates (Table 3). Across three independent IOD runs on GSM8K, the mean final-cycle accuracy is 57.59%, with a standard deviation of 0.19 percentage points (Appendix B.1, Table 11).

Reviewer_h5sBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a multi-cycle framework called “Iterative On-Policy Distillation” (IOD), which manages the trajectories generated by student models as they evolve.

In each cycle:

  • The student model samples K candidate trajectories for each training input.
  • A fixed teacher model scores them based on length-normalized log-likelihood values.
  • Previously retained trajectories are merged with the new ones and jointly filtered using a global top-q rule (older trajectories may be replaced).
  • The student model is updated based on the retained trajectories and the original human-annotated data using token-level JSD.

Experiments: On the XSum (T5v1.1-XL reduced to Small; Qwen2.5-3B reduced to 1.5B) and WMT14 English-German datasets, the method outperforms reproduced KD, ImitKD, DistiLLM, and GKD (e.g., a ROUGE-L score of 35.5 on XSum, while the best T5 baseline achieved only 33.4). Ablation experiments: Removing filtering (32.8) or cross-cycle accumulation (31.0) results in a performance drop.

Critical strengths
  1. The paper identifies a practical problem (namely, which student trajectories should be retained and reused during a distillation cycle) and provides a simple solution with a clear algorithm that does not require a reward model; it candidly acknowledges its limitations (the teacher likelihood function does not represent task quality; retained trajectories are no longer on-policy).

  2. Across two tasks and two architecture families, the approach achieved consistent performance improvements compared to reproduced baseline models, and ablation experiments demonstrated that both filtering and accumulation play important roles.

Critical weaknesses

The performance improvements cannot be clearly attributed to cross-cycle trajectory management, and the approach is not particularly innovative (GKD-style JSD + ReST-style iteration and filtering + teacher likelihood filtering). The authors note that ablation experiments without accumulation also alter the data scale; however, no control experiments were conducted under matched data volume and computational resource conditions (random top-q, retaining all data, and equal update frequencies).

Questions for authors
  1. Is it possible to add control experiments (random Top-q selection; retaining all samples and performing equal-quantity updates) while maintaining the same data scale and computational load?

  2. How many seeds support the main table? Given an equal budget, how does its computational cost per cycle compare to the baseline?

  3. Over the course of multiple cycles, does the teacher likelihood Top-q method cause the retained set to shift toward short/general-purpose outputs? How sensitive are the results to q and K?

2 comments
AuthorsOpenReview ↗

Response to Reviewer h5sB (1)

Dear Reviewer h5sB,

Thank you for your careful review and for highlighting the importance of deciding which student trajectories to reuse as the student evolves. In response to your comments on the ablation controls, we clarified in the revised manuscript that changing the selection rule affects both which trajectories are retained and how many are retained. We address your questions based on the experiments and analyses reported in the revised manuscript.

Critical Weakness and Question 1. Cross-cycle trajectory selection and controls

We agree that token-level JSD and repeated generation and filtering are established components. The specific question we study is how to re-select previously retained student trajectories when the current student generates new candidates. At each cycle, IOD pools the previously retained trajectories with the fresh candidates and ranks them together using the same fixed teacher’s length-normalized conditional log-likelihood. IOD recomputes the global selection threshold on the combined pool. An earlier trajectory can remain in the curriculum if it still meets the updated threshold, and a higher-scoring new candidate can displace it. We describe pooling and re-selection in §3.3, Eqs. (8)–(9). The JSD update uses the selected trajectories and the original human-written references (§3.4).

We evaluated the retain-all setting on T5/XSum. Full IOD achieves 35.50 ROUGE-L, compared with 32.79 when retaining all generated trajectories and 31.01 when applying teacher-based filtering only to candidates from the current cycle (§4.3, Table 4). The retain-all result also appears as the q=1.00q=1.00 setting in Table 5. Retain-all keeps the entire combined pool. The current-cycle variant applies teacher-based selection to a pool consisting only of newly generated candidates. As you noted, these changes also affect the number of retained trajectories. §4.3 now makes clear that Table 4 compares the resulting training configurations, including differences in retained data volume.

The optimizer-update comparison in Appendix A.4, Table 10 evaluates GKD and DistiLLM at the cumulative number of student optimizer updates reached by IOD at each cycle endpoint. On both T5/XSum and T5/WMT14, IOD scores higher than both reproduced baselines at every matched endpoint. The difference in performance therefore persists when the number of student optimizer updates is held fixed.

AuthorsOpenReview ↗

Response to Reviewer h5sB (2)

Question 2. Random seeds and computational budget

We ran IOD with three independent random seeds for each task–model combination and report the mean and standard deviation of final-cycle performance in Appendix B.1, Table 11. On XSum, the three-seed results are 35.51 ± 0.12 ROUGE-L with T5 and 34.11 ± 0.12 ROUGE-L with Qwen2.5. On GSM8K, the mean final-cycle accuracy is 57.59%, with a standard deviation of 0.19 percentage points.

For the cycle-wise comparison, Appendix A.4, Table 10 aligns GKD and DistiLLM with IOD’s cumulative post-SFT student optimizer updates at the end of each cycle. At the final T5/XSum endpoint of 106,437 updates, IOD achieves 35.502 ROUGE-L, compared with 30.70 for GKD and 31.10 for DistiLLM. At the final T5/WMT14 endpoint of 590,273 updates, IOD achieves 26.254 BLEU, compared with 25.51 for GKD and 24.79 for DistiLLM. Table 10 also reports the corresponding comparisons at the earlier cycle endpoints.

Appendix B.3, Table 14 reports IOD’s total wall-clock time across candidate generation, teacher scoring, and student optimization. For the added GSM8K comparison, we also report generation volume alongside the common budget of 2,893 student optimizer updates. IOD, GKD, and DistiLLM (SRKL) generate 62,757, 92,576, and 5,952 student sequences, respectively, and achieve 57.62%, 56.03%, and 53.83% test accuracy (Table 3).

Question 3. Retention ratio, candidate count, and trajectory scoring

The retention-ratio study on T5/XSum fixes the candidate count at K=3K=3 per training input and varies qq (§4.4, Table 5, and Appendix A.3). Among the evaluated retention ratios, q=0.70q=0.70 gives the highest ROUGE-L (35.50), followed closely by q=0.50q=0.50 (35.13). Retaining the full pool at q=1.00q=1.00 gives 32.79, and retaining a smaller fraction at q=0.10q=0.10 gives 31.26. These results characterize sensitivity to qq at the candidate count used in the reported experiments.

Regarding shorter or more generic outputs, the teacher ranks trajectories by mean conditional log-likelihood per output token (§3.2, Eq. (5)). Averaging reduces the direct effect of sequence length that comes from summing token log-likelihoods. Appendix B.4 tracks the fixed teacher’s scores on outputs generated from held-out inputs by successive student checkpoints. Those distributions describe how teacher likelihood evolves across checkpoints. The length and specificity of retained training trajectories are separate properties of the selected pool.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Gentle Reminder to All Reviewers

Dear reviewers,

Thank you again for your constructive feedback. We have responded directly to every concern raised in the reviews and revised the manuscript accordingly. Before the discussion closes, we would appreciate a brief note on any remaining concerns we can address.

Best, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.