- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies how to utilize cross-cycle data accumulation in OPD to improve performance. This paper introduces a data filtering process, selecting data with higher scores labeled by a fixed teacher, combining it with some human-labeled data, and then doing JSD-based distillation. This paper conducts experiments on the XSum and WMT14 datasets for evaluation.
Critical strengths
The paper presents a method for selecting and reusing historical trajectories during iterative distillation.
Experiments cover multiple model architectures and tasks, with several baselines and ablation studies.
Critical weaknesses
The soundness of this paper is very low. In the title, the paper covers “on-policy” and “self-improvement”. But this method is neither “on-policy” nor “self-improvement”. With “Data filter” and “JSD”, I don’t know how to classify this paper as an “On-Policy Distillation Framework”. With ”human-labeled data” and “distillation”, I don’t know how to classify it as “self-improvement”. The paper should adopt more precise terminology and reframe its claims accordingly.
The experimental justification is insufficient. The setup is too low. XSum and WMT14 are less representative in nowadays LLM research. ICLR submissions are supposed to adopt frontier tasks such as math reasoning, coding, or agentic tasks. In addition, the ablation studies do not isolate the contribution of cross-cycle re-selection, as removing accumulation also changes the training data volume and the optimization budget, which makes the actual source of the performance gains unclear.
Questions for authors
I don't have questions.
2 comments
Response to Reviewer 8Jsb (2)
Your comments on the scope of the evaluation and the interpretation of the ablation prompted us to add a mathematical reasoning task and clarify the comparison conditions in the revised manuscript.
3. Mathematical reasoning evaluation
We added GSM8K using a Qwen2.5-3B teacher and a Qwen2.5-1.5B student (§§4.1–4.2 and Appendix A.1). On the 1,319-example test set, the common SFT student achieves 52.08% accuracy. At 2,893 post-SFT student optimizer updates for each method, IOD reaches 57.62%, compared with 56.03% for GKD and 53.83% for DistiLLM (Table 3). IOD’s accuracy also increases across the three cycles from 55.19% to 56.71% to 57.62% (Appendix B.2, Table 12). Across three independent IOD runs, the mean final-cycle accuracy is 57.59%, with a standard deviation of 0.19 percentage points (Appendix B.1, Table 11).
4. Scope of the ablation and optimizer-update comparison
As you point out, removing cross-cycle reuse changes both the trajectories eligible for selection and the number retained. We explicitly state this in §4.3, where we present Table 4 as a comparison of the resulting training configurations. On T5/XSum, full IOD achieves 35.50 ROUGE-L, compared with 32.79 when retaining all generated trajectories and 31.01 when selecting only from the current cycle. These scores capture the performance of the three configurations, including their associated differences in retained data volume.
We address the optimizer-update budget in Appendix A.4, Table 10. There, GKD and DistiLLM are evaluated at the same cumulative number of post-SFT student optimizer updates reached at each IOD cycle endpoint. IOD achieves higher ROUGE-L on T5/XSum and higher BLEU on T5/WMT14 than both baselines at every matched endpoint. Thus, Table 4 compares trajectory-retention configurations with different retained data volumes, and Table 10 shows that the baseline performance gap persists at equal optimizer update counts. The scope of each comparison is stated in §4.3 and Appendix A.4, respectively.
Response to Reviewer 8Jsb (1)
Dear Reviewer 8Jsb,
Thank you for your careful reading and for asking us to be more precise about the terms “on-policy” and “self-improvement.” In response, we revised the title and abstract and clarified how each source of training data enters IOD.
1. On-policy sampling and the role of the teacher
At the start of cycle c, the student samples fresh candidates from the decoding policy of its current checkpoint, πS(c−1). This is the part of IOD that we describe as on-policy (§3.1, Eq. (3)). The fixed teacher then assigns sequence-level, length-normalized log-likelihood scores to student-generated trajectories. IOD uses these scores to select from the combined pool of fresh and previously retained trajectories (§§3.2–3.3, Eqs. (8)–(9)). The revised paragraph immediately after Eq. (9) explicitly identifies both sources: fresh samples from the checkpoint at the start of the current cycle and trajectories generated by earlier checkpoints.
Teacher scoring determines which student-generated trajectories are retained. During the update, token-level JSD matches the teacher’s and student’s next-token distributions on both the selected student trajectories and the human-written references (§3.4). Thus, the decoding policy defines how fresh candidates are collected, the teacher score guides trajectory selection, and JSD defines the distillation objective.
2. Human-written references and revised title
We retain the original human-labeled examples in every cycle to preserve a fixed source of human-written reference prefixes as the student-generated curriculum changes. The paragraph immediately after Eq. (10) in §3.3 now states this rationale and specifies that teacher-based filtering applies to student-generated trajectories.
Our original “self-improvement” wording referred to the student learning over successive cycles from its own generations under the guidance of the same fixed teacher. The teacher supplies both the selection scores and the next-token distillation targets, and the procedure requires no separate reward model. To describe the contribution more precisely, we replaced the original wording with “IOD: Iterative On-Policy Distillation with Cross-Cycle Trajectory Selection.” The revised abstract describes the method as teacher-guided distillation on selected student trajectories and original human-written references.