- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies privileged information (PI) for on-policy self-distillation (OPSD), evaluating 14 PI configurations across three agentic benchmarks and three student models. It compares PI formats and generation sources, proposes offline metrics for predicting post-training performance, and evaluates combining OPSD with GRPO.
Critical strengths
- PI selection is practically relevant, and assessing PI before training could reduce the cost of choosing training configurations.
- The experiments cover multiple PI formats, generation sources, benchmarks, and student models.
Critical weaknesses
- The paper reads as a record of combinatorial experiments and does not establish a research contribution sufficient for a top-tier conference. Its main approach is to enumerate combinations of PI formats, generation sources, and student models, report their performance, and provide post-hoc explanations for the observed outcomes. The PI recommendations and proposed metrics lack a convincing theoretical foundation or a clear a priori framework capable of predicting these outcomes. The correlations establish empirical associations, but the paper does not develop them into a substantive, generalizable understanding of PI effectiveness. Extensive experimental coverage alone does not establish the conceptual depth or scientific insight expected of an ICLR research paper.
- The presentation is disorganized and difficult to follow. The transitions between OPSD, OPD, and SDPO are insufficiently explained, making the terminology and training formulation hard to track. Several figures are densely packed with small labels, numerous annotations, and repeated legends, obscuring the key comparisons. The text frequently moves from experimental observations to explanatory claims without clearly establishing the evidential link. Together, these issues make it difficult to identify the central argument and assess how strongly the experiments support it.
Questions for authors
What theoretical foundation or a priori framework supports the PI recommendations and proposed metrics? Please explain what predictions this framework makes before observing training outcomes, and what general understanding the paper establishes beyond configuration rankings and correlations.
We thank the reviewer for the feedback. We respectfully disagree that the work does not establish a research contribution, and we address each concern below. In response, we added three controlled studies to the revision (changes are marked in blue): ALFWorld-Passcode (Section 5; Appendix E, Figure 6), a corrupted-PI ablation (Section 5; Appendix F, Figure 7), and a per-task PI selection experiment (Section 5, Figure 5; Appendix G).
W1 + Q1 (a priori framework and predictions). Our metrics follow from the OPSD objective and were not fitted to the results. In OPSD, the student and the teacher share parameters and differ only in access to PI, so the training signal at each token is the difference between the two policies on the student's own rollouts (Section 2.1). This gives a framework with three predictions that can be stated before any training:
The revision tests each prediction by intervention, and not by correlation:
This is an empirical framework and not a formal theory, and we do not claim otherwise. The general understanding goes beyond a ranking of configurations: PI is effective when it corrects the student at the steps where the student makes mistakes. By contrast, PI that only helps the student succeed while it is in context, or that only shifts the student's distribution by a large amount, does not lead to a training gain. The 126 training runs over 14 PI configurations, three benchmarks, and three students are the evidence base that makes these conclusions testable across settings (Section 3), and Section 5 now opens by stating how the controlled studies test them.
W2 (presentation). We thank the reviewer for these points. The terminology was inconsistent: Section 4 used SDPO for what the rest of the paper calls OPSD. We removed SDPO, so the method has one name throughout, and Section 2.1 defines how OPSD relates to OPD. On the figures and the link between observations and claims, we would like to improve both and would appreciate more specific pointers: which figures obscure a key comparison, and which claims the reviewer finds are not supported by the evidence that precedes them. We will revise these directly once we know where the problems are.