acceptodds
Withdrawn from ICLR 2027

Controlled In-Rollout Privileged Information Intervention for Reliable On-Policy Self-Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

On-policy self-distillation (OPSD) trains a model on its own rollouts using token-level supervision from a privileged teacher instantiated from the same base model. In standard OPSD, the teacher receives the complete reference solution in its initial prompt. We study a specific failure mode of this design: full prompt-level privileged information (PI) can induce a concentrated teacher–student discrepancy near the beginning of a rollout, while its corrective influence weakens as an erroneous student prefix grows. We introduce Controlled In-Rollout Privileged-Information Intervention (CIPI), which periodically selects PI fragments by balancing their guidance benefits against the resulting teacher–student policy discrepancy, and inserts them into the teacher context along student-generated rollouts. Theoretical analysis provides conceptual motivation for balancing teacher guidance and teacher–student discrepancy in PI selection. Experiments with Qwen3-1.7B and Qwen3-4B on three competition-level mathematical reasoning benchmarks show that CIPI consistently outperforms OPSD, with macro Avg@12 gains of 3.3 and 5.1 percentage points (ppts) in non-thinking mode and 1.7 and 1.0 ppts in thinking mode, respectively. Ablation studies show that removing periodic in-rollout intervention while retaining prompt-level PI selection reduces average performance by 2.7 ppts, further confirming the effectiveness of in-rollout intervention beyond prompt-level PI selection.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Self-Distillation, On-Policy Learning, Privileged Information, Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZVVNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why a self-distillation teacher can become unreliable even when given a correct reference solution: erroneous student prefixes can weaken its corrective guidance, while providing the entire solution upfront can create excessive teacher–student disagreement. To address this issue, this paper proposed CIPI to selectively insert reference-solution fragments into the teacher’s context during a student rollout, which balances increased correct-answer likelihood against teacher–student KL divergence. Experiments on two small Qwen models demonstrated the effectiveness of the proposed method.

Critical strengths
  • The proposed CIPI method outperforms OPSD consistently in all settings with statistically significant advantages.
  • The proposed method is clearly motivated by the failure-mode analysis.
  • Detailed ablation study demonstrates the advantage of controlled intervention over simply adding reminders.
Critical weaknesses
  • CIPI selects fragments using the teacher’s likelihood of the known answer, but increasing answer confidence does not necessarily mean the teacher provides better intermediate corrections. Independent evaluation of intermediate correction quality is missing.
  • The theoretical analysis is useful, but does not establish improvement guarantees for CIPI. For example, Theorem 1 considers a single gradient step with fixed prefix and reference distributions, whereas actual training changes the student and its rollout distribution.
  • Experiments only cover two small Qwen3 models.
Questions for authors

Please refer to my comments in the weakness part.

Reviewer_EvX2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the on-policy self-distillation setting and hypothesizes that using a complete solution as privileged information (PI) is detrimental, as the discrepancy between the student and teacher is too large and the teacher may be mistakenly pulled towards the student's incorrect solution as generation length increases. The paper considers CIPI, which partitions the student's answer and selects fragments of the reference solution to insert into the teacher's context at segment boundaries. The student is then trained on the resulting teacher predictions. Experiment setup uses Qwen3-1.7B and Qwen3-4B and evaluates on AIME24, AIME25, and HMMT25.

Critical strengths
  • Experimental setup is rigorous.
Critical weaknesses
  • (Major) The number improvements are too marginal to consider this a real increase in the numbers. The paper seems to lack an interesting scientific finding that sheds more light on how to better do OPSD.
Questions for authors
  • Please address W1. What is the scientific insight gained from this paper that is novel?
Reviewer_ipcKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies on-policy self-distillation, where a teacher built from the same base model receives privileged information (PI), typically the reference solution, only in the initial prompt. It argues that this design makes the teacher's guidance decay along erroneous student rollouts and causes an abrupt teacher–student divergence at the start, and proposes Controlled In-Rollout Privileged-Information Intervention (CIPI), which splits the PI into fragments and periodically inserts selected fragments into the teacher context during the rollout. Fragments are chosen greedily to raise the teacher's likelihood of the correct answer while limiting the resulting teacher–student KL, and intervention is triggered only when the teacher's guidance appears to degrade. A local surrogate analysis motivates this trade-off. On competition mathematics benchmarks, CIPI improves average accuracy over OPSD and the baselines it compares with.

Critical strengths
  1. The paper is well written and easy to follow.

  2. The ablation experiments show that uncontrolled ways of inserting teacher information do not beat the baseline, while the controlled version does, demonstrating the effectiveness of the proposed method.

Critical weaknesses
  1. Figure 2 does not show what it is said to show. What is measured is the teacher's perplexity on the gold answer tokens, not on the correct solution the caption refers to, and it is taken by appending that answer directly to a truncated and wrong prefix, which is a probe of answer confidence rather than of the supervision the teacher would actually provide. With no curve on correct trajectories and no control for prefix length or content, a rising perplexity cannot be separated from the ordinary degradation that comes with a longer context, and CIPI's lower value is expected by construction, since it places the relevant fragments next to the prediction. The figure is therefore consistent with the mechanism the method assumes, but it does not test it, and it cannot show that CIPI provides a better supervision signal.

  2. Everything is measured on Qwen3-1.7B and Qwen3-4B, one model family that now looks somewhat dated, and only on competition mathematics, yet the story is told about OPSD in general.

  3. The method is validated for only one kind of PI. Every PI used in the paper is the full reference solution, split into eight fragments with the answer sentence removed, and the ablations change only how these fragments are inserted, not what the PI is. It is therefore unclear whether the selection rule transfers to other common forms such as answer-only supervision, partial hints, execution feedback or a correct sibling rollout.

  4. The method adds a scoring pass at every intervention point and a substantial amount of extra training time.

Questions for authors

See Weaknesses

3 comments

Question. Figure 2 does not show what it is said to show. What is measured is the teacher's perplexity on the gold answer tokens, not on the correct solution the caption refers to, and it is taken by appending that answer directly to a truncated and wrong prefix, which is a probe of answer confidence rather than of the supervision the teacher would actually provide. With no curve on correct trajectories and no control for prefix length or content, a rising perplexity cannot be separated from the ordinary degradation that comes with a longer context, and CIPI's lower value is expected by construction, since it places the relevant fragments next to the prediction. The figure is therefore consistent with the mechanism the method assumes, but it does not test it, and it cannot show that CIPI provides a better supervision signal.

Response. Low gold-answer perplexity reflects the teacher’s confidence in the correct answer. We use this diagnostic to track how that confidence changes along student rollouts, providing the motivation for CIPI. We understand the reviewer's two concerns: whether NLL changes reflect student rollout quality or simply distance from PI, and whether the benefit extends to actual continuation accuracy. Our additional experiments indicate that both PI placement and rollout quality matter, and that CIPI improves teacher continuation accuracy, particularly on trajectories ending in incorrect answers. We would also clarify that the statement “it places the relevant fragments next to the prediction” does not match our measurement timing: PI is inserted before a student rollout segment (Section 3, lines 179–180). Thus, at least one student segment separates PI insertion from answer scoring; a low NLL at that point indicates that the teacher retains confidence in the correct answer after observing the student segment, which we interpret as indicating that the teacher can maintain effective supervision throughout that segment.

(1) Separating trajectory quality from PI distance. We pair natural correct-answer and incorrect-answer rollouts within the same question and truncate them to identical lengths.

Prefix tokensOPSD: CorrectOPSD: IncorrectOPSD: Incorrect − CorrectCIPI: CorrectCIPI: IncorrectCIPI: Incorrect − Correct
1282.6272.605−0.0222.5882.601+0.014
2562.6052.706+0.1012.5592.652+0.093
5122.5492.600+0.0512.5132.591+0.078
10242.7172.883+0.1662.5272.644+0.117
20482.9573.240+0.2832.9083.087+0.179

Reading the table vertically, as the probe moves further away from PI, NLL exhibits a continuous increasing trend, particularly at 512 tokens and beyond. Reading horizontally, incorrect-answer trajectories have higher NLL at most positions. These patterns show contributions from both PI distance and student rollout quality to the observed NLL increases. Both observations motivate our in-rollout intervention: refreshing PI during the rollout can help sustain its influence and counteract misleading student context.

(2) Evaluating actual teacher continuations. We evaluate teacher continuations at 384 planned points across 128 questions. The 12 unassessable continuation points are counted as incorrect for both methods. CIPI with full PI insertion achieves 72.66% accuracy versus 68.23% for OPSD, a 4.43-point improvement. On the 78 planned points from trajectories whose final answers were incorrect, accuracy improves from 32.05% to 46.15% (+14.10 points). These results show that in-rollout intervention not only lowers gold-answer NLL but also improves the accuracy of the teacher's generated answers.

Teacher conditioningall trajectoriesincorrect-answer trajectories
OPSD68.23%32.05%
CIPI72.66%46.15%

Questions

  1. Everything is measured on Qwen3-1.7B and Qwen3-4B, one model family that now looks somewhat dated, and only on competition mathematics, yet the story is told about OPSD in general.
  2. The method is validated for only one kind of PI. Every PI used in the paper is the full reference solution, split into eight fragments with the answer sentence removed, and the ablations change only how these fragments are inserted, not what the PI is. It is therefore unclear whether the selection rule transfers to other common forms such as answer-only supervision, partial hints, execution feedback or a correct sibling rollout.

Response. Our study primarily focuses on mathematical reasoning, while the PI-based intervention framework can also be adapted to other settings. We have extended CIPI to MuSiQue and HotpotQA, two multi-hop QA benchmarks, using supporting-evidence annotations as PI. CIPI outperforms OPSD on both answer EM and F1 across both datasets and model scales. These results extend our evaluation beyond competition mathematics and demonstrate the applicability of CIPI to a different form of privileged information.

Adapting PI to evidence-based reasoning. On MuSiQue, each gold supporting paragraph is a candidate PI unit; on HotpotQA, each gold supporting sentence is a candidate unit. We preserve these natural units and allow their number to vary with the example. The student receives the question and candidate documents, while the teacher additionally receives the gold supporting-evidence annotations. CIPI applies its likelihood–discrepancy selection criterion to these evidence units and inserts selected evidence during the student rollout. The answer EM (exact match) and F1 scores (%) of the trained models are reported below.

DatasetMethod1.7B EM1.7B F14B EM4B F1
MuSiQueBase27.6838.1940.2651.43
MuSiQueOPSD28.8439.3740.7350.75
MuSiQueCIPI30.8939.8741.8452.37
HotpotQABase48.8363.9156.9973.40
HotpotQAOPSD49.4765.0457.4973.33
HotpotQACIPI50.9367.0058.7873.66

On MuSiQue, CIPI improves EM over OPSD by 2.05 points at 1.7B and 1.11 points at 4B. On HotpotQA, the corresponding gains are 1.46 and 1.29 points. These gains support the usefulness of in-rollout intervention for multi-hop evidence integration and show that its application extends beyond reference-solution PI in mathematics.


Question. The method adds a scoring pass at every intervention point and a substantial amount of extra training time.

Response. As reported in Appendix E, the end-to-end training throughput on 2 ×\times H20, including rollout generation, PI selection, teacher evaluation, and optimization, is 0.913 samples/s for CIPI versus 1.203 samples/s for OPSD, corresponding to a 31.8% increase in time per training step. Since the overall training process is relatively fast, the additional overhead is modest. Meanwhile, Figure 11 in Appendix E shows that CIPI still outperforms OPSD under a comparable wall-clock training budget, demonstrating that its gains are not merely a result of increased computational expenditure.

Thanks for your response. My concerns about dataset generalization and extra training time are partly solved, but my other concerns are still not well addressed.

  1. To my understanding, the new table matches the prefix length, but not where the PI is placed. In OPSD, the full PI is at the beginning, so the distance is the whole prefix. In CIPI, the PI is added again at each segment boundary, so the distance is shorter and starts again at every boundary. At 128 tokens, CIPI may have only the first insertion, or one more. The response does not say where the boundaries are, or how many tokens sit between the last PI and the answer.

  2. The student does not learn from answer NLL. It is trained with KL loss on every token. Answer NLL only looks at the final answer, so it cannot show that the teacher gives better guidance at each token. The response still does not close this gap.

  3. My concerns about different model families and different kinds of PI are still not answered.

Comments on the paper

1 comment

We sincerely thank all reviewers and the Area Chair for their time, effort, and constructive feedback on our work.

After carefully considering the reviewers' comments and reflecting on our current work, we recognize that the paper still has several limitations that require further improvement. We have therefore decided to withdraw our submission and take additional time to strengthen the work.

We greatly appreciate the reviewers' insightful suggestions, which will be valuable in guiding our future revisions. Thank you again for your thoughtful feedback and for the time and effort you have devoted to reviewing our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.