acceptodds
Under review as a conference paper at ICLR 2027

DOSD: Difficulty-privileged On-policy Self-Distillation

Abstract

On-policy self-distillation trains an unassisted student using a reference-conditioned teacher evaluated on the student's own trajectories. Reference access can supply useful supervision, but can also make that supervision depend on information unavailable to the student. We introduce Difficulty-privileged On-policy Self-Distillation (DOSD), which allocates teacher-side reference context using a per-problem rolling record of verified outcomes. An offline-initialized, fixed-length queue is updated with each current student response before the teacher scores that same response. Its success fraction determines the token fraction of a solution prefix exposed to the teacher, while the final answer remains separately available to the teacher and all privileged information remains hidden from the student. We formalize this training procedure, establish bounded changes in its exposure budget, and characterize the additional outcome-conditioned target variability introduced by immediate feedback. These properties motivate controlled tests of adaptive exposure; they do not establish a performance improvement or the elimination of privileged-information dependence. Across Qwen3-1.7B, 4B, and 8B, DOSD changes three-benchmark average accuracy relative to OPSD by , , and percentage points, respectively. In a Qwen3-8B-thinking controller comparison, DOSD exceeds a reported budget-calibrated fixed-prefix controller by 2.32 points and is 0.18 points below full-prefix OPSD. Because the available experiment record contains aggregate point estimates but not repeated seeds, complete run metadata, or realized context budgets, these results are descriptive rather than evidence of statistical significance or reduced end-to-end cost.

Primary area:
reinforcement learning
Keywords:
Post-training

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_hJFcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DOSD, an adaptive on-policy self-distillation method that adjusts the amount of reference reasoning exposed to the teacher based on a per-problem FIFO queue of verified student outcomes. The student generates unassisted rollouts, while the teacher provides token-level supervision conditioned on a difficulty-dependent reference prefix. The paper presents theoretical analyses of exposure stability and gradient variability, along with experiments on mathematical reasoning benchmarks using Qwen3 models.

Critical strengths

DOSD introduces a lightweight, training-free controller for allocating privileged teacher context based on historical student performance, without additional online competence rollouts. The paper also provides a careful theoretical characterization of exposure changes and outcome-conditioned gradient variability.

Critical weaknesses

W1. Insufficient justification for prefix-based context selection.

DOSD controls teacher context by truncating the reference solution from the beginning according to a difficulty-dependent token ratio (Eq. 5). However, informative reasoning steps may appear anywhere in a solution, and shorter prefixes may omit the most critical information. The paper provides neither empirical evidence that prefix length is an effective proxy for supervision quality nor comparisons with alternative context selection strategies. This weakens the motivation for the proposed allocation mechanism.

W2. Insufficient experimental validation of the proposed mechanism.

The improvements over OPSD are marginal for the 1.7B and 4B models (+0.44 and +0.36 points), while performance decreases for 8B (−0.78 points). Although DOSD outperforms the reported budget-calibrated fixed-prefix baseline in the 8B-thinking setting, the actual context budgets are not provided. Moreover, the experiments lack multiple seeds, history-specific ablations, and direct comparisons with existing adaptive exposure methods. Consequently, the evidence is insufficient to establish the benefits of the proposed difficulty-based allocation.

Questions for authors

Q1. Have the authors considered alternative context selection strategies, such as selecting informative reasoning steps or using middle/random segments instead of fixed prefixes? How does the choice of prefix truncation affect performance under matched token budgets?

Q2. Can the authors provide controlled ablations isolating the contribution of historical outcome memory (e.g., current-only or shuffled-history allocation), together with actual context budgets and multi-seed results? These would help determine whether the reported improvements arise from the proposed difficulty-aware mechanism.

Reviewer_5FiDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes DOSD, an on-policy self-distillation method that adaptively controls the amount of privileged information (PI) provided to the teacher based on the student's performance on each problem. Experimental results show that DOSD has some advantages over OPSD on several benchmarks.

Critical strengths
  1. The paper explores an interesting direction for using PI in OPSD.
  2. The paper provides some theoretical analysis of self-distillation.
Critical weaknesses
  1. Insufficient justification for the core motivation.

The paper does not justify why the student's historical accuracy should determine the amount of PI provided to the teacher. Simply truncating the reference reasoning to a prefix seems heuristic, and it is unclear why this would work.

  1. Missing experimental details and insufficient ablations.

The experimental setup is seriously underspecified. Essential details such as the FIFO queue length N, training steps, epochs, rollout configurations, and wall-clock time are not provided. Basic ablations of the core components are also missing.

  1. Marginal improvements and incomplete baselines.

The improvements over OPSD are marginal, and DOSD even underperforms OPSD in the 8B non-thinking setting. Most baseline comparisons are conducted in non-thinking mode, while DOSD additionally reports thinking-mode results. Although Table 3 includes a thinking-mode OPSD comparison for 8B, corresponding comparisons are missing for other methods and scales. Recent related methods such as RLSD are also missing. More complete thinking-mode comparisons are needed.

Questions for authors
  1. Could the authors provide stronger justification for why truncating the reference reasoning prefix is useful?

  2. Could the authors provide the missing experimental configurations and ablation studies?

  3. Could the authors provide thinking-mode results for the baselines and include recent related methods such as RLSD?

  4. Could the authors explain how the theoretical analysis supports the design choices of DOSD?

Reviewer_xLegOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

They work on On-policy self-distillation (OPSD) and modify how the teacher sees the privileged information. Instead of the teacher seeing the whole answer, the teacher is limited to the final answer plus a portion of the target reasoning chain. That portion is one minus the student’s accuracy on the task. For example, if the student has a 70% accuracy on a task, the teacher sees 30% of the target reasoning when guiding the student. To measure the accuracy, they maintain a queue for each question initially filled with N-1 success/fail records and after each student roleout, a new success/fail record is added to the queue, and the student accuracy is considered the performance on the last N roleouts.

Their results show that they on Qwen3 1.7B improve OPDS from 9.93 to 10.37 (avg on three math tasks) and in Qwen3-4B from 23.34 to 23.70, but on Qwen3 8B, their method reduces accuracy from 33.37 to 32.5. They also tested with thinking on Qwen3 8B, which OPSD is 64.72, and they reduced it to 64.54.

Critical strengths
  1. The writing is clear, and the idea is explained clearly.
  2. The theoretical results behind their method seem interesting.
Critical weaknesses
  1. Few baseline are evaluated. The PAINT method, for example, seems close enough to your method to warrant a comparison.
  2. The differences in the accuracies are not significant enough.
  3. Not having seed runs makes the above weakness more critical.
Questions for authors
  1. The 10% floor on the portion of the target reasoning chain seems to be arbitrary. Is there any ablation or reasoning behind the 10%?
  2. One interesting ablation can be the effect of the privileged information portion to the kl loss of teacher-student or gradient norm of the teacher. I'm curious to see if the method has some lr tunning effects by reducing the teacher gradient norm or not.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.