acceptodds
Under review as a conference paper at ICLR 2027

OPSRD: On-Policy Self-Role Distillation

Abstract

Role prompting elicits specialized behavior from large language models through an expert identity, offering a lightweight way to guide reasoning on demanding tasks. However, evaluating or distilling complete role-prompted answers can miss useful next-token preferences when the sampled solution remains incorrect. Transferring these preferences also requires an objective that reaches alternatives the student rarely predicts. We introduce OPSRD, which uses a fixed expert role as privileged teaching context for on-policy self-distillation without reference solutions. A role-free student generates a trajectory, and a frozen instance of the same base model supplies role-conditioned distributions on its exact prefixes, exposing alternatives beyond the sampled continuation. Teacher-weighted forward KL targets alternatives the student underestimates, with clipping to limit individual vocabulary contributions. Supervision is restricted to the highest-entropy half of student positions, concentrating learning where predictions are uncertain. Experiments on three competition-math benchmarks with Qwen3-1.7B, 4B, and 8B show improvements over the base models without role prompts at inference. Forward KL achieves the highest macro-averaged accuracy among the three evaluated divergences at every scale.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, self-distillation, large language models, role prompting, knowledge distillation, mathematical reasoning, selective distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WmnJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces OPSRD, a reference-free, on-policy self-role distillation method that investigates whether expert role prompting can still provide effective training supervision even when it fails to directly improve answer accuracy. The method employs a frozen version of the same base model as a "role teacher" to provide next-token distributions over reasoning prefixes generated by the student model; distillation is performed using truncated forward KL divergence and high-entropy position selection, thereby enabling the student to perform inference without role prompts. Experiments conducted across three model sizes (based on Qwen3) and three competitive mathematics benchmarks demonstrate that the method improves average accuracy, while ablation studies further analyze the effects of role positioning, distillation objectives, and supervision density.

Critical strengths
  1. This paper discusses the distinction between the direct utility and teacher value of role prompting. When the role of math expert was directly applied to Qwen3-1.7B, it yields no improvement; however, employing this role for distillation-based supervision on the teacher side did enhance student performance.

  2. The proposed OPSRD method achieves an average accuracy exceeding that of the base models across the Qwen3-1.7B, 4B, and 8B variants, as well as on the AIME 2024, AIME 2025, and HMMT 2025 mathematics benchmarks.

Critical weaknesses
  1. The independent contribution of the expert persona was not fully isolated. The expert prompt not only established the identity of a "mathematics expert" but also mandated identifying key structures, verifying steps, reconsidering questionable branches, and checking answers. The performance gains may stem from these specific reasoning instructions rather than the expert persona itself. Although the neutral teacher matched the "thinking" template and message placement, it did not match the instruction length and reasoning requirements.

  2. Mechanistic explanations for the core phenomenon remain largely speculative. The authors argue that the "role-model" teacher provides information about "preferences for the next token that are useful but invisible to full-answer evaluation," yet the primary evidence stems from the final accuracy achieved after distillation. The paper does not directly verify whether the candidate tokens boosted by the teacher are indeed more correct or correspond to critical reasoning steps, nor does it confirm whether students consequently make fewer specific reasoning errors.

  3. The experiments covered only the Qwen3 model family, mathematical tasks, and three benchmarks. It is recommended to conduct experiments on large-scale models from various model series and to test tasks with verifiable outcomes, such as code generation.

Questions for authors

See Weakness

Reviewer_q1UxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed a reference-free on-policy self-distillation method that uses an expert role prompt as privileged teacher context. The proposed method is evaluated on Qwen3 families across three different math benchmarks. The reported results show improvements over base models and over the answer-conditioned OPSD baseline. The paper further investigates the effects of role placement, divergence choice, and supervision density, and provides additional analysis of training stability and problem-level accuracy changes.

Critical strengths
  • The motivation is clear and interesting.
  • The ablation is adequate to support the author's method.
Critical weaknesses
  • Lack comparison between other OPSD method baselines, eg, U-OPSD, PW-OPSD, and ROSD.
  • Ablation not convincing. Please add Neutral Teacher + Top 50% Entropy KL setting.
Questions for authors
  • The benchmark score in Table 1 does not match OPSD paper, including Base and Answer OPSD. Please explain it and convince me the advantage does not come from weaken baseline.
  • What will happen if teacher thinking off? I am curious about it.
  • Can you provide ablation study that only enable teacher thinking but don't provide any priviledged information?
Reviewer_mtHxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes OPSRD, using a role-conditioned teacher to supervise a role-free student through on-policy distillation, clipped forward KL, and high-entropy token selection. Results on three mathematics benchmarks show improvements over the base models without role prompting at inference.

Critical strengths
  1. Interesting premise: role-conditioned distributions may provide useful supervision even when complete-answer accuracy does not improve.

  2. Relevant comparisons examine teacher context, role placement, divergence objectives, and selective supervision.

Critical weaknesses
  1. The method combines established techniques, making the methodological contribution relatively incremental.

  2. Evaluation covers small number of questions within one domain and model family, limiting evidence of generalization and robustness.

  3. Component benefits are not fully isolated: a matched neutral-teacher selection baseline is missing, and the advantage of entropy selection is inconclusive.

  4. Logical gaps and inconsistent presentation, particularly in the abstract, obscure the motivation and contribution.

Questions for authors

NA

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.