acceptodds
Under review as a conference paper at ICLR 2027

Reference-Free On-policy Self-Distillation via Self-Reflection

Abstract

On-policy self-distillation (OPSD) provides dense, token-level supervision through a teacher policy conditioned on privileged information (PI), typically in the form of external reference solutions that require additional annotation or generation. Besides this acquisition cost, mismatches in reasoning paths and styles between these references and the student's own rollouts can impair student learning, motivating the use of PI that better aligns with the student's own reasoning. Reflection offers a natural source of such PI: it revisits a student's responses to provide feedback tailored to the reasoning they contain. We therefore investigate whether conditioning the teacher on self-reflections generated by the shared policy can improve on-policy self-distillation by providing more effective token-level supervision. We propose Reference-Free Self-Distillation via Self-Reflection (RF-SD), where a single policy serves as the student, reflector, and teacher. Given a question and a student rollout, the reflector is routed by the verifier outcome to summarize useful reasoning in correct trajectories or diagnose errors in incorrect ones. The resulting reflection replaces external references as the teacher's PI for token-level supervision on the student's own rollout. We further introduce RF-SD-e, which evolves the privileged information by optimizing the reflector using the correctness of reflection-conditioned teacher responses as rewards. Experiments on challenging mathematical reasoning benchmarks show that RF-SD outperforms strong baselines across different model scales, with further gains from reflector evolution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-distillation, Self-reflection, Reinforcement learning, Large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EgoiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed a self-distillation method that does not need a golden solution, replacing it instead with self-generated reflections. For each student rollout, a verifier determines its correctness, and a reflector prompts. Either summarize a useful reasoning pattern from a correct answer, or diagnose the incorrect part of an incorrect answer. The reflections serve as privileged information for the teacher and provide token-level supervision on a student's trajectory. The loss function is a combination of GRPO loss and the self-distillation loss, while in the successful case, the coefficient of the self-distillation loss is very small. They also further propose and improve the method that optimizes the reflector using GRPO. Experiments on QN3 and MATH benchmarks show that the proposed method significantly increases the accuracy.

Critical strengths
  • The method is well-defined and the presentation is easy to follow.
  • The method reduces dependency on the golden solution, and can work even if no rollout is correct.
  • The experiment covers many models and many benchmarks, which is convincing. The improvement on benchmark score is significant.
Critical weaknesses
  • Using reflection information as priviledged information looks like not novel while many other work has done that, eg, ROSD.
  • Lack key comparison to other method that use reflection as priviledged info, eg, ROSD, AMR-SD, and SD-Zero.
  • It looks like a OPSD-enhanced GRPO method, rather than an OPSD method, since the coefficient of self-distillation loss is chosen as 1000$\times$ smaller than GRPO loss.
  • Code is not provided in supp materials.
Questions for authors
  • Please add baseline of self distillation method that use reflection as priviledged information.
  • What will happen if GRPO term in loss is removed?
  • Please also compare some improved GRPO method, eg, Dr. GRPO, DAPO.
  • Compare with other method combine self-distillation and RLVR, eg, SDAR.
  • The author mentioned Dr. SD in many figures, eg, Figure 3, looks like typo.
Reviewer_RETYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper proposes RF-SD, an on-policy self-distillation framework that uses verifier-guided reflections as privileged information for a teacher sharing the student’s parameters. Separate reflection prompts preserve useful reasoning in correct responses and diagnose errors in incorrect ones. Training combines GRPO with token-level distillation. An extension, RF-SD-e, optimizes the reflector using teacher-answer correctness. Experiments on three Qwen3 model scales show improvements over the evaluated baselines in non-thinking mode.

Critical strengths
  1. The paper is well organized and easy to follow, with clear descriptions of the framework, training procedure, and experimental setup.
  2. Using reflections on the student’s own responses as teacher guidance is well motivated. It avoids requiring external reference solutions while retaining verifier feedback.
Critical weaknesses
  1. Experiments are restricted to non-thinking mode, leaving RF-SD’s effectiveness with native thinking models unclear. Since explicit reasoning may already involve self-checking and correction, does reflection-conditioned supervision still provide additional benefits when the student’s thinking mode is enabled?
  2. At the final checkpoint, the average improvement on 4B comes entirely from AIME25, while the other three benchmarks decline. HMMT25 performance decreases at all three model scales. The authors should explain these trade-offs.
  3. Equal training steps do not imply equal computational budgets. RF-SD adds reflection generation and teacher scoring; RF-SD-e further adds candidate reflections, full teacher-answer generation, and reflector updates. Please report training time or GPU-hours on identical hardware and include compute-matched GRPO baselines.
  4. Table 2 changes both access to verifier labels and prompt specialization, preventing attribution of the gains specifically to dual reflection. The hinted variant introduces answers and reference solutions without resolving this confound. Table 3 inherits this issue and compares different training runs, limiting its support for the proposed mechanism.
  5. The figures consistently use Dr. SD and Dr. SD-e, whereas the text refers to RF-SD and RF-SD-e. The former names are never defined. Please clarify the correspondence and use consistent terminology throughout.
Questions for authors

See Weaknesses

Reviewer_yB3WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RF-SD, a reference-free on-policy self-distillation framework that uses self-generated reflections as privileged information for a teacher sharing the student's underlying model. A verifier routes each student rollout to either a success-oriented reflection prompt that preserves useful reasoning or a failure-oriented prompt that attempts to diagnose mistakes. The resulting reflection-conditioned teacher provides token-level supervision alongside GRPO. The paper further introduces RF-SD-e, which optimizes a separate reflector adapter using the correctness of reflection-conditioned teacher responses. Experiments across three Qwen3 model scales and four mathematical reasoning benchmarks show improvements over RL and self-distillation baselines, with additional gains from reflector evolution.

Critical strengths
  • The paper addresses an important limitation of reference-conditioned self-distillation by constructing privileged information from the student's own reasoning trajectory.
  • The verifier-guided dual-reflection design is intuitive and well motivated.
  • The experimental analysis is impressive. The incorrect-to-correct and correct-to-incorrect transition statistics provide useful evidence that the proposed reflection mechanism improves teacher behavior.
Critical weaknesses
  • The experiments do not sufficiently isolate the contribution of reflection content from the correctness information used to construct it. The verifier outcome already provides a strong supervision signal and determines which reflection prompt is used. It is therefore unclear whether the gains come from meaningful diagnostic information in the generated reflection or simply from correctness-conditioned teacher prompting.
  • The actual quality and faithfulness of the generated reflections are not directly evaluated. A reflection may lead to a correct teacher answer without accurately identifying the student's reasoning error. This concern also applies to RF-SD-e, where reflector quality is optimized using teacher-answer correctness rather than the accuracy of the reflection itself. Furthermore, the incremental gains from reflector evolution are relatively small compared with the additional generation and optimization costs, and a compute-matched comparison is missing.
Questions for authors
  • How much of RF-SD's improvement remains if the teacher receives only the verifier outcome rather than the generated reflection? A comparison among label-only, empty, shuffled, and genuine reflections would help isolate the contribution of reflection content.
  • It is unclear whether failure reflections correctly identify the first or most consequential reasoning error.
  • In RF-SD-e, why is correctness of the teacher's newly generated answer an appropriate proxy for reflection quality? Are there examples where a reflection receives a high reward despite incorrectly describing the student's original reasoning?
  • How does RF-SD-e compare with spending the same additional compute on more student GRPO rollouts or additional RF-SD training steps? This seems important given the relatively small incremental gains from reflector optimization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.