- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposed a self-distillation method that does not need a golden solution, replacing it instead with self-generated reflections. For each student rollout, a verifier determines its correctness, and a reflector prompts. Either summarize a useful reasoning pattern from a correct answer, or diagnose the incorrect part of an incorrect answer. The reflections serve as privileged information for the teacher and provide token-level supervision on a student's trajectory. The loss function is a combination of GRPO loss and the self-distillation loss, while in the successful case, the coefficient of the self-distillation loss is very small. They also further propose and improve the method that optimizes the reflector using GRPO. Experiments on QN3 and MATH benchmarks show that the proposed method significantly increases the accuracy.
Critical strengths
- The method is well-defined and the presentation is easy to follow.
- The method reduces dependency on the golden solution, and can work even if no rollout is correct.
- The experiment covers many models and many benchmarks, which is convincing. The improvement on benchmark score is significant.
Critical weaknesses
- Using reflection information as priviledged information looks like not novel while many other work has done that, eg, ROSD.
- Lack key comparison to other method that use reflection as priviledged info, eg, ROSD, AMR-SD, and SD-Zero.
- It looks like a OPSD-enhanced GRPO method, rather than an OPSD method, since the coefficient of self-distillation loss is chosen as 1000$\times$ smaller than GRPO loss.
- Code is not provided in supp materials.
Questions for authors
- Please add baseline of self distillation method that use reflection as priviledged information.
- What will happen if GRPO term in loss is removed?
- Please also compare some improved GRPO method, eg, Dr. GRPO, DAPO.
- Compare with other method combine self-distillation and RLVR, eg, SDAR.
- The author mentioned Dr. SD in many figures, eg, Figure 3, looks like typo.