- Rating
- 2
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 2(−1)
- Soundness
- 2
Read review
Paper summary
The paper studies credit assignment in on-policy distillation (OPD), where the common sampled-token update gives each token credit only from its own teacher–student log-ratio. It shows that, under ideal on-policy sampling, broadcasting the summed log-ratio of a whole response to every token is an unbiased estimate of the reverse-KL gradient. Building on this, it proposes GR-OPD, which turns token feedback into a discounted trajectory score (γ), normalizes the scores within same-prompt groups, and adds back a fraction λ of the group mean, termed collective teacher pressure. Experiments with five Qwen3 teacher–student pairings on six math benchmarks show large gains over vanilla OPD, including stable training in two pattern-mismatched pairings where vanilla OPD collapses.
Critical strengths
-
Large and consistent gains, including where vanilla OPD fails. GR-OPD improves macro Avg@K over vanilla OPD in all five pairings, by more than 10 points in the two matched Instruct pairings. It also trains stably in both think-to-base pairings where vanilla OPD collapses (Table 1).
-
A careful look at what makes the method stable. The paper does not only report the final recipe. It shows that training directly on the trajectory score collapses (Figure 1), and it traces how the feedback horizon (γ) and the retained group mean (λ) affect reward and response length (Figure 3, Table 2). This makes the failure modes and the role of each component easy to understand.
Critical weaknesses
-
Vanilla OPD is the only baseline, and it barely improves the student. Table 1 does not report the starting scores of the student or teacher. According to the reference scores in Appendix C.4, vanilla OPD moves the Qwen3-1.7B-Base student from 23.1 to 23.3 and the Qwen3-1.7B-Instruct student from 33.0 to 34.1. With the 32B teacher, it lowers the Instruct student to 30.3. These are matched pairings, where OPD is expected to work. Table 6 uses different generation caps, so this comparison is approximate, but it suggests that vanilla OPD changes the student very little. Meanwhile, GR-OPD lifts the Instruct student to 45.1, above its own 4B teacher's 43.0. The 10-point margins in Table 1 are therefore measured against a baseline that barely moves the student. Whether the gains come from the teacher signal or from the group-relative update itself is not tested.
-
The theory supports a different estimator from the one that is evaluated. Proposition 1 covers broadcasting the raw sum of log-ratios, which corresponds to γ = 1 and full retention of the group mean. The evaluated method instead uses a length-normalized discounted mean, group normalization, and λ = 0 in three of the five pairings. The paper itself describes λ = 0 as "a GRPO-style relative update." The configurations closest to the theory, γ = 1 and λ = 1, are exactly the ones that degrade or collapse in Sections 5.3 and 5.4. The contribution of "grounding sequence-level credit in OPD's original objective" therefore does not apply to the estimator behind the main results.
Questions for authors
-
How does GRPO trained on the DAPO-Math verifier reward perform with the same prompts and training budget? This would show how much of the gain comes from the teacher signal rather than from the group-relative update.
-
Why does vanilla OPD barely improve the student in the matched pairings? Were its learning rate and other settings tuned separately, or shared with GR-OPD?
-
How should λ be chosen for a new teacher–student pair? Nearby values separate stable training from collapse, such as 0 versus 0.15 in the cross-pattern pairing.