acceptodds
Under review as a conference paper at ICLR 2027

GR-OPD:Group-Relative Sequence Credit for On-Policy Distillation

10 views

Abstract

On-policy distillation (OPD) provides token-level teacher feedback, yet common sampled-token updates assign credit locally and omit the effects of earlier actions on subsequent feedback. We introduce GR-OPD (Group-Relative On-Policy Distillation), which decouples feedback computation from credit assignment by aggregating token feedback into calibrated sequence-level credit. We establish a theoretical basis for this approach by showing that, under ideal on-policy assumptions, broadcasting raw summed feedback to every response token yields an unbiased estimate of the negative gradient of the sequence-level reverse-KL objective. To support effective learning in practice, GR-OPD discounts later feedback that may become less reliable under distribution shift and uses same-prompt rollout groups to calibrate teacher guidance. It preserves relative response preferences while retaining a controlled fraction of the group mean, which we term collective teacher pressure. Across five Qwen3 teacher–student settings and six mathematical reasoning benchmarks, GR-OPD outperforms vanilla OPD in macro Avg@K in every setting, including gains of 10.9 and 11.5 percentage points in two settings with matched thinking patterns. It also maintains stable training under mismatched thinking patterns, where vanilla OPD collapses. Ablations show that retaining an appropriate amount of collective teacher pressure can improve learning, while excessive collective teacher pressure or an unrestricted feedback horizon can destabilize training.

Primary area:
reinforcement learning
Keywords:
on-policy distillation, sequence-level distillation, group-relative reinforcement learning, distribution shift, cross-pattern distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4XAUOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
2(−1)
Soundness
2
Read review
Paper summary

The paper studies credit assignment in on-policy distillation (OPD), where the common sampled-token update gives each token credit only from its own teacher–student log-ratio. It shows that, under ideal on-policy sampling, broadcasting the summed log-ratio of a whole response to every token is an unbiased estimate of the reverse-KL gradient. Building on this, it proposes GR-OPD, which turns token feedback into a discounted trajectory score (γ), normalizes the scores within same-prompt groups, and adds back a fraction λ of the group mean, termed collective teacher pressure. Experiments with five Qwen3 teacher–student pairings on six math benchmarks show large gains over vanilla OPD, including stable training in two pattern-mismatched pairings where vanilla OPD collapses.

Critical strengths
  1. Large and consistent gains, including where vanilla OPD fails. GR-OPD improves macro Avg@K over vanilla OPD in all five pairings, by more than 10 points in the two matched Instruct pairings. It also trains stably in both think-to-base pairings where vanilla OPD collapses (Table 1).

  2. A careful look at what makes the method stable. The paper does not only report the final recipe. It shows that training directly on the trajectory score collapses (Figure 1), and it traces how the feedback horizon (γ) and the retained group mean (λ) affect reward and response length (Figure 3, Table 2). This makes the failure modes and the role of each component easy to understand.

Critical weaknesses
  1. Vanilla OPD is the only baseline, and it barely improves the student. Table 1 does not report the starting scores of the student or teacher. According to the reference scores in Appendix C.4, vanilla OPD moves the Qwen3-1.7B-Base student from 23.1 to 23.3 and the Qwen3-1.7B-Instruct student from 33.0 to 34.1. With the 32B teacher, it lowers the Instruct student to 30.3. These are matched pairings, where OPD is expected to work. Table 6 uses different generation caps, so this comparison is approximate, but it suggests that vanilla OPD changes the student very little. Meanwhile, GR-OPD lifts the Instruct student to 45.1, above its own 4B teacher's 43.0. The 10-point margins in Table 1 are therefore measured against a baseline that barely moves the student. Whether the gains come from the teacher signal or from the group-relative update itself is not tested.

  2. The theory supports a different estimator from the one that is evaluated. Proposition 1 covers broadcasting the raw sum of log-ratios, which corresponds to γ = 1 and full retention of the group mean. The evaluated method instead uses a length-normalized discounted mean, group normalization, and λ = 0 in three of the five pairings. The paper itself describes λ = 0 as "a GRPO-style relative update." The configurations closest to the theory, γ = 1 and λ = 1, are exactly the ones that degrade or collapse in Sections 5.3 and 5.4. The contribution of "grounding sequence-level credit in OPD's original objective" therefore does not apply to the estimator behind the main results.

Questions for authors
  1. How does GRPO trained on the DAPO-Math verifier reward perform with the same prompts and training budget? This would show how much of the gain comes from the teacher signal rather than from the group-relative update.

  2. Why does vanilla OPD barely improve the student in the matched pairings? Were its learning rate and other settings tuned separately, or shared with GR-OPD?

  3. How should λ be chosen for a new teacher–student pair? Nearby values separate stable training from collapse, such as 0 versus 0.15 in the cross-pattern pairing.

Reviewer_mqFKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to convert token-level teacher feedback into sequence-level credit for on-policy distillation. It shows that broadcasting the summed teacher–student log-probability ratio yields an unbiased reverse-KL descent estimator under ideal on-policy assumptions. The practical method, GR-OPD, instead forms a discounted, normalized trajectory score, subtracts a fraction of the same-prompt group mean, normalizes by the group standard deviation, and broadcasts the resulting advantage to all response tokens for a clipped policy update. Across five Qwen3 teacher–student pairings and six mathematical reasoning benchmarks, it improves macro Avg@K over vanilla sampled-token OPD and avoids the reported collapse in two mismatched thinking-pattern settings.

Critical strengths
  1. Stable training where vanilla OPD collapses. In the two cross-pattern pairings, vanilla OPD collapses (8.45 and 10.07 macro Avg@K), whereas GR-OPD remains trainable and reaches 29.57 and 24.46 under the same reported prompt, rollout-batch, and step budgets. Macro Avg@K also improves in all five pairings. Because the collapse of vanilla OPD under pattern mismatch is a known practical problem, this stability result is the main reason I see practical value in the method.

  2. Meaningful component ablations and explicit limitations. Figure 3 varies the retained group-mean fraction and compares discounted with full-horizon feedback while holding the retention fraction fixed. These experiments show that simply aggregating feedback is insufficient and identify regimes of length growth and collapse. The appendix is careful about what the theory does and does not cover, and the conclusion acknowledges the restriction to Qwen3 mathematical reasoning and the lack of token-specific credit within a response. These ablations make the behavior of λ and γ transparent and the paper's claims testable.

Critical weaknesses
  1. The evaluation relies on a single, possibly under-optimized baseline. The only baseline, vanilla sampled-token OPD, barely changes the initial students' scores (+0.2, +1.1, and −2.7 points in the three matched pairings; Tables 1 and 6, although Table 6 is not a matched comparison), and its training-verifier score stays flat in the matched Instruct pairings of Figure 3(a). The two methods share hyperparameters, but vanilla OPD uses raw log-ratios as advantages while GR-OPD uses group-standardized scores, so their update magnitudes may differ substantially; it is unclear whether vanilla OPD is effectively optimized in this setup. A stronger sequence-credit baseline, such as the return-to-go-style estimator of MiniLLM (Gu et al., ICLR 2024), is also absent. This matters to my assessment because the main evidence for the method is its margin over this single baseline.

  2. The evidence does not show that the gains come from the proposed sequence-credit mechanism. The largest gains (λ = .25 in the matched Instruct pairings) coincide with response lengths growing several-fold beyond those of the nothink teachers, whereas λ = 0 keeps length stable and yields limited training-reward gains (Section 5.3, Figure 3(a)). Section 5.3 suggests that the student extends its reasoning with teacher-preferred content, but this explanation is not tested. Meanwhile, with γ = 0.99 the trajectory score is dominated by the first few hundred tokens, so by the argument in Appendix A.2, most tokens in a long response receive almost none of the future credit that Section 3 uses to motivate broadcasting. This matters to my assessment because the paper attributes the gains to better credit assignment for matching the teacher, while the evidence is equally consistent with the retained group mean pushing the student toward longer reasoning that departs from the teacher.

Questions for authors
  1. (Clarification, requested numbers) For vanilla OPD and GR-OPD in the two matched Instruct pairings, please provide per-step training curves of (i) the mean token-level reverse KL on rollouts (i.e., the negative mean of r_t), (ii) the mean absolute per-token advantage entering Eq. (6), and (iii) the gradient norm. Please also state whether vanilla OPD applies the same [−5, 5] feedback clipping as GR-OPD.
  2. (Requested experiment) Can the authors compare against (i) a return-to-go baseline, such as MiniLLM's length-normalized future-return estimator, and (ii) vanilla OPD with the positional weight γ^{t−1} and standard batch-level advantage normalization, so that it differs from GR-OPD at λ = 0 mainly in sequence-level aggregation, using the same teacher feedback, rollouts, clipping, and training budget?
  3. (Requested analysis) Please report evaluation response lengths and truncation rates for every row of Table 1, together with held-out results for GR-OPD at λ = 0 in the matched Instruct pairings. Since the λ = .25 students become several times longer than their nothink teachers during training, do they actually move closer to the teacher? Reporting the teacher–student reverse KL (or mean log-ratio) on held-out responses for vanilla OPD and GR-OPD would show whether the gains come with closer teacher matching, as the credit-assignment interpretation implies. Please also report the training wall-clock time of each method.
  4. (Requested experiment) How was γ = 0.99 chosen, and how does its concentration on roughly the first 100–300 tokens relate to the 1,500–3,000-token peaks in Figure 1(b)? A short-prefix scoring control that keeps full student rollouts and the same update, together with intermediate γ values (e.g., 0.999, 0.9995), would clarify whether later feedback contributes to the gains.
  5. (Clarification) λ differs across pairings in Table 1 (0 vs. .25). How was λ selected for each pairing, and were the evaluation benchmarks used for this selection? All runs appear to use a single seed (42, Table 3); can the main gains and the collapse/stability patterns be supported with independent runs?
Reviewer_4VVCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies credit assignment in on-policy distillation (OPD). Standard OPD uses each token’s teacher-student log-ratio as its advantage. The authors argue this credit is too local, and propose GR-OPD, which aggregates token-level feedback into a sequence-level score and broadcasts it to all tokens. They first show that broadcasting the summed log-ratio is an unbiased estimate of the reverse-KL gradient under ideal on-policy assumptions. Because direct broadcasting is unstable in practice, GR-OPD adds two controls. A discount factor gamma down-weights later tokens. A GRPO-style group normalization subtracts the same-prompt mean while retaining a fraction lambda of it, which the authors call “collective teacher pressure.” Experiments on five Qwen3 teacher–student pairs and six math benchmarks show consistent gains over vanilla OPD, including stable training in cross-pattern settings where vanilla OPD collapses. Ablations show that a large lambda or gamma=1 leads to length explosion and reward collapse. Although the empirical results demonstrate the effectiveness of the method on Qwen3 model families, some major concerns (see the weakness part) prevent me from giving higher scores.

Critical strengths
  1. Strong and consistent empirical gains. GR-OPD improves over vanilla OPD on all five teacher–student pairs, with gains of about 11 points in macro Avg@K on the matched Instruct pairs. It also trains stably in the cross-pattern settings, where vanilla OPD collapses. The method is simple and easy to adopt.

  2. Clear and honest presentation. The paper is upfront about the gap between theory and practice: Appendix B states that the unbiasedness result does not carry over to the practical estimator, and A.5 avoids over-interpreting lambda. The implementation details are thorough (EOS alignment, pinned checkpoints, clipping), which makes the work easy to reproduce.

Critical weaknesses
  1. Experiments are limited to the Qwen3 family. All five teacher-student pairs come from Qwen3, so it is unclear whether the gains transfer to other model families. This matters more than usual here, for two reasons. First, prior work has shown that Qwen models can improve on math benchmarks even under weak or spurious training signals (e.g., Shao et al., 2025, “Spurious Rewards”), so gains observed only on Qwen are hard to attribute to the method itself. Second, Qwen3 Instruct models are hybrid think/nothink models. A nothink student may simply be pushed into its built-in long-reasoning behavior, which would be consistent with the large length growth in Figure 3(a). In addition, every pair shares the same tokenizer and pretraining lineage, so the claimed robustness to teacher-student mismatch is tested only in a narrow sense. Results on at least one other family (e.g., Llama or OLMo), ideally including a cross-family pair, would make the conclusions much more convincing.
  2. The explanation of lambda (“collective teacher pressure”) is not convincing. Under on-policy sampling, E[r_t | h_t] = −KL ≤ 0, so the group mean is typically negative. Setting lambda > 0 therefore adds a negative constant to all advantages in a group. Because the loss is averaged within each response, a constant negative advantage yields an expected gradient proportional to ∇E[1/L], which pushes responses to become longer. This is the length bias identified by Dr. GRPO. It may explain the observed collapse at large lambda, and also the collapse of vanilla OPD, which uses the same aggregation. Under this view, lambda mainly controls length inflation rather than how much “teacher guidance” the student receives. The authors should report the sign of S̄ during training and rerun the lambda ablation with token-sum aggregation to check whether the collapse persists.
  3. Limited novelty. The method is largely a combination of OPD and GRPO. Proposition 1 is the standard REINFORCE estimator with the sequence log-ratio as the return, already derived in MiniLLM (Gu et al., 2024), as the authors acknowledge. At lambda = 0, which is used in three of the five main settings, GR-OPD reduces to GRPO with the (discounted, length-normalized) teacher-student log-ratio as the reward. Group normalization of trajectory-level OPD scores has also been proposed in GC-OPD (Zhang et al., 2026). What remains new is the discount gamma and the retention fraction lambda. Both are simple heuristics, and as discussed above, the justification for lambda is questionable. The paper should clarify what it contributes beyond “GRPO with a teacher-derived reward,” and compare directly with GC-OPD.
Questions for authors
  1. The text in Figures 1 and 2 is too small to read at normal print size. Please enlarge the fonts, especially in Figure 2.
  2. With gamma = 0.99, about 95% of the score weight falls on the first ~300 tokens, while Figure 1(b) shows that J_H peaks at 1.5k–3k tokens. Why not use a larger gamma (e.g., 0.999)?
  3. Vanilla OPD improves the student only marginally (33.0 → 34.1 for 4B→1.7B Instruct, and 23.1 → 23.3 for Base→Base). I’m wondering whether the baseline is well-tuned. GR-OPD’s advantages are normalized to O(1) while vanilla OPD uses raw log-ratios, so their effective step sizes differ a lot under the same learning rate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.