acceptodds
Withdrawn from ICLR 2027

When and Where to Trust the Teacher: Unifying On-Policy Distillation and GRPO through Entropy-Calibrated Credit Assignment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Reinforcement learning with verifiable rewards (RLVR) supervises mathematical reasoning through final-answer correctness, but provides little guidance on individual tokens. On-policy distillation (OPD) supplies dense feedback on student-generated responses, yet teacher preference need not reflect correctness. Recent hybrids combine OPD and verifier-derived advantages or reweight task credit using teacher ratios. However, teacher guidance enters after verifier-based group normalization, and token reweighting need not preserve the total task credit assigned to each response. We introduce Unified Entropy-Calibrated Credit Redistribution for GRPO (UECR-GRPO), which integrates verifier and teacher signals within a single GRPO-style update at both the response and token levels. Path-Utility Unification (PUU) combines verifier reward and a teacher-to-anchor path log-ratio in a single KL-regularized objective. Its on-policy implementation uses a length-normalized teacher score and combines both rewards before group normalization and PPO clipping, allowing teacher evidence to influence the response ranking. Entropy-Calibrated Redistribution (ECR) then uses the signed teacher–old-policy token gap to redistribute the verifier-derived component. Full-vocabulary teacher entropy attenuates uncertain guidance, while a response-wise zero-sum projection preserves the total task credit and its token-wise sign before clipping. Across five mathematical reasoning benchmarks, UECR-GRPO achieves average \(Avg@12\) accuracies of 17.21% and 65.09% with Qwen3-1.7B and Qwen3-4B students, respectively, exceeding the strongest baseline at each scale by 0.89 and 0.56 percentage points.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, On-Policy Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mUdgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Unified Entropy-Calibrated Credit Redistribution for GRPO (UECR-GRPO), a framework integrating reinforcement learning with verifiable rewards (RLVR) and on-policy distillation (OPD) into a single clipped GRPO update for mathematical reasoning. It operates in two layers -- Path-Utility Unification (PUU) and Entropy-Calibrated Redistribution (ECR). Evaluated across five mathematical benchmarks (AIME24, AIME25, AMC23, HMMT25-Feb/Nov) using Qwen3-1.7B and Qwen3-4B students, UECR-GRPO outperforms strong baselines (Distilled RL, ATOD) by 0.89% and 0.56% average Avg@12 accuracy, respectively.

Critical strengths
  1. The paper identifies a fundamental tension between RLVR and On-Policy Distillation (OPD): verifiers provide true correctness but lack local resolution (creating zero-variance groups), whereas teachers offer dense token-level signals that frequently conflict with final correctness and process quality. Rather than mixing signals ad hoc, the method cleanly separates trajectory-level ranking (PUU) from token-level credit assignment (ECR). This decoupling makes the role of the teacher transparent at both macro and micro scales.

  2. Beyond standard benchmark scores, the paper designs targeted offline experiments (step-level validity alignment, group-resolution tracking, and held-out verifier gradient similarity) that directly test why and how the mechanism works.

Critical weaknesses
  1. While the method shows consistent gains on the smaller 1.7B model, its advantage largely plateaus at the 4B scale (where it trails simpler baselines on 3 out of 5 benchmarks). This raises questions about whether complex credit redistribution remains necessary as base models become stronger and produce fewer degenerate rollout groups.

  2. The framework introduces four new hyperparameters and requires computing full-vocabulary teacher entropy alongside standard log-probabilities. Without end-to-end retraining sensitivity studies or wall-clock efficiency comparisons, it is unclear if the modest accuracy gains (~0.5–0.9%) justify the added engineering and tuning overhead.

Questions for authors
  1. Could you clarify the statement in Appendix F.7 that decontamination screening and checkpoint selection "still require confirmation from the completed runs"?

  2. In Table 3, the jump from PUU only + w/o entropy (21.11%) to PUU only + w/o projection (25.17%) is massive (+4.06%), while adding projection (Full UECR-GRPO) adds 0.45%. Can you provide the actual numbers for the 6 additional controls listed in Appendix F.8 (especially α=0\alpha=0 with ECR enabled) and the exact wall-clock/memory overhead from Appendix F.9?

Reviewer_a92aOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes UECR-GRPO, combining verifier rewards with on-policy teacher guidance. PUU adds a response-level teacher log-probability-ratio score to the task reward before GRPO normalization. ECR then uses teacher entropy and a zero-sum projection to redistribute the task-derived advantage across tokens while preserving its sum and sign before clipping. Evaluation covers two Qwen3 student sizes and five mathematics benchmarks, supported by component ablations, reasoning-step audits, and gradient-alignment diagnostics.

Critical strengths
  1. ECR provides a controlled way to redistribute task credit. Its positive token weights and zero-sum adjustment preserve the task component’s sign and response-wise sum before PPO clipping. This allows teacher-preferred tokens in failed responses to receive less negative task credit without reversing that component’s sign. The paper clearly states the limits of these guarantees.

  2. The experiments provide evidence that the combined method is useful. UECR-GRPO improves the five-benchmark average over baseline at the two model sizes. On the three-benchmark ablation suite, adding ECR improves PUU. The blinded reasoning-step audit also finds better error localization than shuffled-entropy controls, providing evidence beyond aggregate accuracy.

Critical weaknesses
  1. The novelty of PUU’s pre-normalization integration is overstated.

    Equations (6)–(7) implement teacher-based reward shaping followed by GRPO normalization:

    RiU=Ritask+αRiT,AiU=RiU−μUσU+ϵ.R_{i}^{U}=R_{i}^{\mathrm{task}}+\alpha R_{i}^{T},\qquad A_{i}^{U}=\frac{R_{i}^{U}-\mu_{U}}{\sigma_{U}+\epsilon}.

    KDRL (Xu et al., 2025) already studies adding summed teacher/student log-probability ratios to outcome rewards before calculating GRPO advantages. Although the paper cites KDRL, its discussion presents this integration order as a distinction of PUU. PUU uses a token average instead of a sum and combines it with ECR. These differences may matter as KDRL reported instability with its reward-shaping variant, but pre-normalization integration itself is not new. This narrows the contribution to the specific changes to reward construction and token-level redistribution.

  2. The entropy ablation does not separate better token placement from weaker redistribution.

    Equation (10) defines

    qi,t=12mi,tci,t(di,t−μic),wi,t=1+ρqi,t.q_{i,t}=\frac{1}{2}m_{i,t}c_{i,t}(d_{i,t}-\mu_{i}^{c}),\qquad w_{i,t}=1+\rho q_{i,t}.

    Setting ci,t=1c_{i,t}=1 removes both entropy-dependent token weighting and attenuation of the redistribution magnitude. Even a constant confidence below one would reduce the effective strength of ρ\rho. Consequently, the improvement from without entropy to with full ECR does not isolate the benefit of using entropy to decide where credit should go.

    The offline shuffled-entropy audit provides useful evidence that entropy helps locate errors. However, the training ablation does not separate this positional benefit from the effect of reducing redistribution strength. This distinction matters because the paper motivates entropy calibration as a way to improve token-level credit assignment.

  3. The small gains over strong baselines lack evidence of statistical reliability.

    Table 2 reports average gains of only 0.89 percentage points over ATOD-aligned for Qwen3-1.7B and 0.56 over Distilled RL for Qwen3-4B. Neither confidence intervals for these accuracy differences nor variability across independent training runs are reported. Avg@12 averages decoding samples, while the uncertainty analyses elsewhere concern offline diagnostics. It therefore remains unclear whether these gains are robust to training and evaluation variability.

Questions for authors
  1. Beyond replacing the token sum with a token average, what distinguishes PUU from KDRL’s reward-shaping variant?

  2. Does full ECR still outperform the no-entropy variant when ρ\rho is adjusted to match the magnitude of token-weight deviations from one?

  3. When all task rewards in a group are identical, Aitask∣U=0A_{i}^{\mathrm{task}\mid U}=0, so ECR disappears and learning relies on the teacher-derived advantage. What evidence shows that these updates improve downstream accuracy, particularly for all-incorrect groups where the teacher term receives no entropy attenuation?

Reviewer_X1ssOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes UECR-GRPO for training math reasoning models with a teacher and a final-answer verifier. Its first component, PUU, combines the verifier reward with a teacher score before GRPO compares answers to the same problem. Its second component, ECR, uses the teacher’s token preferences to decide which parts of an answer receive more credit. When the teacher is uncertain, ECR makes a smaller adjustment. Across five math benchmarks, the method reaches 17.21% average accuracy with a 1.7B student and 65.09% with a 4B student.

Critical strengths
  • ECR can give a smaller penalty to a good reasoning step inside a wrong answer. Its rule preserves each response’s total verifier credit and keeps the sign of that credit before clipping.

  • The comparisons include two recent methods that also combine teacher feedback with reinforcement learning. All teacher-assisted methods use the same teacher within each model size. Distilled RL, ATOD

  • Table 3 separates the two components. On three benchmarks with the 1.7B student, adding ECR to PUU raises average accuracy from 20.28% to 25.62%.

Critical weaknesses
  • The gains over the strongest baselines are 0.89 and 0.56 percentage points. The 4B student leads on only two of five benchmarks, and Table 2 gives no uncertainty estimates for these differences.

  • When all answers receive the same verifier reward, ECR has no verifier credit to redistribute. The teacher then determines the ranking, yet its preference agreed with a blinded judge on reasoning quality in only 56.8% of 347 tied pairs.

  • Appendix F.7 says the training–test overlap checks and the checkpoint selection rule still need confirmation. Both details matter when the reported lead is less than one percentage point.

Questions for authors
  • How was the 1.7B model’s 52.04% AMC23 score calculated? With 40 problems and 12 answers each, the nearest single-run result is 250 correct out of 480, or 52.08%. Was the score averaged across runs or checkpoints?

  • Do the gains hold across independent training runs using a fixed checkpoint rule? Can the authors report the overlap checks for all five benchmarks?

  • What happens if the teacher signal is reduced in groups where every answer is wrong? This would test whether breaking those ties helps the student learn.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.