- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Unified Entropy-Calibrated Credit Redistribution for GRPO (UECR-GRPO), a framework integrating reinforcement learning with verifiable rewards (RLVR) and on-policy distillation (OPD) into a single clipped GRPO update for mathematical reasoning. It operates in two layers -- Path-Utility Unification (PUU) and Entropy-Calibrated Redistribution (ECR). Evaluated across five mathematical benchmarks (AIME24, AIME25, AMC23, HMMT25-Feb/Nov) using Qwen3-1.7B and Qwen3-4B students, UECR-GRPO outperforms strong baselines (Distilled RL, ATOD) by 0.89% and 0.56% average Avg@12 accuracy, respectively.
Critical strengths
-
The paper identifies a fundamental tension between RLVR and On-Policy Distillation (OPD): verifiers provide true correctness but lack local resolution (creating zero-variance groups), whereas teachers offer dense token-level signals that frequently conflict with final correctness and process quality. Rather than mixing signals ad hoc, the method cleanly separates trajectory-level ranking (PUU) from token-level credit assignment (ECR). This decoupling makes the role of the teacher transparent at both macro and micro scales.
-
Beyond standard benchmark scores, the paper designs targeted offline experiments (step-level validity alignment, group-resolution tracking, and held-out verifier gradient similarity) that directly test why and how the mechanism works.
Critical weaknesses
-
While the method shows consistent gains on the smaller 1.7B model, its advantage largely plateaus at the 4B scale (where it trails simpler baselines on 3 out of 5 benchmarks). This raises questions about whether complex credit redistribution remains necessary as base models become stronger and produce fewer degenerate rollout groups.
-
The framework introduces four new hyperparameters and requires computing full-vocabulary teacher entropy alongside standard log-probabilities. Without end-to-end retraining sensitivity studies or wall-clock efficiency comparisons, it is unclear if the modest accuracy gains (~0.5–0.9%) justify the added engineering and tuning overhead.
Questions for authors
-
Could you clarify the statement in Appendix F.7 that decontamination screening and checkpoint selection "still require confirmation from the completed runs"?
-
In Table 3, the jump from PUU only + w/o entropy (21.11%) to PUU only + w/o projection (25.17%) is massive (+4.06%), while adding projection (Full UECR-GRPO) adds 0.45%. Can you provide the actual numbers for the 6 additional controls listed in Appendix F.8 (especially with ECR enabled) and the exact wall-clock/memory overhead from Appendix F.9?