- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a reinforcement learning method called Delta-GRPO. By leveraging the policy shift accumulated by the model since its initial state, this method enables a redistribution of credit—refined to the token level—that aligns with the final outcome within the GRPO framework.
Critical strengths
It integrates "token-level policy-shift" information into the GRPO framework, eliminating the need for an external teacher model or an additional distillation stage. The formula is intuitively designed while preserving the mean and sign characteristics of the original "response-level advantages."
When applied to Qwen3-4B and Qwen3-8B models, Delta-GRPO substantially outperforms GRPO and other baselines across eight mathematical reasoning benchmarks. Notably, in terms of macro accuracy, it achieves improvements of 13.59 and 12.04 percentage points, respectively, over GRPO.
The empirical analysis is comprehensive, covering component ablation studies, investigations into "modulation strength," Pass@k evaluations, and an analysis of training dynamics. The ablation results confirm the importance of both "outcome alignment" and "response centering."
Critical weaknesses
Several closely related methods—such as Direct-OPD, OPD², EP-GRPO, and SC-GRPO—have already explored policy divergence signals or credit reweighting at the token granularity level. The proposed approach appears to be merely an incremental combination of existing ideas, and the distinction between it and EP-GRPO is not clearly defined.
Changes in token probabilities do not necessarily serve as an accurate reflection of a token's contribution to the correctness of the reasoning process. The paper lacks compelling ablation studies—such as shuffling the sequence of policy shifts or employing random, mean-preserving token weights—to demonstrate that the observed performance gains genuinely stem from meaningful historical information.
Delta-GRPO increases computational time per training step by approximately 55.6%; however, the primary comparisons presented in the paper do not account for the total computational budget used during training. Furthermore, the weak performance of EP-GRPO raises questions about whether the chosen baseline implementations are sufficiently competitive.
Delta-GRPO significantly reduces token entropy and increases the proportion of sample groups with zero reward variance, raising concerns about a potential decline in the model's exploratory capacity. Moreover, the experiments are limited to two Qwen3 models and mathematical reasoning tasks, leaving the broader applicability of the method unclear.
Questions for authors
see weaknesses
4 comments
Response to Reviewer qbaV [Part 2/3]
2. Shuffle and Random Mean-Preserving Ablations (W2)
We ran the shuffle ablation you propose. Removing the token-to-history correspondence drops macro accuracy from 66.39 to 49.63, below GRPO.
Theoretical safeguard. The external answer verifier supplies fresh correctness feedback on current rollouts, which, together with length shaping, determines the group-relative advantage. Historical shifts redistribute this advantage through positive multipliers within (), preserving its sign and response-level total while changing token priorities (Proposition 1, Eq. 6).
Table R3. Qwen3-4B, average accuracy (%).
| Method | MATH500 | AIME24 | AIME25 | AMC23 | Minerva | Olymp. | HMMT25 | AIME26 | Macro | Δ vs. GRPO |
|---|---|---|---|---|---|---|---|---|---|---|
| GRPO | 92.10 | 41.35 | 35.31 | 87.11 | 43.84 | 67.06 | 18.85 | 36.77 | 52.80 | — |
| No alignment | 93.27 | 46.46 | 38.65 | 87.89 | 44.39 | 68.43 | 21.15 | 40.94 | 55.15 | +2.35 |
| Shuffled shift (new) | 90.22 | 33.36 | 30.08 | 82.54 | 41.91 | 62.17 | 20.03 | 36.69 | 49.63 | −3.17 |
| Delta-GRPO | 95.50 | 66.88 | 55.21 | 96.17 | 46.23 | 74.87 | 36.98 | 59.27 | 66.39 | +13.59 |
The shuffled multipliers are themselves random, mean-preserving token weights: each response's weights have mean one and are assigned to tokens at random. They also match the exact multiset of the full method, so the shuffle isolates the token-to-history correspondence from any change in the weight distribution. Without that correspondence, the method falls below the GRPO baseline it modifies.
Token-level case studies (Appendix E.3, Figure 8). Figure 8 visualizes token by token across four responses (two correct and two incorrect), with red/blue indicating multipliers above/below one. In the two correctly solved, positive-advantage cases, self-verification windows (e.g., "But wait, let me think" and "But wait, maybe I need to think more generally…") have mean multipliers of 1.00345 and 1.00407, versus 0.99680 and 0.99749 for the respective final-answer windows. Relative to uniform GRPO weighting, this redistributes positive credit toward self-verification reasoning and away from final-answer tokens.
Manuscript Revision. We have added the shuffled-shift row to Table 2 (§5.2).
3. Training Compute and Baseline Competitiveness (W3)
(a) Compute. Delta-GRPO reaches better results in fewer steps and stays ahead as training continues. On online MATH500, it exceeds GRPO's final step-140 accuracy at step 60 (92.03 vs. 91.80) and reaches 93.98 at step 140. Checkpoints taken well before GRPO's 20.668 training hours already exceed GRPO's final checkpoint:
Table R4. Online validation accuracy (%) within the GRPO wall-clock budget, Qwen3-4B.
| Benchmark | GRPO, step 140 (20.668 h) | Delta-GRPO, earlier checkpoint | Δ |
|---|---|---|---|
| MATH500 | 91.80 | 92.20 (step 80, 15.796 h) | +0.40 |
| AIME24 | 41.88 | 46.04 (step 70, 12.755 h) | +4.16 |
| AIME25 | 33.13 | 36.77 (step 70, 12.755 h) | +3.64 |
The training reward, logged at every step, is also higher at nearly matched elapsed training times: 0.3232 for GRPO at 20.668 h (step 140) vs. 0.3342 for Delta-GRPO at 20.471 h (step 96).
Response to Reviewer qbaV [Part 3/3]
Table R5. Total training cost and accuracy, Qwen3-4B (140 updates each).
| Item | GRPO | Delta-GRPO | ExOPD | Direct-OPD |
|---|---|---|---|---|
| Teacher preparation | — | — | 20.668 h | 20.668 h |
| Main / student training | 20.668 h | 32.168 h | 12.953 h | 13.242 h |
| Total | 20.668 h | 32.168 h | 33.621 h | 33.910 h |
| Macro (%) | 52.80 | 66.39 | 53.26 | 49.84 |
(b) Baseline competitiveness. SC-GRPO, implemented under the same setting, improves over GRPO by 4.69 and 4.16 points at 4B and 8B. Our EP-GRPO implementation follows the original method and EP hyperparameters exactly (λ=0.1, η=0.2, ten progress buckets, γ=5). Its weak result stems from non-terminating generation: EP-GRPO policies fall into repetitive output loops and fail to stop, so responses are truncated at the length cap (at 8B, 80.73% of endpoint samples reach the 16,384-token cap). The original paper operates with 2,048-token generation (Qwen2.5-3B/7B), where this behavior is cut off early.
Manuscript Revision. We have added the matched-time and total-cost comparisons, and EP-GRPO diagnostics, to Appendix C.3 (Tables 9–11).
4. Exploration and Generality (W4)
Exploration. If lower entropy reflected collapsed exploration, Delta-GRPO's gain would shrink toward zero as grows. Instead, at 4B the gain persists at pass@32 and is often larger than at pass@1:
| Gain over GRPO (pp) | AIME24 | AIME25 | HMMT25 | AIME26 |
|---|---|---|---|---|
| pass@1 | +25.53 | +19.90 | +18.13 | +22.50 |
| pass@32 | +16.67 | +20.00 | +26.67 | +23.34 |
pass@k is higher on all eight benchmarks at both scales (Tables 12–13). The larger share of zero-variance groups (36.72% vs. 21.88% at step 70) includes groups in which all responses are correct, consistent with faster growth of training correctness (Figure 5).
Generality. We added a larger model and a different task domain.
Table R6. Qwen3-32B, average accuracy (%); protocol as in Table 1.
| Method | MATH500 | AIME24 | AIME25 | Mean |
|---|---|---|---|---|
| GRPO | 93.65 | 49.06 | 36.67 | 59.79 |
| Delta-GRPO | 96.02 | 71.94 | 58.33 | 75.43 |
| Gain | +2.37 | +22.88 | +21.66 | +15.64 |
On the same three benchmarks, the gain is +16.28 at 4B and +14.92 at 8B, so it does not shrink with scale.
Table R7. Code, Qwen3-4B trained on mathematics only; same evaluation protocol.
| Benchmark | GRPO | Delta-GRPO | Gain |
|---|---|---|---|
| HumanEval | 81.71 | 84.15 | +2.44 |
| HumanEval+ | 74.39 | 77.44 | +3.05 |
| MBPP | 69.31 | 72.49 | +3.18 |
| MBPP+ | 61.11 | 62.17 | +1.06 |
| CRUXEval-O | 45.50 | 46.13 | +0.63 |
| Mean | 66.40 | 68.48 | +2.07 |
Although trained only on mathematics, Delta-GRPO improves all five code benchmarks.
Manuscript Revision. We have added the Qwen3-32B and code results to Appendix D.5 (Tables 17–18), and the pass@1 versus pass@k comparison to Appendix D.1 (Table 14).
We hope these clarifications and additional results address your concerns. We welcome any further questions or discussion.
A Kind Reminder for Discussion
Dear Reviewer qbaV,
Thank you again for your time and constructive comments on our work.
We have submitted our detailed response, which we hope will address your initial concerns. As the author–reviewer discussion period is coming to an end, we respectfully ask if you could review our updates at your convenience.
If you find that our response has adequately resolved your concerns, we would be deeply grateful if you might consider reflecting this in your final evaluation. We would be happy to provide any further clarification if needed.
Best regards,
The Authors
Response to Reviewer qbaV [Part 1/3]
Dear Reviewer qbaV,
Thank you for your careful review and constructive feedback. We appreciate your recognition that the method integrates token-level policy-shift information into GRPO without an external teacher or distillation stage, and that the empirical analysis covers ablations, modulation strength, pass@k and training dynamics. We ran the ablations you proposed and address each weakness below.
1. Distinction from Direct-OPD, OPD², EP-GRPO and SC-GRPO (W1)
Delta-GRPO is the only method in which the outcome advantage fixes both the sign and the total credit of each response, while a token-level signal from the learner's own training history decides only where that credit goes.
Table R1. Comparison with related methods.
Table R2. Delta-GRPO vs. EP-GRPO.
These choices matter empirically: removing alignment drops 4B macro accuracy from 66.39 to 55.15, and breaking the token-to-history correspondence drops it to 49.63 (Table R3). Under the same setting, SC-GRPO reaches 57.49 and EP-GRPO 42.10.